Hybrid Meeting Zoom & in-Person

I represent a men's breakfast group that met weekly before the pandemic & has been meeting by Zoom for about the last 14 months.

We have evolved to a situation where 3 of our attendees live over 50 miles away from where the core group is, this works fine, of course, with Zoom.

We want to return to in-person meetings but also allow others to participate remotely.

We ran a test where 4 of us met in a room at our congregation & 3 remotely via zoom. It did not work well. One issue was that unless all but one of us in person has their device muted, there is annoying, very strong feedback.

Is there any way to avoid that WITHOUT every body in person being on mute? Would speaker view work. But even then, would not everyone have to be on speaker view, or can it be forced by the host? Or some other way?

Thank you.