Packet loss and network congestion when running DNSMASQ on PFsense

We are having a strange problem in which when we run dnsmasq on our pfsense box, we experience massive network congestion and packet loss. We have one pfsense box with two network cards and when we enable dnsmasq, our ISP drops 85% of the packets and ping times shoot up near 1000ms. As soon as I stop the dnsmasq service, packet loss is 0% and average ping time is 45ms.

We have a second pfsense box that only has one network card and when dnsmasq is enabled, we lose massive amounts of packets on our local network and our ping times internally shoot up past 4000ms. What in the world could cause this? As soon as dnsmasq is stopped, the problem goes away.