Outside emails take a long time to receive

We have SBS 2008 with Exchange Server 2007.  Outgoing emails to outside the organization recipients are being sent and received without a problems.  Incoming emails from outside the organization are not.  The receiver eventually receives them but hours after it was sent.  I ran Microsoft Remote Connectivity test for incoming emails from an outside source and found that when the Windows Firewall, it failed the test pointing to a firewall issue.  I turned off the Windows Firewall and got connectivity.  When I try to email from another outside email, it sends but the intended receiver still doesn't get the email until several hours after it is sent.  I thinking it is something internal that is causing the problem but not sure what it is.  Any insight would be helpful.