Spam is causing real mail to be delayed

Dear experts,

I have an SBS2003 box. Standard enough setup - nothing fancy, 20 users. Mail is delivered directly to the SBS box. Recently the amount of spam which we receive is unbelievably high. Daily we receive 100,000 +  emails of which probably around 120 are legitimate. 99.9% of the spam are for nonexistinguser@domain.com

I have GFI Mailessentials installed to manage the spam and it is working well - it blocks the spam and delivers the real mail. But the problem is that the real mail is taking so long to arrive. Having done a few tests, the fastest took 25 minutes to deliver. A lot of the time, senders get "message delayed" messages sent back.

The way I see it, if the mail is delivered to the ip address (according to the mx record) which is the router, then it has to deal with it. The traffic is already present on my system.

Is there any way I can get the spam dropped before my system has to process it?

I have been considering a few options:
1. Have the ISP host the email to let their spam filters do the most of the work, then collect mail using POP3 connector
2. Change our mail domain - a bit drastic I know.

I'd rather not do either option. All help appreciated.

Thanks in advance.