Exchange local delivery queue backs up

I run an exchange server with GFI MailEssentials on it. About 2 months ago, I noticed the local delivery Q building up with messages as many as 3000. I would go through, clean out what was spam and it would be fine for a few weeks until I had to do this again. I finally emailed GFI support about this and they had a look through the diagnostic information that their program picks up but he was unable to see anything that was out of place. The last email that I received from him was that the GFI ME does not touch the local delivery queue and that the local delivery queue holds messages that are sent between users.

So, here's the technical stuff
Server hardware IBM dual Xeon 3Ghz, with 1GB ram
Windows 2003 Standard Server with SP2
Exchange 2003 v6.5 (build 7638.2 SP2)

So, to fix the problem, I simple restart the server and all is fine until the next day. I have gone though the logs and have found nothing that stands out. Meanwhile, my users are getting frustrated because they can't get their email during the evening or early in the morning (until I've restarted the server)

Spam volume for example, this is pretty standard each day. It's between 7500 and 8000 emails each day
Total messages processed 7997, total spam 7143. Total processed 8056, 7345 spam etc.

Any help is appreciated