Windows 7 Explorer - Favorites Slow to load

I have a weird issue i have noticed on a few Windows 7 PC's at my work and at home.

When try to go to the Downloads Favorite it will take forever. I have waited 5 minutes at times for it to open the folder. The green progress bar in the Address bar will be moving, but very slowly. If I go to C:\Users\Username\Downloads it immediately opens though.

There has been times, out of habit, that i will click the Favorites->Downloads link, remember the problem, then just navigate to the folder directly and will be able to open it before the Favorites link does. It doesn't freeze Explorer or anything, as i can still navigate to the folder directly while waiting for it to load, the progress bar is moving, it is just very annoying.

I have tried deleting the Downloads link and recreating it, but it still does the same thing, and it happens on my Home PC and I have seen it happen on 3 PC's at work. Both 64-bit and 32-bit PC's. The hardware on these PC's are all very fast and new.

Any Solutions out there? Thanks.