Despeckle Scanned Document in Photoshop Without Damaging Text

This is an interesting-sounding job! Do you have an image sample by any chance?

Offhand, the first thing I would suggest that if you need to protect text, you may be able to use some neat selection techniques to prevent the despeckle from affecting things you want to keep.

If you have black text on a grayscale image, here's a neat set of steps for selecting everything except the text:
1) Pull up your Channels panel--it's usually next to your layers panel, or can be accessed from Window>Channels
2) For an RGB image, you'll see 4 items. You have an RGB channel (showing the composite of all three channels) and 3 additional channels labeled Red, Green, and Blue. Click each channel to view a single-color (grayscale) representation of each color and select whichever one has the text looking the darkest and the non-text looking the lightest.
3) Duplicate that channel (Right-click and choose to duplicate that channel (we're going to tweak it, and we don't want to change the actual color balance of the photo, so we use a copy)

Now, the object will be to make this duplicate channel into a black and white image with white representing what is selected and black representing what is not. This way, we can load the channel as a selection and apply the despeckle filter to only the specific areas we need.

4) Your best bet is to initially apply the Levels adjustment to your duplicated channel. By dragging the white point slider to the left, you can cause most of the grey areas in the image to turn white. If the text is not completely black, you can drag the black point slider to the right as well to darken that up.
5) All that you'll have left to do is clean up the selection a bit by pulling out the brush tool or lasso tool and filling in any areas that were left out with either white or black.
5) You can load your channel as a selection by holding your Control key and clicking on it, or by clicking the "Load channel as a selction" button at the bottom of the channels panel.
6) Select the RGB composite channel to return to a normal-looking image and apply filters as needed!

With a sample, I can probably provide a more targeted solution, but give this method a try!

I agree with photoshopgrl.
The best way I've found it to just get in there with a small brush the same color as the background.
The time you will waste tweaking levels to lose a few pixels here and there, you may as well brush them out. If you do try the levels approach I can guarantee you will not get all of them and end up using the brush anyway.