Adobe Photoshop CS4 - smooth rough edges? Illustrator a better choice?

You can use Illustrator or Photoshop, it is a matter of preference. The ability to zoom without ever becoming pixelated comes from being vector art. Photoshop is better with raster (but it also has vector capabilities). The pen tool and shape tools are vector, if you make a square and try to paint it, it will tell you it must be rasterized (now it is no longer vector).

If you don't understand all of that, the simple answer is Illustrator is the preferred method to work with vector art. If you save the document as a jpeg, png, gif, it will become rasterized and not be zoomable without pixelation. The available formats to remain vector are .ai (adobe illustrator format), .eps (encapsuled postscript file), .pdf (portable document format). If you have any questions please ask them and I will try to help.

Thanks.