I use the plugin Menu Custom Images in a Wordpress installation. I also use Shadowbox. In the menu setup for each menu choice there is an entry for "Link Relationship" In here you can put "Shadowbox" which will render rel="shadowbox" in your menu code. However, if I enter Shadowbox[myGallery] the menu plugin automatically strips the brackets.
I edited a file a while back to make this work but for the life of me can not remember what I have to edit to make it work. Any ideas???