Are you talking about the new "Add Media" button in WordPress 3.5 or do you have a different version of WordPress? Or are you attempting to use the "Advanced Image" option in TinyMCE Advanced?
I have TinyMCE Advanced running on a number of sites with no errors so I know it works with 3.5. If you are running 3.5 and having problems, enable TinyMCE Advanced and then disable everything else and see if it works. If so, you are having a plugin conflict with one of your other plugins. Enable one at a time to figure out which one is causing the problem.
all browsers have a configurable cache and temporary internet files.
delete the temporary files and retry.
also try from another PC/laptop.
it sounds like a javascript problem which could mean a conflict as jason mentioned or corrupt javascript files - whch could be sorted by clearing out your local copies or try reinstalling the extension if they aren't not installed properly on the server
I would try it in a different browser - Chrome, Firefox and IE
and see if this happens.
empty your browsers cache or temporary internet files ad retry.
also check the javascript files have been updated in wordpress