Opening Files in .exe

You have to create a setup package, and while youc reate the setup, open the file types editor by right clicking on view, and selecting file types. Select the file types on target machine node and choose action, add file type. Open the Properties window, select the command property, and click the ellipsis (...) button. The select item in project dialog box appears. Select primary output from your project from the Application folder node and click OK.
 Change the Name propery to whtever u want, and the Extensions property to the extensions you want  your target files to contain. I guess you build and install.. you should be ready to go.. I assume you use VS .net. Let me know if this doesnt work for you..

Right, that works for the file extension.

I'm also wondering in terms of opening the file, what is the best way to accomplish getting an existing instance of the .exe to learn that someone has double click on a file associated with the program.  My goal is to have it work in such a way as word or paintshop file where it opens inside the existing .exe rather than somewhere else.  is there an easier way to do this than setting up a remoting instance?

You can create a registry entry:

Create a ".reg"-file with this contents:

Windows Registry Editor Version 5.00

[HKEY_CLASSES_ROOT\.ext]

[HKEY_CLASSES_ROOT\.ext\shell]

[HKEY_CLASSES_ROOT\.ext\shell\open]
@="Menu item text for open"

[HKEY_CLASSES_ROOT\.ext\shell\open\command]
@="c:\\winnt\\notepad.exe \"%1\""


Replace .ext with your file extension and the path in open\command with your application. You can use * instead of a file extension to provide a command on every file type.

----------------

Now about the instance:

When you use this open\command registry entry then there is always a new instance started. Usually to avoid two instances of the same program there is a startup code that checks for an existing instance.
You need inter process communication (memory mapped files, shared memory, ...) to send the command to the existing instance.
Some programs are realized as COM-Components and they are using the HKEY_CLASSES_ROOT\.ext\shellex\ entries (e.g.: Microsoft Office). I don't know if it would be easier that way.

Is it too big of a hack to write the file name the registry then send a windows message to the running instance?  Or should I actually take the type to create a remoting object from one instance to the other?

For that you need a target instance. You could start a TCP listener on a fixed port and when the second instance is started you try to create the listener and fail (cause the port is already occupied) then create a TcpClient and send the data to the first instance. Of course you could also use TCP remoting but you have to configure it in the code since only the first instance can obtain the port.

The way I have it setup now is that a newly created instance loops through all running processes, looks for the filename, and if it finds it, sends it a custom windows message.  the running process then gets this windows message, and goes to the registry to find the file name.  Is this a bad way to do it?

ptmcomp. If I want to do it by hand what would the actual reg key look like?  

it doesn't appear than any other file extenions behave in this manner.  I see some OpenWithList and what not, but even then paths aren't sepcified.  I'm on an XP box.  

I added that key (I to my xp box) and it didn't do anything.  It was original associated win an exe of an old version of program and wouldn't change, so I deleted all old versions.  when I tried to create the file association again, it didn't work.  I go to choose program and select my program, but it brings me back to the previous Choose Program dialog but my selected program is not in the view

I don't have any PC with Windows XP, but it worked on Windows 95, 98, ME, NT4 and 2000. I think there is a typing mistake if it doesn't work. Often also the extension linked to a filetype and then on the filetype the commands are defined.

Windows Registry Editor Version 5.00

[HKEY_CLASSES_ROOT\.ext]
@="MyFileType"

[HKEY_CLASSES_ROOT\MyFileType\DefaultIcon]

[HKEY_CLASSES_ROOT\MyFileType\DefaultIcon]
@="%SystemRoot%\system32\shell32.dll,-152"

[HKEY_CLASSES_ROOT\MyFileType\shell]

[HKEY_CLASSES_ROOT\MyFileType\shell\open]
@="Menu item text for open"

[HKEY_CLASSES_ROOT\MyFileType\shell\open\command]
@="c:\\winnt\\notepad.exe \"%1\""

May be a log off / log on  or a restart is needed to make the changes work.
Replace ".ext" with your file extension and "c:\\winnt\\notepad.exe" with your application path.