Is there a way to insert a file (dll) into a vb.Net setup project from command line?

If you want to do that, you cannot do it simply from a command line. You need an application that can open the .msi file and recreate it.

And even doing so, you would probably end up with problems, because the reference compiled into your application would probably not match the updated dll.

Your best bet is to recompile the application with the updated dll and recreate the setup project.

VB6 comes with a Package and Deployment Wizard which creates an installation package comprising a set-up program and the necessary files. The user can specify extra files that the Wizard doesn't add automatically.

But once you create the installation package, how do you then add/update files to it from a command line?

You don't. You create a new package.

The package comes in the form a .msi file. Just as you cannot modifiy a .exe without recompiling it, you cannot modify a .msi without recreating it.

If you could explain more in detail what you want to "update" in the installation, maybe we could help you.

I have a dll which is part of the setup project. The included dll gets updated from time to time. So i would like an easy way from the command line to recompile the solution/project with the newly added dll.

First of all, something I did not catch when I first read your question: "a vb6 setup solution/project". The notion of a solution did not exist in VB6. And VB6 did not have a Setup Project, it used as Wizard to prepare setups. Are you sure that you are in VB6 and not VB.NET? Both use different ways of preparing setups and updates.

"The included dll gets updated from time to time" if it's only from time to time, then recompiling and creating a new setup manually takes only a few minutes, while creating a command line compile is a lot of work prone to errors and omissions. And this, no matter if you are in VB6 or VB.NET.

If you are in .NET however, and especially if this is for an application that is deployed in-house, then you might consider a ClickOnce deployment over a Setup deployment. This is the easiest way to do what you want to do, and a new update is prepared with only the click of a button, making it even easier than creating a script that can be called from a command line.

Yes, you can compile an .exe or a .dll with command line switches. This is from the MSDN under Command Line Arguments:

"Syntax

vb5[.exe] [[{/run | /r} projectname] [[{/d compileconst}] {/make | /m } projectname] [{/makedll | /l} projectname] {/cmd argument | /c argument}][{/runexit} projectname][{/m} or {/runexit} projectname /out filename}][{/m}][/sdi] or [/mdi]"

and

"/makedll or /l Tells Visual Basic to compile projectname and make an in-process ActiveX server (.dll) file from it. "

You could run that in a .bat file or a VBScript snippet and then distribute the dll on its own.
However, how would the source code editing be done? You would still have to open an editor, which might as well be VB6.

Note that, VB6 has a Project Group (.vbg) which does the more or less the same as a Solution in .net.

It turns out I am using:
Microsoft (R) Visual Studio Version 8.0.50727.42.
Copyright (C) Microsoft Corp 1984-2005. All rights reserved.

Does that make a difference?

Will ClickOnce work with external customers that only have an ftp site as they have protected environments? It sounds like a great solution however not sure if would work for our purposes. We usually just ftp the setup and let there IT install at will.

What about updating the metadata in the solution file with the updated file path information and then building?
I noticed you can open a solution file and see all the files listed.