ShellExecute Issue within VB6 on Windows 8

Posted on 2014-01-15
Last Modified: 2014-01-21
We have the following function in VB6..

Public Function OpenWithDefault(ByVal FileName As String) As Boolean

   'ShellExecute returns a value greater than 32 if it was successful
    OpenWithDefault = (ShellExecute(0&, "open", FileName, vbNullString, vbNullString, SW_SHOWNORMAL) > 32)

End Function

Basically we are forcing the opening of a report in a pdf after the user runs a certain process.

In windows XP this code works fine and the report pops up fine in the PDF. HOWEVER, in Windows 8, when the  user runs the process, the report does not come up, All we see is like a flash...

Any ideas on what in Windows 8 is stopping this??
Question by:etsellinc
Welcome to Experts Exchange

Add your voice to the tech community where 5M+ people just like you are talking about what matters.

  • Help others & share knowledge
  • Earn cash & points
  • Learn & ask questions
  • 9
  • 7
LVL 40
ID: 39782866
What happens if you double-click on a pdf file in Windows Explorer on that computer?

This is basically what ShellExecute does. It starts the program registered for the extension of the file. If none is installed, nothing happens.

What is the value in OpenWithDefault after the call?

Author Comment

ID: 39783234
When I double click a pdf file it opens right up
LVL 40
ID: 39783332
And what is the value of OpenWithDefault after the call?

Have you tried with another .PDF viewer? Since this is an external application, the problem can come from an incompatibility between the viewer and Windows 8.

Have you tried running your application in Windows XP compatibility mode? You can set that by right clicking on the .exe file in Windows Explorer, through a Compatibility tab that appears under Properties.

You need to be aware that although Microsoft still supports the runtime of VB6 in Windows 8, it stopped supporting the Visual Basic 6 development environment in 2008, so there might be problems if you develop/compile VB6 applications while working in Windows 8 (

Is it possible that you were running in 32-bit under XP and 64-bit Under Windows 8? Then you enter another realm. Although 32-bit applications run in an emulation mode for 32-bit on 64-bit computers, there are issues with some programs.
Industry Leaders: We Want Your Opinion!

We value your feedback.

Take our survey and automatically be enter to win anyone of the following:
Yeti Cooler, Amazon eGift Card, and Movie eGift Card!


Author Comment

ID: 39783339
Yes the new computers are 64bit Windows 8  Machines. The XP PCs are all 32bit of course..

I have read online to try this:
replace the "open" with NULL in the command line.

Im going to try that and see if it goes.
LVL 40
ID: 39783502
Make your changes and compile on one of the old computers. As stated, the Visual Basic development environment is not supported on either Windows 8 or 64-bits and can cause problems when you compile in these.

Author Comment

ID: 39783578
I doubt its VB6. We have .net of course and we are going to try the same command in it and test it as well to eliminate that its VB6...

I of course blame Windows 8...LOL
LVL 40
ID: 39783716
Why put VB6 in your question if you are in .NET?

If you are in .NET, do not use ShellExecute. .NET uses the Process class to do the same thing. You would not even need your method with Process, since it already returns a Boolean telling you if the start was a success:

If Process.Start(FileName) Then

If that does not exactly suit your needs, look it up in the documentation. The Start method has 4 other overloads that enable you to pass parameters if needed.

Author Comment

ID: 39783723
We have an entire application written in VB6. I was just trying to eliminate that it was VB6..Its probably a permission issue on Windows 8...8 is very bad about security these days..
LVL 40
ID: 39783781
I completely agree with the overdone security. We are to a point where it is like if we had a policeman in front of our house that asks for our ID every time we show at the door.

I would not be so fast putting everything on security however. Calling ShellExecute is the same thing as a double click on the file and thus executes in the same security context. So the issue is probably somewhere else.

Unfortunately, I do not have VB6 installed in my actual environment and cannot make any test.

Author Comment

ID: 39783785
thats ok, we are going to try the replace the "open" with NULL in the command line and see if that works. Ill let you know if it does...will probably be tomorrow or the next day. We have other pressing issues to deal with first.

Accepted Solution

etsellinc earned 0 total points
ID: 39786479
Ok we have fixed the problem.

 'ShellExecute returns a value greater than 32 if it was successful
    OpenWithDefault = (ShellExecute(0&, "open", FileName, vbNullString, vbNullString, SW_SHOWNORMAL) > 32)

 'ShellExecute returns a value greater than 32 if it was successful
    OpenWithDefault = (ShellExecute(0&, "", FileName, vbNullString, vbNullString, SW_SHOWNORMAL) > 32)

What was happening is that in Windows XP under File Associations, under advance for pdf, it has an open command. Windows 8 does not have this. I would assume Windows 8 knows by default to open without having a File Association of open set..

So long story short, VB6 works fine on Windows 8, its just Microsoft has changed ALOT of things in Windows 8. This isnt the first time we have run into stuff like this. Microsoft has made many decisions about things in Windows 8 and it takes hours sometimes to track down exactly what they have changed...
LVL 40
ID: 39786554
Why do you always blame Windows 8? This is useless bashing on an OS that is not perfect but is as solid if not more than XP.

Windows 8 (not to be confused with Windows RT that is new interface and is not Windows 8 although it is presented as such almost everywhere) is Windows 7 with a few more features and new interfaces for some of the old stuff such as the Task Manager. Contrary to Vista, Windows 7 proved to be a very useful and solid Platform, and Windows 8 follows in that step.

Windows 8 has the same file association system as Windows XP. Right click on any file in Windows Explorer, select Open With, and you have the possibility to select the application that will be default for that type of file. Doing so creates the file association.

You should normally not have to do that however when only one application handles a given type of file on a computer.

Problems such as yours come from applications that installed themselves through undocumented features of older operating systems to register their file association, or do not record them at all. Many applications were rushed out to make use of the touch features of Windows 8 and have not been properly tested. And this happens to "big" applications, not only to small stuff done by independant software developers. (Acrobat Reader is one of these. Its Windows RT version is featureless and they broke a lot of useful feature in their touch version for Windows 8 compared to previous ones.)

You can thus end up with an application that installs without registering the association properly. In such a case you need to do it manually.

This is probably what happens with the pdf viewer that you have on your Windows 8 computers. It did not record Open properly. When you call ShellExecute with an empty string, you are calling the default verb, which ends up being Open.

Its not Windows 8 fault, its the pdf viewer fault.

Author Comment

ID: 39786562
ok, so its Adobe's fault on this one..BUT why it didnt set the open in the association is beyond me. I bet Adobe would blame Microsoft and Microsoft would blame them. Also, I prefer Windows 8 over XP, but there are things in it that have changed that makes no sense, but either way our problem is fixed now..

Also we have 4 Win 8 PCs that we installed Adobe Reader on and all 4 had the SAME problem with the open not being set. So either Adobe or Microsoft has a problem they need to fix!!
LVL 40
ID: 39786684
Adobe has the problem as they often have. Sometimes they seem to think that they are more important than the operating system. They do not follow the rules.

I have installed many applications on Windows 8, even some that are not supposed to be supported on Windows 8, and they create their file associations when needed.

I have some of my own application that do file associations, they have for years, and they installed perfectly and without any change in Windows 8. I do nothing fancy.

I simply follow the rules.

Author Comment

ID: 39786690
good deal..Yeah, its seem like such a minor thing...

Author Closing Comment

ID: 39796429
worked fine

Featured Post

How Do You Stack Up Against Your Peers?

With today’s modern enterprise so dependent on digital infrastructures, the impact of major incidents has increased dramatically. Grab the report now to gain insight into how your organization ranks against your peers and learn best-in-class strategies to resolve incidents.

Question has a verified solution.

If you are experiencing a similar issue, please ask a related question

The way I use Experts Exchange to assist me in analyzing and diagnosing a problem is I first enter a Verbose Question at Experts Exchange like: Office 2007 will hang when opening and saving files I then launch WordPad (any text editor will do) an…
Entering time in Microsoft Access can be difficult. An input mask often bothers users more than helping them and won't catch all typing errors. This article shows how to create a textbox for 24-hour time input with full validation politely catching …
This video Micro Tutorial explains how to clone a hard drive using a commercial software product for Windows systems called Casper from Future Systems Solutions (FSS). Cloning makes an exact, complete copy of one hard disk drive (HDD) onto another d…
This is Part 3 in a 3-part series on Experts Exchange to discuss error handling in VBA code written for Excel. Part 1 of this series discussed basic error handling code using VBA.…

726 members asked questions and received personalized solutions in the past 7 days.

Join the community of 500,000 technology professionals and ask your questions.

Join & Ask a Question