ShellExecute Issue within VB6 on Windows 8

Posted on 2014-01-15
Medium Priority
Last Modified: 2014-01-21
We have the following function in VB6..

Public Function OpenWithDefault(ByVal FileName As String) As Boolean

   'ShellExecute returns a value greater than 32 if it was successful
    OpenWithDefault = (ShellExecute(0&, "open", FileName, vbNullString, vbNullString, SW_SHOWNORMAL) > 32)

End Function

Basically we are forcing the opening of a report in a pdf after the user runs a certain process.

In windows XP this code works fine and the report pops up fine in the PDF. HOWEVER, in Windows 8, when the  user runs the process, the report does not come up, All we see is like a flash...

Any ideas on what in Windows 8 is stopping this??
Question by:etsellinc
Welcome to Experts Exchange

Add your voice to the tech community where 5M+ people just like you are talking about what matters.

  • Help others & share knowledge
  • Earn cash & points
  • Learn & ask questions
  • 9
  • 7
LVL 40
ID: 39782866
What happens if you double-click on a pdf file in Windows Explorer on that computer?

This is basically what ShellExecute does. It starts the program registered for the extension of the file. If none is installed, nothing happens.

What is the value in OpenWithDefault after the call?

Author Comment

ID: 39783234
When I double click a pdf file it opens right up
LVL 40
ID: 39783332
And what is the value of OpenWithDefault after the call?

Have you tried with another .PDF viewer? Since this is an external application, the problem can come from an incompatibility between the viewer and Windows 8.

Have you tried running your application in Windows XP compatibility mode? You can set that by right clicking on the .exe file in Windows Explorer, through a Compatibility tab that appears under Properties.

You need to be aware that although Microsoft still supports the runtime of VB6 in Windows 8, it stopped supporting the Visual Basic 6 development environment in 2008, so there might be problems if you develop/compile VB6 applications while working in Windows 8 (http://msdn.microsoft.com/en-us/vstudio/ms788708.aspx).

Is it possible that you were running in 32-bit under XP and 64-bit Under Windows 8? Then you enter another realm. Although 32-bit applications run in an emulation mode for 32-bit on 64-bit computers, there are issues with some programs.
Get 15 Days FREE Full-Featured Trial

Benefit from a mission critical IT monitoring with Monitis Premium or get it FREE for your entry level monitoring needs.
-Over 200,000 users
-More than 300,000 websites monitored
-Used in 197 countries
-Recommended by 98% of users


Author Comment

ID: 39783339
Yes the new computers are 64bit Windows 8  Machines. The XP PCs are all 32bit of course..

I have read online to try this:
replace the "open" with NULL in the command line.

Im going to try that and see if it goes.
LVL 40
ID: 39783502
Make your changes and compile on one of the old computers. As stated, the Visual Basic development environment is not supported on either Windows 8 or 64-bits and can cause problems when you compile in these.

Author Comment

ID: 39783578
I doubt its VB6. We have .net of course and we are going to try the same command in it and test it as well to eliminate that its VB6...

I of course blame Windows 8...LOL
LVL 40
ID: 39783716
Why put VB6 in your question if you are in .NET?

If you are in .NET, do not use ShellExecute. .NET uses the Process class to do the same thing. You would not even need your method with Process, since it already returns a Boolean telling you if the start was a success:

If Process.Start(FileName) Then

If that does not exactly suit your needs, look it up in the documentation. The Start method has 4 other overloads that enable you to pass parameters if needed.

Author Comment

ID: 39783723
We have an entire application written in VB6. I was just trying to eliminate that it was VB6..Its probably a permission issue on Windows 8...8 is very bad about security these days..
LVL 40
ID: 39783781
I completely agree with the overdone security. We are to a point where it is like if we had a policeman in front of our house that asks for our ID every time we show at the door.

I would not be so fast putting everything on security however. Calling ShellExecute is the same thing as a double click on the file and thus executes in the same security context. So the issue is probably somewhere else.

Unfortunately, I do not have VB6 installed in my actual environment and cannot make any test.

Author Comment

ID: 39783785
thats ok, we are going to try the replace the "open" with NULL in the command line and see if that works. Ill let you know if it does...will probably be tomorrow or the next day. We have other pressing issues to deal with first.

Accepted Solution

etsellinc earned 0 total points
ID: 39786479
Ok we have fixed the problem.

 'ShellExecute returns a value greater than 32 if it was successful
    OpenWithDefault = (ShellExecute(0&, "open", FileName, vbNullString, vbNullString, SW_SHOWNORMAL) > 32)

 'ShellExecute returns a value greater than 32 if it was successful
    OpenWithDefault = (ShellExecute(0&, "", FileName, vbNullString, vbNullString, SW_SHOWNORMAL) > 32)

What was happening is that in Windows XP under File Associations, under advance for pdf, it has an open command. Windows 8 does not have this. I would assume Windows 8 knows by default to open without having a File Association of open set..

So long story short, VB6 works fine on Windows 8, its just Microsoft has changed ALOT of things in Windows 8. This isnt the first time we have run into stuff like this. Microsoft has made many decisions about things in Windows 8 and it takes hours sometimes to track down exactly what they have changed...
LVL 40
ID: 39786554
Why do you always blame Windows 8? This is useless bashing on an OS that is not perfect but is as solid if not more than XP.

Windows 8 (not to be confused with Windows RT that is new interface and is not Windows 8 although it is presented as such almost everywhere) is Windows 7 with a few more features and new interfaces for some of the old stuff such as the Task Manager. Contrary to Vista, Windows 7 proved to be a very useful and solid Platform, and Windows 8 follows in that step.

Windows 8 has the same file association system as Windows XP. Right click on any file in Windows Explorer, select Open With, and you have the possibility to select the application that will be default for that type of file. Doing so creates the file association.

You should normally not have to do that however when only one application handles a given type of file on a computer.

Problems such as yours come from applications that installed themselves through undocumented features of older operating systems to register their file association, or do not record them at all. Many applications were rushed out to make use of the touch features of Windows 8 and have not been properly tested. And this happens to "big" applications, not only to small stuff done by independant software developers. (Acrobat Reader is one of these. Its Windows RT version is featureless and they broke a lot of useful feature in their touch version for Windows 8 compared to previous ones.)

You can thus end up with an application that installs without registering the association properly. In such a case you need to do it manually.

This is probably what happens with the pdf viewer that you have on your Windows 8 computers. It did not record Open properly. When you call ShellExecute with an empty string, you are calling the default verb, which ends up being Open.

Its not Windows 8 fault, its the pdf viewer fault.

Author Comment

ID: 39786562
ok, so its Adobe's fault on this one..BUT why it didnt set the open in the association is beyond me. I bet Adobe would blame Microsoft and Microsoft would blame them. Also, I prefer Windows 8 over XP, but there are things in it that have changed that makes no sense, but either way our problem is fixed now..

Also we have 4 Win 8 PCs that we installed Adobe Reader on and all 4 had the SAME problem with the open not being set. So either Adobe or Microsoft has a problem they need to fix!!
LVL 40
ID: 39786684
Adobe has the problem as they often have. Sometimes they seem to think that they are more important than the operating system. They do not follow the rules.

I have installed many applications on Windows 8, even some that are not supposed to be supported on Windows 8, and they create their file associations when needed.

I have some of my own application that do file associations, they have for years, and they installed perfectly and without any change in Windows 8. I do nothing fancy.

I simply follow the rules.

Author Comment

ID: 39786690
good deal..Yeah, its seem like such a minor thing...

Author Closing Comment

ID: 39796429
worked fine

Featured Post


Modern healthcare requires a modern cloud. View this brief video to understand how the Concerto Cloud for Healthcare can help your organization.

Question has a verified solution.

If you are experiencing a similar issue, please ask a related question

This article describes how to set permissions to allow a limited-permissions user to start and stop a particular System Service.   It is always best to give users only the permissions that they need to perform their job, so tweaking particular permi…
When you upgrade from Windows 8 to 8.1 or to Windows 10 or if you are like me you are on the Insider Program you may find yourself with many 450MB recovery partitions.  With a traditional disk that may not be a problem but with relatively smaller SS…
This video Micro Tutorial explains how to clone a hard drive using a commercial software product for Windows systems called Casper from Future Systems Solutions (FSS). Cloning makes an exact, complete copy of one hard disk drive (HDD) onto another d…
With the advent of Windows 10, Microsoft is pushing a Get Windows 10 icon into the notification area (system tray) of qualifying computers. There are many reasons for wanting to remove this icon. This two-part Experts Exchange video Micro Tutorial s…

771 members asked questions and received personalized solutions in the past 7 days.

Join the community of 500,000 technology professionals and ask your questions.

Join & Ask a Question