Solved

ShellExecute Issue within VB6 on Windows 8

Posted on 2014-01-15
16
4,048 Views
Last Modified: 2014-01-21
We have the following function in VB6..

Public Function OpenWithDefault(ByVal FileName As String) As Boolean

   'ShellExecute returns a value greater than 32 if it was successful
    OpenWithDefault = (ShellExecute(0&, "open", FileName, vbNullString, vbNullString, SW_SHOWNORMAL) > 32)

End Function

Basically we are forcing the opening of a report in a pdf after the user runs a certain process.

In windows XP this code works fine and the report pops up fine in the PDF. HOWEVER, in Windows 8, when the  user runs the process, the report does not come up, All we see is like a flash...

Any ideas on what in Windows 8 is stopping this??
0
Comment
Question by:etsellinc
  • 9
  • 7
16 Comments
 
LVL 40
ID: 39782866
What happens if you double-click on a pdf file in Windows Explorer on that computer?

This is basically what ShellExecute does. It starts the program registered for the extension of the file. If none is installed, nothing happens.

What is the value in OpenWithDefault after the call?
0
 

Author Comment

by:etsellinc
ID: 39783234
When I double click a pdf file it opens right up
0
 
LVL 40
ID: 39783332
And what is the value of OpenWithDefault after the call?

Have you tried with another .PDF viewer? Since this is an external application, the problem can come from an incompatibility between the viewer and Windows 8.

Have you tried running your application in Windows XP compatibility mode? You can set that by right clicking on the .exe file in Windows Explorer, through a Compatibility tab that appears under Properties.

You need to be aware that although Microsoft still supports the runtime of VB6 in Windows 8, it stopped supporting the Visual Basic 6 development environment in 2008, so there might be problems if you develop/compile VB6 applications while working in Windows 8 (http://msdn.microsoft.com/en-us/vstudio/ms788708.aspx).

Is it possible that you were running in 32-bit under XP and 64-bit Under Windows 8? Then you enter another realm. Although 32-bit applications run in an emulation mode for 32-bit on 64-bit computers, there are issues with some programs.
0
 

Author Comment

by:etsellinc
ID: 39783339
Yes the new computers are 64bit Windows 8  Machines. The XP PCs are all 32bit of course..

I have read online to try this:
replace the "open" with NULL in the command line.

Im going to try that and see if it goes.
0
 
LVL 40
ID: 39783502
Make your changes and compile on one of the old computers. As stated, the Visual Basic development environment is not supported on either Windows 8 or 64-bits and can cause problems when you compile in these.
0
 

Author Comment

by:etsellinc
ID: 39783578
I doubt its VB6. We have .net of course and we are going to try the same command in it and test it as well to eliminate that its VB6...

I of course blame Windows 8...LOL
0
 
LVL 40
ID: 39783716
Why put VB6 in your question if you are in .NET?

If you are in .NET, do not use ShellExecute. .NET uses the Process class to do the same thing. You would not even need your method with Process, since it already returns a Boolean telling you if the start was a success:

If Process.Start(FileName) Then

If that does not exactly suit your needs, look it up in the documentation. The Start method has 4 other overloads that enable you to pass parameters if needed.
0
 

Author Comment

by:etsellinc
ID: 39783723
We have an entire application written in VB6. I was just trying to eliminate that it was VB6..Its probably a permission issue on Windows 8...8 is very bad about security these days..
0
Why You Should Analyze Threat Actor TTPs

After years of analyzing threat actor behavior, it’s become clear that at any given time there are specific tactics, techniques, and procedures (TTPs) that are particularly prevalent. By analyzing and understanding these TTPs, you can dramatically enhance your security program.

 
LVL 40
ID: 39783781
I completely agree with the overdone security. We are to a point where it is like if we had a policeman in front of our house that asks for our ID every time we show at the door.

I would not be so fast putting everything on security however. Calling ShellExecute is the same thing as a double click on the file and thus executes in the same security context. So the issue is probably somewhere else.

Unfortunately, I do not have VB6 installed in my actual environment and cannot make any test.
0
 

Author Comment

by:etsellinc
ID: 39783785
thats ok, we are going to try the replace the "open" with NULL in the command line and see if that works. Ill let you know if it does...will probably be tomorrow or the next day. We have other pressing issues to deal with first.
0
 

Accepted Solution

by:
etsellinc earned 0 total points
ID: 39786479
Ok we have fixed the problem.

OLD CODE:
 'ShellExecute returns a value greater than 32 if it was successful
    OpenWithDefault = (ShellExecute(0&, "open", FileName, vbNullString, vbNullString, SW_SHOWNORMAL) > 32)

NEW CODE:
 'ShellExecute returns a value greater than 32 if it was successful
    OpenWithDefault = (ShellExecute(0&, "", FileName, vbNullString, vbNullString, SW_SHOWNORMAL) > 32)

What was happening is that in Windows XP under File Associations, under advance for pdf, it has an open command. Windows 8 does not have this. I would assume Windows 8 knows by default to open without having a File Association of open set..

So long story short, VB6 works fine on Windows 8, its just Microsoft has changed ALOT of things in Windows 8. This isnt the first time we have run into stuff like this. Microsoft has made many decisions about things in Windows 8 and it takes hours sometimes to track down exactly what they have changed...
0
 
LVL 40
ID: 39786554
Why do you always blame Windows 8? This is useless bashing on an OS that is not perfect but is as solid if not more than XP.

Windows 8 (not to be confused with Windows RT that is new interface and is not Windows 8 although it is presented as such almost everywhere) is Windows 7 with a few more features and new interfaces for some of the old stuff such as the Task Manager. Contrary to Vista, Windows 7 proved to be a very useful and solid Platform, and Windows 8 follows in that step.

Windows 8 has the same file association system as Windows XP. Right click on any file in Windows Explorer, select Open With, and you have the possibility to select the application that will be default for that type of file. Doing so creates the file association.

You should normally not have to do that however when only one application handles a given type of file on a computer.

Problems such as yours come from applications that installed themselves through undocumented features of older operating systems to register their file association, or do not record them at all. Many applications were rushed out to make use of the touch features of Windows 8 and have not been properly tested. And this happens to "big" applications, not only to small stuff done by independant software developers. (Acrobat Reader is one of these. Its Windows RT version is featureless and they broke a lot of useful feature in their touch version for Windows 8 compared to previous ones.)

You can thus end up with an application that installs without registering the association properly. In such a case you need to do it manually.

This is probably what happens with the pdf viewer that you have on your Windows 8 computers. It did not record Open properly. When you call ShellExecute with an empty string, you are calling the default verb, which ends up being Open.

Its not Windows 8 fault, its the pdf viewer fault.
0
 

Author Comment

by:etsellinc
ID: 39786562
ok, so its Adobe's fault on this one..BUT why it didnt set the open in the association is beyond me. I bet Adobe would blame Microsoft and Microsoft would blame them. Also, I prefer Windows 8 over XP, but there are things in it that have changed that makes no sense, but either way our problem is fixed now..

Also we have 4 Win 8 PCs that we installed Adobe Reader on and all 4 had the SAME problem with the open not being set. So either Adobe or Microsoft has a problem they need to fix!!
0
 
LVL 40
ID: 39786684
Adobe has the problem as they often have. Sometimes they seem to think that they are more important than the operating system. They do not follow the rules.

I have installed many applications on Windows 8, even some that are not supposed to be supported on Windows 8, and they create their file associations when needed.

I have some of my own application that do file associations, they have for years, and they installed perfectly and without any change in Windows 8. I do nothing fancy.

I simply follow the rules.
0
 

Author Comment

by:etsellinc
ID: 39786690
good deal..Yeah, its seem like such a minor thing...
0
 

Author Closing Comment

by:etsellinc
ID: 39796429
worked fine
0

Featured Post

How to run any project with ease

Manage projects of all sizes how you want. Great for personal to-do lists, project milestones, team priorities and launch plans.
- Combine task lists, docs, spreadsheets, and chat in one
- View and edit from mobile/offline
- Cut down on emails

Join & Write a Comment

Have you ever had a hard drive that you can't boot into, but need to change the registry? Here is the solution! This article guides you through accessing and editing a registry of a non-primary drive. To read registry information on a non-prim…
What my article will show is if you ever had to do processing to a listbox without being able to just select all the items in it. My software Visual Studio 2008 crystal report v11 My issue was I wanted to add crystal report to a form and show…
This video Micro Tutorial explains how to clone a hard drive using a commercial software product for Windows systems called Casper from Future Systems Solutions (FSS). Cloning makes an exact, complete copy of one hard disk drive (HDD) onto another d…
With the advent of Windows 10, Microsoft is pushing a Get Windows 10 icon into the notification area (system tray) of qualifying computers. There are many reasons for wanting to remove this icon. This two-part Experts Exchange video Micro Tutorial s…

758 members asked questions and received personalized solutions in the past 7 days.

Join the community of 500,000 technology professionals and ask your questions.

Join & Ask a Question

Need Help in Real-Time?

Connect with top rated Experts

24 Experts available now in Live!

Get 1:1 Help Now