Classpath problems with IBM App Srv: Jython & ORO conflicts
Hi Experts,
this seems to be a classic Java/IWebphere problem, considering the amount of Google hits I get about classpath issues with IBM Websphere and Jython. Main story that precisely describes the problem I have: http://www.ibm.com/developerworks/websphere/techjournal/0406_brown/0406_brown.html (the suggestions on IBM's website do not suffice, unfortunately.)
The issue: I need to use a newer ORO library from Apache Commons, namely Jakarta Oro 2.0.8. However, IBM comes bundled with a ton of libraries and preferably it loads these libraries first (PARENT_FIRST). This means that if you have any newer jar file on the lib path of your application (a WAR in my situation) you can be certain it will never be loaded.
According to the aforementioned link, Java will never "see" my newer version. To change that I should set the classloader behavior to "PARENT_LAST" (either on server, app or module level). Unfortunately this raises a ton of new issues which seem hard to solve (many incompatibilities).
I also tried adding it to the MANIFEST.MF, but that wouldn't work because IBM first loads its own classes and then the ones in the MANIFEST, which in turn must be part of the EAR.
When I renamed jython.jar to jython.old, my NoSuchMethodError went away. The reason that my development environment (Rational Rose XDE) does not complain is because it uses its own classpath for the application, hence the error only appears after deployment.
I do not have access to the deployment servers and thus I cannot disable jython.jar by hand. It has to be done using the server interface.
The question is simple, but possibly hard (?) to answer: how can I force the loading of the new ORO lib or how can I force Websphere to stop loading jython.jar. The means I have are:
1. change the configuration of the WAR or EAR
2. change the configuration of the server settings
3. programming (I have full access to applicable sources)
Anybody any idea? Any help's appreciated,
Cheers,
-- Abel --
this seems to be a classic Java/IWebphere problem, considering the amount of Google hits I get about classpath issues with IBM Websphere and Jython. Main story that precisely describes the problem I have: http://www.ibm.com/developerworks/websphere/techjournal/0406_brown/0406_brown.html (the suggestions on IBM's website do not suffice, unfortunately.)
The issue: I need to use a newer ORO library from Apache Commons, namely Jakarta Oro 2.0.8. However, IBM comes bundled with a ton of libraries and preferably it loads these libraries first (PARENT_FIRST). This means that if you have any newer jar file on the lib path of your application (a WAR in my situation) you can be certain it will never be loaded.
According to the aforementioned link, Java will never "see" my newer version. To change that I should set the classloader behavior to "PARENT_LAST" (either on server, app or module level). Unfortunately this raises a ton of new issues which seem hard to solve (many incompatibilities).
I also tried adding it to the MANIFEST.MF, but that wouldn't work because IBM first loads its own classes and then the ones in the MANIFEST, which in turn must be part of the EAR.
When I renamed jython.jar to jython.old, my NoSuchMethodError went away. The reason that my development environment (Rational Rose XDE) does not complain is because it uses its own classpath for the application, hence the error only appears after deployment.
I do not have access to the deployment servers and thus I cannot disable jython.jar by hand. It has to be done using the server interface.
The question is simple, but possibly hard (?) to answer: how can I force the loading of the new ORO lib or how can I force Websphere to stop loading jython.jar. The means I have are:
1. change the configuration of the WAR or EAR
2. change the configuration of the server settings
3. programming (I have full access to applicable sources)
Anybody any idea? Any help's appreciated,
Cheers,
-- Abel --
ASKER CERTIFIED SOLUTION
membership
This solution is only available to members.
To access this solution, you must be a member of Experts Exchange.
ASKER
Hmm, when I think about it, with .NET or plain old (not so good) COM+ it was possible to use the version information in the dll to call the correct one (or when not using version information it called the latest version). Isn't there a mechanism somehow with Java that does something similar? That would give the possibility to mix and match older libs that are dependent on other older libs, together with more recent libs that you need to use yourself.
(example, again: Jython is dependent on an old ORO, and you want the new ORO *and* Jython).
(example, again: Jython is dependent on an old ORO, and you want the new ORO *and* Jython).
SOLUTION
membership
This solution is only available to members.
To access this solution, you must be a member of Experts Exchange.
ASKER
Well, yes, I know which specific library needs to be referenced, I just generalized the solution. The lookup is done once (singleton) and a reference to the needed classes is then kept in memory.
It feels a bit as a large way around to get a specific class from a specific jar file. Version conflicts can happen more often (in situations you are stuck with a default set of libraries and you cannot touch this default set, however, you need the new versions of these libs), why is it so hard to choose a specific lib, or, put differently, a lib with a specific version in the Manifest file. I mean the information is all there, it shouldn't be too hard should it?
It feels a bit as a large way around to get a specific class from a specific jar file. Version conflicts can happen more often (in situations you are stuck with a default set of libraries and you cannot touch this default set, however, you need the new versions of these libs), why is it so hard to choose a specific lib, or, put differently, a lib with a specific version in the Manifest file. I mean the information is all there, it shouldn't be too hard should it?
SOLUTION
membership
This solution is only available to members.
To access this solution, you must be a member of Experts Exchange.
ASKER
Yes, of course. And they do. This one too. However, I needed added functionality (new methods in the new version). And the classpath system of IBM WSAS was "bottom up", i.e., it first looks at Java, then at IBM's classpath and lastly at you own added libs. Where the default should be the other way around.
You don't add libs for them to not be used or found.
So, actually what we are talking of here is forward compatibility. IBM finds the old jar first and sees it as the "right" jar (however in my dev IDE, the normal way of "most specific first" was followed and it finds the added libs correctly, but at runtime, the IBM server takes over and uses this backward method), instead of the new one. And forward compatibility is, by its nature, limited.
You don't add libs for them to not be used or found.
So, actually what we are talking of here is forward compatibility. IBM finds the old jar first and sees it as the "right" jar (however in my dev IDE, the normal way of "most specific first" was followed and it finds the added libs correctly, but at runtime, the IBM server takes over and uses this backward method), instead of the new one. And forward compatibility is, by its nature, limited.
SOLUTION
membership
This solution is only available to members.
To access this solution, you must be a member of Experts Exchange.
SOLUTION
membership
This solution is only available to members.
To access this solution, you must be a member of Experts Exchange.
SOLUTION
membership
This solution is only available to members.
To access this solution, you must be a member of Experts Exchange.
ASKER
> If you have only set this in one part of your app, it just has no effect.
See my earlier comments: it costs too many other problems. I assume that when the system was designed like this from the ground up, there weren't any problems at all. But this is an age old system (hence Struts 1.0.2, WSAS 5.1 with an old Jython and Java 1.4 etc) which has been build and changed over the last years.
Though I do think that changing this order should not really cause "class not found" errors, as the only location on the classpath is further away, and nearer libraries are newer and shouldn't give "class not found" errors. Hmm, maybe I should look into that solution again and try to analyse some of the errors. However, the classes not found were hundredths, which is why I didn't bother to look further and I changed it back to "normal" (why IBM thinks this is normal is beyond me).
See my earlier comments: it costs too many other problems. I assume that when the system was designed like this from the ground up, there weren't any problems at all. But this is an age old system (hence Struts 1.0.2, WSAS 5.1 with an old Jython and Java 1.4 etc) which has been build and changed over the last years.
Though I do think that changing this order should not really cause "class not found" errors, as the only location on the classpath is further away, and nearer libraries are newer and shouldn't give "class not found" errors. Hmm, maybe I should look into that solution again and try to analyse some of the errors. However, the classes not found were hundredths, which is why I didn't bother to look further and I changed it back to "normal" (why IBM thinks this is normal is beyond me).
ASKER
> Unfortunately, the classloader stuff is sometimes counter intuitive. :-(
that's an understatement. I've programmed in Java for years, and Java always sticked with me easily, even if I stopped programming for some time. But classpath issues had the tendency to always come up and once there, were hard to solve.
Yes, I think I can share that code. However, it is quite straightforward. I'm not in the office today, so bear with me a while.
that's an understatement. I've programmed in Java for years, and Java always sticked with me easily, even if I stopped programming for some time. But classpath issues had the tendency to always come up and once there, were hard to solve.
Yes, I think I can share that code. However, it is quite straightforward. I'm not in the office today, so bear with me a while.
SOLUTION
membership
This solution is only available to members.
To access this solution, you must be a member of Experts Exchange.
SOLUTION
membership
This solution is only available to members.
To access this solution, you must be a member of Experts Exchange.
ASKER
I haven't been clear about my dev env, it isn't WSAD, but XDE from Rational Rose, which is built on top of Eclipse 2.1. To add to the confusion, the dev env uses Java 1.3 and WSAS (WAS?) uses Java 1.4. But that's something I can configure differently.
I think that I am quite aware of the different possibilities of PARTEN_LAST by now. These several options are there for a reason of course. To find out what the culprit is, I reckon I'll have to create a small project and let it use this new library and see how it goes.
If it were only for ORO I probably go with just the workaround solution. But we are planning to moving from Struts 1.0.2 to Struts 1.3.8 (no, not 2.0) and likely we will hit some lib/classpath problems there too. Anyway, I will try a couple of scenario's and suggestions of you two to see how it goes (basically, the suggestion of "everywhere" didn't work out-of-the box and the suggestion of "only war level" didn't either, but I'll see what I find now that I know more of the process).
I think that I am quite aware of the different possibilities of PARTEN_LAST by now. These several options are there for a reason of course. To find out what the culprit is, I reckon I'll have to create a small project and let it use this new library and see how it goes.
If it were only for ORO I probably go with just the workaround solution. But we are planning to moving from Struts 1.0.2 to Struts 1.3.8 (no, not 2.0) and likely we will hit some lib/classpath problems there too. Anyway, I will try a couple of scenario's and suggestions of you two to see how it goes (basically, the suggestion of "everywhere" didn't work out-of-the box and the suggestion of "only war level" didn't either, but I'll see what I find now that I know more of the process).
ASKER
My apologies for not reacting sooner (well, I did react, but I didn't follow up either). I have been moved to another assignment. The solution to this particular problem was and is using my own ClassLoader.
I still don't understand why the class loading strategy of IBM is the reverse of any other class loading strategy I have ever seen (IBM puts most specific, i.e., your own classes, by default at the end of the path), I can't seem to find any use case for that though I'm sure IBM has thought it over (well, there must be someone who found this more useful than other strategies).
Anyway, what shall we do about the points? I can share them, of course, but in the end, I feel that I have actually answered my own question, but I don't know whether I can accept my own question (actually, I think I can't).
Cheers,
-- Abel --
I still don't understand why the class loading strategy of IBM is the reverse of any other class loading strategy I have ever seen (IBM puts most specific, i.e., your own classes, by default at the end of the path), I can't seem to find any use case for that though I'm sure IBM has thought it over (well, there must be someone who found this more useful than other strategies).
Anyway, what shall we do about the points? I can share them, of course, but in the end, I feel that I have actually answered my own question, but I don't know whether I can accept my own question (actually, I think I can't).
Cheers,
-- Abel --
The decision to accept one or more answers, or split the points is entirely up to you.
If you believe that none of the responses warrant acceptance, then you can open a separate question in the General Community Support area, and request a refund.
If you believe that none of the responses warrant acceptance, then you can open a separate question in the General Community Support area, and request a refund.
ASKER
Well, doing this pragmatically, I'll split the points equally to everybody and I'll post a note about the solution.
ASKER
Excellent for the helpful pointers. Though they didn't result in a complete solution, they did help me understand the classpath problems, solutions and strategies available from IBM WS.
The final solution to this particular situation (i.e., changing the load order is not an option as a result of other existing dependencies) is to add your own classpath resolver *and* dynamically load the libs + classes. This has the disadvantage of having no compile time checks anymore, so good error handling, and solid classpath searching (i.e., recursively searching from the WEB-INF dir in my case instead of from a fixed or programmable location) is vital.
After you have a working prototype, encapsulate it using for instance the Facade pattern (depending on your intents, a decorator, or even a proxy may be more appropriate instead).
The final solution to this particular situation (i.e., changing the load order is not an option as a result of other existing dependencies) is to add your own classpath resolver *and* dynamically load the libs + classes. This has the disadvantage of having no compile time checks anymore, so good error handling, and solid classpath searching (i.e., recursively searching from the WEB-INF dir in my case instead of from a fixed or programmable location) is vital.
After you have a working prototype, encapsulate it using for instance the Facade pattern (depending on your intents, a decorator, or even a proxy may be more appropriate instead).
ASKER
hmm, interesting to find that the grading comments show up on all places where I graded... ;)
thanks people for helping on this rather complex issue!
thanks people for helping on this rather complex issue!
interesting indeed.
Thanks for the points. Good luck
Thanks for the points. Good luck
ASKER
It's probably because of the way the system has been build. It doesn't really help either that the EAR hosts several projects that are interdependent (the legacy of indoor development...).
I did find one workaround, though it feels like cracking a walnut using a sledgehammer: using my own classloader that loops through the jars in the WEB-INF/lib directory. Because of the conflicts I cannot use the objects directly, so I added reflection and introspection (using java reflect api). And as it turns out, it works like a charm. Luckily I didn't need it in a performance sensitive part.
Cheers,
-- Abel --