Solved

Classpath problems with IBM App Srv: Jython & ORO conflicts

Posted on 2007-12-04
21
449 Views
Last Modified: 2013-12-29
Hi Experts,

this seems to be a classic Java/IWebphere problem, considering the amount of Google hits I get about classpath issues with IBM Websphere and Jython. Main story that precisely describes the problem I have: http://www.ibm.com/developerworks/websphere/techjournal/0406_brown/0406_brown.html (the suggestions on IBM's website do not suffice, unfortunately.)

The issue: I need to use a newer ORO library from Apache Commons, namely Jakarta Oro 2.0.8. However, IBM comes bundled with a ton of libraries and preferably it loads these libraries first (PARENT_FIRST). This means that if you have any newer jar file on the lib path of your application (a WAR in my situation) you can be certain it will never be loaded.

According to the aforementioned link, Java will never "see" my newer version. To change that I should set the classloader behavior to "PARENT_LAST" (either on server, app or module level). Unfortunately this raises a ton of new issues which seem hard to solve (many incompatibilities).

I also tried adding it to the MANIFEST.MF, but that wouldn't work because IBM first loads its own classes and then the ones in the MANIFEST, which in turn must be part of the EAR.

When I renamed jython.jar to jython.old, my NoSuchMethodError went away. The reason that my development environment (Rational Rose XDE) does not complain is because it uses its own classpath for the application, hence the error only appears after deployment.

I do not have access to the deployment servers and thus I cannot disable jython.jar by hand. It has to be done using the server interface.

The question is simple, but possibly hard (?) to answer: how can I force the loading of the new ORO lib or how can I force Websphere to stop loading jython.jar. The means I have are:

 1. change the configuration of the WAR or EAR
 2. change the configuration of the server settings
 3. programming (I have full access to applicable sources)

Anybody any idea? Any help's appreciated,

Cheers,
-- Abel --
0
Comment
Question by:abel
  • 11
  • 7
  • 3
21 Comments
 
LVL 15

Accepted Solution

by:
bpmurray earned 250 total points
Comment Utility
First, PARENT_LAST is important, but has to be both on the WAR *AND* the EAR. My guess is that you've only done this in one place.

Next, make sure that no spurious jars have been added to your project, i.e. that'll be sucked into your deployed app to override what you expect.

That should do the trick.
0
 
LVL 39

Author Comment

by:abel
Comment Utility
Well, that is precisely what has happened, these spurious jars. They come from the ancient IBM WSAS 5.1. I've tried your suggestion and some other combinations too, but none seem to help. It's not like a couple of errors that I could readily solve, but a whole lot of them, from different libraries, seem to be unresolvable when I change it.

It's probably because of the way the system has been build. It doesn't really help either that the EAR hosts several projects that are interdependent (the legacy of indoor development...).

I did find one workaround, though it feels like cracking a walnut using a sledgehammer: using my own classloader that loops through the jars in the WEB-INF/lib directory. Because of the conflicts I cannot use the objects directly, so I added reflection and introspection (using java reflect api). And as it turns out, it works like a charm. Luckily I didn't need it in a performance sensitive part.

Cheers,
-- Abel --
0
 
LVL 39

Author Comment

by:abel
Comment Utility
Hmm, when I think about it, with .NET or plain old (not so good) COM+ it was possible to use the version information in the dll to call the correct one (or when not using version information it called the latest version). Isn't there a mechanism somehow with Java that does something similar? That would give the possibility to mix and match older libs that are dependent on other older libs, together with more recent libs that you need to use yourself.

(example, again: Jython is dependent on an old ORO, and you want the new ORO *and* Jython).
0
 
LVL 41

Assisted Solution

by:HonorGod
HonorGod earned 250 total points
Comment Utility
Since you have iterated through the jar files in the WEB-INF/lib directory, you should now know which needs to be referenced.

Can you keep this information across invocations in order to remove the lookup penalty?

By the way, interesting solution to the problem.
0
 
LVL 39

Author Comment

by:abel
Comment Utility
Well, yes, I know which specific library needs to be referenced, I just generalized the solution. The lookup is done once (singleton) and a reference to the needed classes is then kept in memory.

It feels a bit as a large way around to get a specific class from a specific jar file. Version conflicts can happen more often (in situations you are stuck with a default set of libraries and you cannot touch this default set, however, you need the new versions of these libs), why is it so hard to choose a specific lib, or, put differently, a lib with a specific version in the Manifest file. I mean the information is all there, it shouldn't be too hard should it?
0
 
LVL 41

Assisted Solution

by:HonorGod
HonorGod earned 250 total points
Comment Utility
That is certainly a reasonable question.
However, another question, from a different perspective might be:

Why do we have to have multiple versions of the "same" jar file?
Wouldn't it be better if newer versions of a jar file provided backward compatability?
0
 
LVL 39

Author Comment

by:abel
Comment Utility
Yes, of course. And they do. This one too. However, I needed added functionality (new methods in the new version). And the classpath system of IBM WSAS was "bottom up", i.e., it first looks at Java, then at IBM's classpath and lastly at you own added libs. Where the default should be the other way around.

You don't add libs for them to not be used or found.

So, actually what we are talking of here is forward compatibility. IBM finds the old jar first and sees it as the "right" jar (however in my dev IDE, the normal way of "most specific first" was followed and it finds the added libs correctly, but at runtime, the IBM server takes over and uses this backward method), instead of the new one. And forward compatibility is, by its nature, limited.
0
 
LVL 41

Assisted Solution

by:HonorGod
HonorGod earned 250 total points
Comment Utility
Is your code for doing the lookup,  reflection, and introspection something that you would be willing to share here?
0
 
LVL 15

Assisted Solution

by:bpmurray
bpmurray earned 250 total points
Comment Utility
This is exactly why the PARENT_LAST/PARENT_FIRST settings are available, but they must be set all over. If you have only set this in one part of your app, it just has no effect.
0
 
LVL 41

Assisted Solution

by:HonorGod
HonorGod earned 250 total points
Comment Utility
Well, that's not completely accurate.
The place where the setting exists identifies the scope of the setting.
In general, you probably want to make sure to set the classloader at
the WAR level, so that the appropriate setting is used.

There can be instances of the application classloader setting causing
problems.  For example, if you have static methods in the EAR, and
these get invoked by the application, that can cause the Application
classloader setting to take precidence, and keep you application from
thereafter accessing resources in the WAR.  Unfortunately, the
classloader stuff is sometimes counter intuitive. :-(
0
Top 6 Sources for Identifying Threat Actor TTPs

Understanding your enemy is essential. These six sources will help you identify the most popular threat actor tactics, techniques, and procedures (TTPs).

 
LVL 39

Author Comment

by:abel
Comment Utility
> If you have only set this in one part of your app, it just has no effect.

See my earlier comments: it costs too many other problems. I assume that when the system was designed like this from the ground up, there weren't any problems at all. But this is an age old system (hence Struts 1.0.2, WSAS 5.1 with an old Jython and Java 1.4 etc) which has been build and changed over the last years.

Though I do think that changing this order should not really cause "class not found" errors, as the only location on the classpath is further away, and nearer libraries are newer and shouldn't give "class not found" errors. Hmm, maybe I should look into that solution again and try to analyse some of the errors. However, the classes not found were hundredths, which is why I didn't bother to look further and I changed it back to "normal" (why IBM thinks this is normal is beyond me).
0
 
LVL 39

Author Comment

by:abel
Comment Utility
> Unfortunately, the classloader stuff is sometimes counter intuitive. :-(

that's an understatement. I've programmed in Java for years, and Java always sticked with me easily, even if I stopped programming for some time. But classpath issues had the tendency to always come up and once there, were hard to solve.

Yes, I think I can share that code. However, it is quite straightforward. I'm not in the office today, so bear with me a while.
0
 
LVL 15

Assisted Solution

by:bpmurray
bpmurray earned 250 total points
Comment Utility
The idea behind the PARENT_LAST solution is that it means that the jars in your app are used first, and these tend to be the ones you want. That's why I originally suggested to watch out for other jars in your project that you actually don't want. If your app works in WSAD/RAD and it fails in WAS, it should revert to the dev behaviour by setting this, but you have to set it everywhere!
0
 
LVL 41

Assisted Solution

by:HonorGod
HonorGod earned 250 total points
Comment Utility
By setting it everywhere you are removing any question about the classloader strategy to be used.  Actually, it really only needs to be set in "the right place".  Unfortunately, understanding which places are "necessary and sufficient" is the challenge.  :-(
0
 
LVL 39

Author Comment

by:abel
Comment Utility
I haven't been clear about my dev env, it isn't WSAD, but XDE from Rational Rose, which is built on top of Eclipse 2.1. To add to the confusion, the dev env uses Java 1.3 and WSAS (WAS?) uses Java 1.4. But that's something I can configure differently.

I think that I am quite aware of the different possibilities of PARTEN_LAST by now. These several options are there for a reason of course. To find out what the culprit is, I reckon I'll have to create a small project and let it use this new library and see how it goes.

If it were only for ORO I probably go with just the workaround solution. But we are planning to moving from Struts 1.0.2 to Struts 1.3.8 (no, not 2.0) and likely we will hit some lib/classpath problems there too. Anyway, I will try a couple of scenario's and suggestions of you two to see how it goes (basically, the suggestion of "everywhere" didn't work out-of-the box and the suggestion of "only war level" didn't either, but I'll see what I find now that I know more of the process).
0
 
LVL 39

Author Comment

by:abel
Comment Utility
My apologies for not reacting sooner (well, I did react, but I didn't follow up either). I have been moved to another assignment. The solution to this particular problem was and is using my own ClassLoader.

I still don't understand why the class loading strategy of IBM is the reverse of any other class loading strategy I have ever seen (IBM puts most specific, i.e., your own classes, by default at the end of the path), I can't seem to find any use case for that though I'm sure IBM has thought it over (well, there must be someone who found this more useful than other strategies).

Anyway, what shall we do about the points? I can share them, of course, but in the end, I feel that I have actually answered my own question, but I don't know whether I can accept my own question (actually, I think I can't).

Cheers,
-- Abel --
0
 
LVL 41

Expert Comment

by:HonorGod
Comment Utility
The decision to accept one or more answers, or split the points is entirely up to you.
If you believe that none of the responses warrant acceptance, then you can open a separate question in the General Community Support area, and request a refund.
0
 
LVL 39

Author Comment

by:abel
Comment Utility
Well, doing this pragmatically, I'll split the points equally to everybody and I'll post a note about the solution.
0
 
LVL 39

Author Closing Comment

by:abel
Comment Utility
Excellent for the helpful pointers. Though they didn't result in a complete solution, they did help me understand the classpath problems, solutions and strategies available from IBM WS.

The final solution to this particular situation (i.e., changing the load order is not an option as a result of other existing dependencies) is to add your own classpath resolver *and* dynamically load the libs + classes. This has the disadvantage of having no compile time checks anymore, so good error handling, and solid classpath searching (i.e., recursively searching from the WEB-INF dir in my case instead of from a fixed or programmable location) is vital.

After you have a working prototype, encapsulate it using for instance the Facade pattern (depending on your intents, a decorator, or even a proxy may be more appropriate instead).
0
 
LVL 39

Author Comment

by:abel
Comment Utility
hmm, interesting to find that the grading comments show up on all places where I graded... ;)

thanks people for helping on this rather complex issue!
0
 
LVL 41

Expert Comment

by:HonorGod
Comment Utility
interesting indeed.

Thanks for the points.  Good luck
0

Featured Post

Why You Should Analyze Threat Actor TTPs

After years of analyzing threat actor behavior, it’s become clear that at any given time there are specific tactics, techniques, and procedures (TTPs) that are particularly prevalent. By analyzing and understanding these TTPs, you can dramatically enhance your security program.

Join & Write a Comment

This article is about some of the basic and important steps to be used to improve the performance in web-sphere commerce application development. 1) Always leverage the Dyna-caching facility provided by the product 2) Remove the unwanted code …
Java functions are among the best things for programmers to work with as Java sites can be very easy to read and prepare. Java especially simplifies many processes in the coding industry as it helps integrate many forms of technology and different d…
Viewers learn about the scanner class in this video and are introduced to receiving user input for their programs. Additionally, objects, conditional statements, and loops are used to help reinforce the concepts. Introduce Scanner class: Importing…
Viewers learn how to read error messages and identify possible mistakes that could cause hours of frustration. Coding is as much about debugging your code as it is about writing it. Define Error Message: Line Numbers: Type of Error: Break Down…

762 members asked questions and received personalized solutions in the past 7 days.

Join the community of 500,000 technology professionals and ask your questions.

Join & Ask a Question

Need Help in Real-Time?

Connect with top rated Experts

10 Experts available now in Live!

Get 1:1 Help Now