Solved

Markview Application needs to have its process restarted every monday

Posted on 2014-07-21
12
542 Views
Last Modified: 2014-07-25
Dear Experts,

I am troubleshooting an application that works with Oracle. It is called Markview 170 systems.

In the Redhat linux server, I have to restart the process that I see listed in crontabs we'll call it xdev. This happens every Monday.

This is done so the user can open images scan in the application

Where do I begin troubleshooting?

So far I get this:

Server 4 that has to be restarted

ps -ef | grep -i $USER | grep -i xdev

mvxpress 11664     1  0 12:43 pts/1    00:00:07 java -Xincgc -Xms1g -Xmx1g -Xss1m -Djava.awt.headless=true -DMvDataSourceLocation=jdbc/OracleDS -XX:NewSize=256m -XX:MaxNewSize=512m -Dmv.log.dir=log -jar oc4j.jar -config config-mvas-vue/server.xml -out log/mvas-vue.xdev.out -err log/mvas-vue.err
mvxpress 13657 18305  0 13:17 pts/0    00:00:00 grep -i xdev


This is from server 3 that does not need to be restarted:

ps -ef | grep -i $USER | grep -i xdev  
                                                           
ps -ef | grep -i $USER | grep -i xdev
mvasdocs 23595     1  0 12:43 pts/1    00:00:02 java -Djava.awt.headless=true -jar oc4j.jar -config config-mvas-vm/server.xml -out log/mvas-vm.xdev.out -err log/mvas-vm.err
mvasdocs 23600     1  0 12:43 pts/1    00:00:02 java -Djava.awt.headless=true -jar oc4j.jar -config config-graphics/server.xml -out log/graphics.xdev.out -err log/graphics.err
mvasdocs 23605     1  0 12:43 pts/1    00:00:02 java -Djava.awt.headless=true -jar oc4j.jar -config config-mvas-bcg/server.xml -out log/mvas-bcg.xdev.out -err log/mvas-bcg.err
mvasdocs 23618     1  0 12:43 pts/1    00:00:04 java -Djava.awt.headless=true -jar oc4j.jar -config config-mvas-dtm/server.xml -out log/mvas-dtm.xdev.out -err log/mvas-dtm.err
mvasdocs 23639     1  0 12:43 pts/1    00:00:04 java -Djava.awt.headless=true -jar oc4j.jar -config config-sfas-mm/server.xml -out ./log/sfas-mm.xdev.out -err ./log/sfas-mm.err

For any tips or solution on how to take on this I thank you!

M
0
Comment
Question by:marceloNYC
  • 7
  • 5
12 Comments
 
LVL 37

Expert Comment

by:Gerwin Jansen
ID: 40209852
Your server 4 has less processes running than server 3:

- is server 4 normally running 4 processes?
- can you have a look at the log folder, it contains several .err files, look at the end of those files, anything worth mentioning?
- how do you restart the processes? which commands do you have to give?

What 'client' applications do your users use? What kind (if any) error messages do they get? Can you post them>

Is server 4 experiencing high CPU load that you know of during certain periods? Do you have a system manager available that can assist you with this?
0
 

Author Comment

by:marceloNYC
ID: 40209898
Nice! I will collect this information and get back to you.
0
 

Author Comment

by:marceloNYC
ID: 40209983
Server 4 that has to be restart to fix the problem runs the express version not the complete version of the software.

I am attaching the script that either start or stops the ps.

That scripts calls for (not too sure) this process:

bbExecuteScript $Parm1-mvas-vm.sh                                     >> $bbOc4jDbug
    bbExecuteScript $Parm1-mvas-vm2.sh                                    >> $bbOc4jDbug
    bbExecuteScript $Parm1-mvas-vm3.sh                                    >> $bbOc4jDbug
    bbExecuteScript $Parm1-mvas-vue.sh                                    >> $bbOc4jDbug
    bbExecuteScript $Parm1-graphics.sh                                    >> $bbOc4jDbug
    bbExecuteScript $Parm1-mvas-bcg.sh                                    >> $bbOc4jDbug
    bbExecuteScript $Parm1-mvas-dtm.sh                                    >> $bbOc4jDbug
#   bbExecuteScript $Parm1-mvas-vm-monitor.sh                             >> $bbOc4jDbug
    bbExecuteScript $Parm1-sfas-mm.sh                                     >> $bbOc4jDbug


This is one log error with information for today:

4/07/21 12:08:12 OrionCMTConnection not closed, check your code!
14/07/21 12:08:12 Logical connection not closed, check your code!
14/07/21 12:08:12 (Use -Djdbc.connection.debug=true to find out where the leaked connection was created)
14/07/21 12:43:53 OrionCMTConnection not closed, check your code!
14/07/21 12:43:53 Logical connection not closed, check your code!
14/07/21 12:43:53 (Use -Djdbc.connection.debug=true to find out where the leaked connection was created)

from another .err log:

14/07/21 12:41:25 Warning: Error reading transaction-log file (/markview/mvxpress/oc4j_xdev/j2ee/home/persistence/transaction-mvas-vue.state) for recovery: premature end of file
14/07/21 12:41:25 Forced or abrupt (crash etc) server shutdown detected, starting recovery process...
14/07/21 12:41:25 Recovery completed, 0 connections committed and 0 rolled back...
14/07/21 12:41:35 *** (CRITICAL) Mon Jul 21 12:41:35 CDT 2014 [Thread[Orion Launcher,5,main]]  java.lang.InstantiationException: jms.xml: port 9465 already in use.: java.net.BindException: Address already in use
      at java.net.PlainSocketImpl.socketBind(Native Method)
      at java.net.PlainSocketImpl.bind(PlainSocketImpl.java:331)
      at java.net.ServerSocket.bind(ServerSocket.java:318)
      at java.net.ServerSocket.<init>(ServerSocket.java:185)
      at com.evermind.server.Server.createServerSocket(Server.java:165)
      at com.evermind.server.jms.JMSServer.initializeSocket(JMSServer.java:1143)
      at com.evermind.server.jms.JMSServer.setConfig(JMSServer.java:180)
      at com.evermind.server.ApplicationServer.initializeJMS(ApplicationServer.java:2363)
      at com.evermind.server.ApplicationServer.setConfig(ApplicationServer.java:1458)
      at com.evermind.server.ApplicationServerLauncher.run(ApplicationServerLauncher.java:92)
      at java.lang.Thread.run(Thread.java:534)

14/07/21 12:41:35 Error starting JMS-Server: jms.xml: port 9465 already in use.: java.net.BindException: Address already in use
      at java.net.PlainSocketImpl.socketBind(Native Method)
      at java.net.PlainSocketImpl.bind(PlainSocketImpl.java:331)
      at java.net.ServerSocket.bind(ServerSocket.java:318)
      at java.net.ServerSocket.<init>(ServerSocket.java:185)
      at com.evermind.server.Server.createServerSocket(Server.java:165)
      at com.evermind.server.jms.JMSServer.initializeSocket(JMSServer.java:1143)
      at com.evermind.server.jms.JMSServer.setConfig(JMSServer.java:180)
      at com.evermind.server.ApplicationServer.initializeJMS(ApplicationServer.java:2363)
      at com.evermind.server.ApplicationServer.setConfig(ApplicationServer.java:1458)
      at com.evermind.server.ApplicationServerLauncher.run(ApplicationServerLauncher.java:92)
      at java.lang.Thread.run(Thread.java:534)

14/07/21 12:41:35 Error starting ORMI-Server.  Unable to bind socket: Address already in use
14/07/21 12:42:09 Could not find mediaLib accelerator wrapper classes. Continuing in pure Java mode.
14/07/21 12:42:48 Warning: Error deleting the transaction-log file
14/07/21 12:44:33 Could not find mediaLib accelerator wrapper classes. Continuing in pure Java mode.
bbStartStop.sh
0
 
LVL 37

Accepted Solution

by:
Gerwin Jansen earned 500 total points
ID: 40210233
Looks like the application itself is detecting that something is wrong (14/07/21 12:41:25 Forced or abrupt ...) and tries a restart. But there it fails as it is trying to start a process for which a port is already in use (9465).

Do you know why this line has been commented out in the script:

#   bbExecuteScript $Parm1-mvas-vm-monitor.sh                             >> $bbOc4jDbug

... monitor should be monitoring things, but it is not started?

Any recent changes to this marview application?

Did you try a stop and start like this (with the monitor enabled):

bbStartStop.sh stop
bbStartStop.sh start

Did  you have a look at $OC4J_HOME_XDEV/log/bbStartStop.log
0
 

Author Comment

by:marceloNYC
ID: 40211621
This is from yesterday's log information for bbStartStop.log


*************************************************************
*** 07/21/14 12:43:16 PM start
*** /markview/mvxpress/oc4j_xdev/j2ee/home
*************************************************************
bbWARNING: 'start-mvas-vm.sh' is not found OR is not hosted on this server.
bbWARNING: 'start-mvas-vm2.sh' is not found OR is not hosted on this server.
bbWARNING: 'start-mvas-vm3.sh' is not found OR is not hosted on this server.
170 MarkView Application Server Viewer Module started at Mon Jul 21 12:43:16 CDT 2014
bbWARNING: 'start-graphics.sh' is not found OR is not hosted on this server.
bbWARNING: 'start-mvas-bcg.sh' is not found OR is not hosted on this server.
bbWARNING: 'start-mvas-dtm.sh' is not found OR is not hosted on this server.
bbWARNING: 'start-sfas-mm.sh' is not found OR is not hosted on this server.
mvxpress 11664     1 39 12:43 pts/1    00:00:01 java -Xincgc -Xms1g -Xmx1g -Xss1m -Djava.awt.headless=true -DMvDataSourceLocation=jdbc/OracleDS -XX:NewSize=256m -XX:MaxNewSize=512m -Dmv.log.dir=log -jar oc4j.jar -config config-mvas-vue/server.xml -out log/mvas-vue.xdev.out -err log/mvas-vue.err
0
 
LVL 37

Expert Comment

by:Gerwin Jansen
ID: 40212326
Looks like only one of the servers/processes is supposed to run on server 4.

Can you copy the existing bbStartStop.sh script to bbStartStop.sh.backup using:

cp -p bbStartStop.sh bbStartStop.sh.backup

and then edit bbStartStop.sh to exclude all but the one that is supposed to run, like this:
# bbExecuteScript $Parm1-mvas-vm.sh                                     >> $bbOc4jDbug
# bbExecuteScript $Parm1-mvas-vm2.sh                                    >> $bbOc4jDbug
# bbExecuteScript $Parm1-mvas-vm3.sh                                    >> $bbOc4jDbug
bbExecuteScript $Parm1-mvas-vue.sh                                    >> $bbOc4jDbug
# bbExecuteScript $Parm1-graphics.sh                                    >> $bbOc4jDbug
# bbExecuteScript $Parm1-mvas-bcg.sh                                    >> $bbOc4jDbug
# bbExecuteScript $Parm1-mvas-dtm.sh                                    >> $bbOc4jDbug
# bbExecuteScript $Parm1-mvas-vm-monitor.sh                             >> $bbOc4jDbug
# bbExecuteScript $Parm1-sfas-mm.sh                                     >> $bbOc4jDbug

Open in new window


Then stop / start again, look at the $bbOc4jDbug log and see if things are OK now?
0
Complete Microsoft Windows PC® & Mac Backup

Backup and recovery solutions to protect all your PCs & Mac– on-premises or in remote locations. Acronis backs up entire PC or Mac with patented reliable disk imaging technology and you will be able to restore workstations to a new, dissimilar hardware in minutes.

 

Author Comment

by:marceloNYC
ID: 40214113
Okay, I will need a little time.
0
 

Author Comment

by:marceloNYC
ID: 40214865
Hello Gerwin Jansen,

Okay, here is an update. I think that we are almost there.

We found a different port that should be free. We think that port 9465 is used by another application and was enter in MarkView as the wrong port when updated back in January.

So we are going to reinstall the app with a proper free port.  We think that fixes the issue.

Do you want closed the question now or rather wait to see how it goes?  

Thank you for your help.
0
 
LVL 37

Expert Comment

by:Gerwin Jansen
ID: 40215210
Thank you for your help.
Thank you :D

>> Do you want closed the question now or rather wait to see how it goes?
Well, that' up to you to be honest ;)

We'd like to know you've got it fixed. sure. But since there is a port conflict and that's what you're going to fix, the solution in itself is clear enough. If reinstalling takes a few days, just leave this question open, no problem.
0
 

Author Closing Comment

by:marceloNYC
ID: 40215239
Your help was great because I am new in the job. So when I pointed to my boss our the product of our discussion, he new the port is the issue, he has the experience with this installation.  I will come back to this post to confirm or let you know if anything else happens.

Regards!
0
 
LVL 37

Expert Comment

by:Gerwin Jansen
ID: 40215332
Glad we could help you out, thanks!
0
 

Author Comment

by:marceloNYC
ID: 40219490
By the way, I think that you are right on the issue with the script.
0

Featured Post

Maximize Your Threat Intelligence Reporting

Reporting is one of the most important and least talked about aspects of a world-class threat intelligence program. Here’s how to do it right.

Join & Write a Comment

Suggested Solutions

Cursors in Oracle: A cursor is used to process individual rows returned by database system for a query. In oracle every SQL statement executed by the oracle server has a private area. This area contains information about the SQL statement and the…
I. Introduction There's an interesting discussion going on now in an Experts Exchange Group — Attachments with no extension (http://www.experts-exchange.com/discussions/210281/Attachments-with-no-extension.html). This reminded me of questions tha…
This video shows how to copy a database user from one database to another user DBMS_METADATA.  It also shows how to copy a user's permissions and discusses password hash differences between Oracle 10g and 11g.
This video shows information on the Oracle Data Dictionary, starting with the Oracle documentation, explaining the different types of Data Dictionary views available by group and permissions as well as giving examples on how to retrieve data from th…

707 members asked questions and received personalized solutions in the past 7 days.

Join the community of 500,000 technology professionals and ask your questions.

Join & Ask a Question

Need Help in Real-Time?

Connect with top rated Experts

18 Experts available now in Live!

Get 1:1 Help Now