Solved

Markview Application needs to have its process restarted every monday

Posted on 2014-07-21
12
565 Views
Last Modified: 2014-07-25
Dear Experts,

I am troubleshooting an application that works with Oracle. It is called Markview 170 systems.

In the Redhat linux server, I have to restart the process that I see listed in crontabs we'll call it xdev. This happens every Monday.

This is done so the user can open images scan in the application

Where do I begin troubleshooting?

So far I get this:

Server 4 that has to be restarted

ps -ef | grep -i $USER | grep -i xdev

mvxpress 11664     1  0 12:43 pts/1    00:00:07 java -Xincgc -Xms1g -Xmx1g -Xss1m -Djava.awt.headless=true -DMvDataSourceLocation=jdbc/OracleDS -XX:NewSize=256m -XX:MaxNewSize=512m -Dmv.log.dir=log -jar oc4j.jar -config config-mvas-vue/server.xml -out log/mvas-vue.xdev.out -err log/mvas-vue.err
mvxpress 13657 18305  0 13:17 pts/0    00:00:00 grep -i xdev


This is from server 3 that does not need to be restarted:

ps -ef | grep -i $USER | grep -i xdev  
                                                           
ps -ef | grep -i $USER | grep -i xdev
mvasdocs 23595     1  0 12:43 pts/1    00:00:02 java -Djava.awt.headless=true -jar oc4j.jar -config config-mvas-vm/server.xml -out log/mvas-vm.xdev.out -err log/mvas-vm.err
mvasdocs 23600     1  0 12:43 pts/1    00:00:02 java -Djava.awt.headless=true -jar oc4j.jar -config config-graphics/server.xml -out log/graphics.xdev.out -err log/graphics.err
mvasdocs 23605     1  0 12:43 pts/1    00:00:02 java -Djava.awt.headless=true -jar oc4j.jar -config config-mvas-bcg/server.xml -out log/mvas-bcg.xdev.out -err log/mvas-bcg.err
mvasdocs 23618     1  0 12:43 pts/1    00:00:04 java -Djava.awt.headless=true -jar oc4j.jar -config config-mvas-dtm/server.xml -out log/mvas-dtm.xdev.out -err log/mvas-dtm.err
mvasdocs 23639     1  0 12:43 pts/1    00:00:04 java -Djava.awt.headless=true -jar oc4j.jar -config config-sfas-mm/server.xml -out ./log/sfas-mm.xdev.out -err ./log/sfas-mm.err

For any tips or solution on how to take on this I thank you!

M
0
Comment
Question by:marceloNYC
  • 7
  • 5
12 Comments
 
LVL 38

Expert Comment

by:Gerwin Jansen, EE MVE
ID: 40209852
Your server 4 has less processes running than server 3:

- is server 4 normally running 4 processes?
- can you have a look at the log folder, it contains several .err files, look at the end of those files, anything worth mentioning?
- how do you restart the processes? which commands do you have to give?

What 'client' applications do your users use? What kind (if any) error messages do they get? Can you post them>

Is server 4 experiencing high CPU load that you know of during certain periods? Do you have a system manager available that can assist you with this?
0
 

Author Comment

by:marceloNYC
ID: 40209898
Nice! I will collect this information and get back to you.
0
 

Author Comment

by:marceloNYC
ID: 40209983
Server 4 that has to be restart to fix the problem runs the express version not the complete version of the software.

I am attaching the script that either start or stops the ps.

That scripts calls for (not too sure) this process:

bbExecuteScript $Parm1-mvas-vm.sh                                     >> $bbOc4jDbug
    bbExecuteScript $Parm1-mvas-vm2.sh                                    >> $bbOc4jDbug
    bbExecuteScript $Parm1-mvas-vm3.sh                                    >> $bbOc4jDbug
    bbExecuteScript $Parm1-mvas-vue.sh                                    >> $bbOc4jDbug
    bbExecuteScript $Parm1-graphics.sh                                    >> $bbOc4jDbug
    bbExecuteScript $Parm1-mvas-bcg.sh                                    >> $bbOc4jDbug
    bbExecuteScript $Parm1-mvas-dtm.sh                                    >> $bbOc4jDbug
#   bbExecuteScript $Parm1-mvas-vm-monitor.sh                             >> $bbOc4jDbug
    bbExecuteScript $Parm1-sfas-mm.sh                                     >> $bbOc4jDbug


This is one log error with information for today:

4/07/21 12:08:12 OrionCMTConnection not closed, check your code!
14/07/21 12:08:12 Logical connection not closed, check your code!
14/07/21 12:08:12 (Use -Djdbc.connection.debug=true to find out where the leaked connection was created)
14/07/21 12:43:53 OrionCMTConnection not closed, check your code!
14/07/21 12:43:53 Logical connection not closed, check your code!
14/07/21 12:43:53 (Use -Djdbc.connection.debug=true to find out where the leaked connection was created)

from another .err log:

14/07/21 12:41:25 Warning: Error reading transaction-log file (/markview/mvxpress/oc4j_xdev/j2ee/home/persistence/transaction-mvas-vue.state) for recovery: premature end of file
14/07/21 12:41:25 Forced or abrupt (crash etc) server shutdown detected, starting recovery process...
14/07/21 12:41:25 Recovery completed, 0 connections committed and 0 rolled back...
14/07/21 12:41:35 *** (CRITICAL) Mon Jul 21 12:41:35 CDT 2014 [Thread[Orion Launcher,5,main]]  java.lang.InstantiationException: jms.xml: port 9465 already in use.: java.net.BindException: Address already in use
      at java.net.PlainSocketImpl.socketBind(Native Method)
      at java.net.PlainSocketImpl.bind(PlainSocketImpl.java:331)
      at java.net.ServerSocket.bind(ServerSocket.java:318)
      at java.net.ServerSocket.<init>(ServerSocket.java:185)
      at com.evermind.server.Server.createServerSocket(Server.java:165)
      at com.evermind.server.jms.JMSServer.initializeSocket(JMSServer.java:1143)
      at com.evermind.server.jms.JMSServer.setConfig(JMSServer.java:180)
      at com.evermind.server.ApplicationServer.initializeJMS(ApplicationServer.java:2363)
      at com.evermind.server.ApplicationServer.setConfig(ApplicationServer.java:1458)
      at com.evermind.server.ApplicationServerLauncher.run(ApplicationServerLauncher.java:92)
      at java.lang.Thread.run(Thread.java:534)

14/07/21 12:41:35 Error starting JMS-Server: jms.xml: port 9465 already in use.: java.net.BindException: Address already in use
      at java.net.PlainSocketImpl.socketBind(Native Method)
      at java.net.PlainSocketImpl.bind(PlainSocketImpl.java:331)
      at java.net.ServerSocket.bind(ServerSocket.java:318)
      at java.net.ServerSocket.<init>(ServerSocket.java:185)
      at com.evermind.server.Server.createServerSocket(Server.java:165)
      at com.evermind.server.jms.JMSServer.initializeSocket(JMSServer.java:1143)
      at com.evermind.server.jms.JMSServer.setConfig(JMSServer.java:180)
      at com.evermind.server.ApplicationServer.initializeJMS(ApplicationServer.java:2363)
      at com.evermind.server.ApplicationServer.setConfig(ApplicationServer.java:1458)
      at com.evermind.server.ApplicationServerLauncher.run(ApplicationServerLauncher.java:92)
      at java.lang.Thread.run(Thread.java:534)

14/07/21 12:41:35 Error starting ORMI-Server.  Unable to bind socket: Address already in use
14/07/21 12:42:09 Could not find mediaLib accelerator wrapper classes. Continuing in pure Java mode.
14/07/21 12:42:48 Warning: Error deleting the transaction-log file
14/07/21 12:44:33 Could not find mediaLib accelerator wrapper classes. Continuing in pure Java mode.
bbStartStop.sh
0
PRTG Network Monitor: Intuitive Network Monitoring

Network Monitoring is essential to ensure that computer systems and network devices are running. Use PRTG to monitor LANs, servers, websites, applications and devices, bandwidth, virtual environments, remote systems, IoT, and many more. PRTG is easy to set up & use.

 
LVL 38

Accepted Solution

by:
Gerwin Jansen, EE MVE earned 500 total points
ID: 40210233
Looks like the application itself is detecting that something is wrong (14/07/21 12:41:25 Forced or abrupt ...) and tries a restart. But there it fails as it is trying to start a process for which a port is already in use (9465).

Do you know why this line has been commented out in the script:

#   bbExecuteScript $Parm1-mvas-vm-monitor.sh                             >> $bbOc4jDbug

... monitor should be monitoring things, but it is not started?

Any recent changes to this marview application?

Did you try a stop and start like this (with the monitor enabled):

bbStartStop.sh stop
bbStartStop.sh start

Did  you have a look at $OC4J_HOME_XDEV/log/bbStartStop.log
0
 

Author Comment

by:marceloNYC
ID: 40211621
This is from yesterday's log information for bbStartStop.log


*************************************************************
*** 07/21/14 12:43:16 PM start
*** /markview/mvxpress/oc4j_xdev/j2ee/home
*************************************************************
bbWARNING: 'start-mvas-vm.sh' is not found OR is not hosted on this server.
bbWARNING: 'start-mvas-vm2.sh' is not found OR is not hosted on this server.
bbWARNING: 'start-mvas-vm3.sh' is not found OR is not hosted on this server.
170 MarkView Application Server Viewer Module started at Mon Jul 21 12:43:16 CDT 2014
bbWARNING: 'start-graphics.sh' is not found OR is not hosted on this server.
bbWARNING: 'start-mvas-bcg.sh' is not found OR is not hosted on this server.
bbWARNING: 'start-mvas-dtm.sh' is not found OR is not hosted on this server.
bbWARNING: 'start-sfas-mm.sh' is not found OR is not hosted on this server.
mvxpress 11664     1 39 12:43 pts/1    00:00:01 java -Xincgc -Xms1g -Xmx1g -Xss1m -Djava.awt.headless=true -DMvDataSourceLocation=jdbc/OracleDS -XX:NewSize=256m -XX:MaxNewSize=512m -Dmv.log.dir=log -jar oc4j.jar -config config-mvas-vue/server.xml -out log/mvas-vue.xdev.out -err log/mvas-vue.err
0
 
LVL 38

Expert Comment

by:Gerwin Jansen, EE MVE
ID: 40212326
Looks like only one of the servers/processes is supposed to run on server 4.

Can you copy the existing bbStartStop.sh script to bbStartStop.sh.backup using:

cp -p bbStartStop.sh bbStartStop.sh.backup

and then edit bbStartStop.sh to exclude all but the one that is supposed to run, like this:
# bbExecuteScript $Parm1-mvas-vm.sh                                     >> $bbOc4jDbug
# bbExecuteScript $Parm1-mvas-vm2.sh                                    >> $bbOc4jDbug
# bbExecuteScript $Parm1-mvas-vm3.sh                                    >> $bbOc4jDbug
bbExecuteScript $Parm1-mvas-vue.sh                                    >> $bbOc4jDbug
# bbExecuteScript $Parm1-graphics.sh                                    >> $bbOc4jDbug
# bbExecuteScript $Parm1-mvas-bcg.sh                                    >> $bbOc4jDbug
# bbExecuteScript $Parm1-mvas-dtm.sh                                    >> $bbOc4jDbug
# bbExecuteScript $Parm1-mvas-vm-monitor.sh                             >> $bbOc4jDbug
# bbExecuteScript $Parm1-sfas-mm.sh                                     >> $bbOc4jDbug

Open in new window


Then stop / start again, look at the $bbOc4jDbug log and see if things are OK now?
0
 

Author Comment

by:marceloNYC
ID: 40214113
Okay, I will need a little time.
0
 

Author Comment

by:marceloNYC
ID: 40214865
Hello Gerwin Jansen,

Okay, here is an update. I think that we are almost there.

We found a different port that should be free. We think that port 9465 is used by another application and was enter in MarkView as the wrong port when updated back in January.

So we are going to reinstall the app with a proper free port.  We think that fixes the issue.

Do you want closed the question now or rather wait to see how it goes?  

Thank you for your help.
0
 
LVL 38

Expert Comment

by:Gerwin Jansen, EE MVE
ID: 40215210
Thank you for your help.
Thank you :D

>> Do you want closed the question now or rather wait to see how it goes?
Well, that' up to you to be honest ;)

We'd like to know you've got it fixed. sure. But since there is a port conflict and that's what you're going to fix, the solution in itself is clear enough. If reinstalling takes a few days, just leave this question open, no problem.
0
 

Author Closing Comment

by:marceloNYC
ID: 40215239
Your help was great because I am new in the job. So when I pointed to my boss our the product of our discussion, he new the port is the issue, he has the experience with this installation.  I will come back to this post to confirm or let you know if anything else happens.

Regards!
0
 
LVL 38

Expert Comment

by:Gerwin Jansen, EE MVE
ID: 40215332
Glad we could help you out, thanks!
0
 

Author Comment

by:marceloNYC
ID: 40219490
By the way, I think that you are right on the issue with the script.
0

Featured Post

What is SQL Server and how does it work?

The purpose of this paper is to provide you background on SQL Server. It’s your self-study guide for learning fundamentals. It includes both the history of SQL and its technical basics. Concepts and definitions will form the solid foundation of your future DBA expertise.

Question has a verified solution.

If you are experiencing a similar issue, please ask a related question

Suggested Solutions

Title # Comments Views Activity
Oracle function to insert records? 15 46
Linux FTP Error 5 32
IP 10.0.1.2 / 255.0.0.0 61 54
How to drop system generated virtual column in a table in12c 15 44
How to Create User-Defined Aggregates in Oracle Before we begin creating these things, what are user-defined aggregates?  They are a feature introduced in Oracle 9i that allows a developer to create his or her own functions like "SUM", "AVG", and…
I. Introduction There's an interesting discussion going on now in an Experts Exchange Group — Attachments with no extension (http://www.experts-exchange.com/discussions/210281/Attachments-with-no-extension.html). This reminded me of questions tha…
This video shows syntax for various backup options while discussing how the different basic backup types work.  It explains how to take full backups, incremental level 0 backups, incremental level 1 backups in both differential and cumulative mode a…
This video shows how to Export data from an Oracle database using the Datapump Export Utility.  The corresponding Datapump Import utility is also discussed and demonstrated.

789 members asked questions and received personalized solutions in the past 7 days.

Join the community of 500,000 technology professionals and ask your questions.

Join & Ask a Question