[Okta Webinar] Learn how to a build a cloud-first strategyRegister Now

x
?
Solved

Unrecoverable Error has occured

Posted on 2009-02-19
45
Medium Priority
?
700 Views
Last Modified: 2013-12-06
Hello  
We have a server that runs an old Linux-based application that we use for referencing data not inputted in our new system.  We stopped getting support for the software/machine about a year ago and it's been running fine.  Today when anyone tries to access the program, the error message "An unrecoverable error has occurred while initializing the system.  Error#12 File does not exist (or already exists) at Line 16."
Rebooting the machine does not help, and it seems that it boots up ok, however I have NO experience with Linux or really anything with the machine.  
0
Comment
Question by:Technical_Specialist
  • 15
  • 15
  • 13
  • +2
45 Comments
 
LVL 7

Expert Comment

by:Morne Lategan
ID: 23689225
In what language is the system written? Do you have access to the source file that is generating the error at Line 16?
0
 
LVL 19

Expert Comment

by:jools
ID: 23689717
how do your users access the program?

How old is the Linux distro and application? Got any names???
0
 
LVL 23

Expert Comment

by:Maciej S
ID: 23690262
The best is you can still boot successfully :)
What is that program you are using? How do you access it?
When exactly this error message ("An unrecoverable error has occurred while initializing the system.  Error#12 File does not exist (or already exists) at Line 16.") appears? Do you access this program with command line or some gui? If command line, please provide it with all arguments, if gui, try to describe your procedure, or give screenshots (or both :)).
Can you login to that server? If so, provide output of commands:
uname -a
df -h
free
ps aux

Anyway - message saying "error because something exists, or doesn't exist" is quite strange IMO :)
0
Veeam and MySQL: How to Perform Backup & Recovery

MySQL and the MariaDB variant are among the most used databases in Linux environments, and many critical applications support their data on them. Watch this recorded webinar to find out how Veeam Backup & Replication allows you to get consistent backups of MySQL databases.

 

Author Comment

by:Technical_Specialist
ID: 23693582
The users access it through a GUI.  After initially installing the program, setting up the host/port, they can click on the icon on their desktop to run the program.  They are prompted for their user/password, then are in.  With this issue, it does not even get to the user/password prompt.  The program is called Megasys; it is a property management system.  

uname -a :  Linux campus 2.6.9-5.EL #1 Wed Jan 5 19:22:18 EST 2005 i686 i686 i386 GNU/Linux

df -h: Filesystem     Size    Used      Avail   Use %  Mounted on
          /dev/sda2      12G     1.6G     9.5G     14           /
          /dev/sda1      114M   7.0M     102M    7            /boot
         none               506M   0          506M    0           /dev/shm
          /dev/sda4      55G      12G     41G       22         /mega

free:         Total       Used      free       shared    buffers     cached
     Mem:1035864   123072   912792     0          32200       56264
-/+ buffers/cache:   34608    1001256
Swap:     522104        0         522104

ps aux:  displays list of 31 users with associated PID,%CPU, %MEM VSR,RSS,TTY,STAT, START, TIME, COMMAND -
Do I need to type out full list, or would there be an item in particular that I should be looking for?  


20090220112736098.pdf
0
 
LVL 16

Expert Comment

by:ai_ja_nai
ID: 23693678
have you done any system update? I'm thinking to the Kernel version
0
 
LVL 19

Expert Comment

by:jools
ID: 23693786
Not getting to the user/password prompt could indicate some sort of network issue.

Has someone changed the network settings?
Do you have iptables configured on the server?
Are there any other errors on the users PCs?
Is the application running on the server to allow users to log in?
0
 

Author Comment

by:Technical_Specialist
ID: 23693982
No one had touched the server to my knowledge besides me to re-boot to see if the problem would simplistically go away.  It was running /being accessed fine on Wed, then Thursday morning... not.  As for iptables, I'm not sure.  I did not set it up, nor have I ever had to work on it.  (Unfortunately I know no linux).  Everything else on the users PCs is fine.  I assume (maybe wrong) that the application is running.  When the server boots up, it gets to :  "campus login: MegaPro Host: Starting...    
                                                                MegaPro Host: {unknown};20000;campus"
0
 
LVL 19

Expert Comment

by:jools
ID: 23694063
I have a concern that something may not have been configured to start automatically on reboot...
Do you reboot regularly or whenever you have a blue moon?

To make sure iptables is not the culprit use `service iptables stop` and try again (I think you're using redhat here).

See if there is anything in /var/log, there are some files you can check, messages, secure are a couple. ls -altr will list the ones modified recently there may be some clues...

J
0
 

Author Comment

by:Technical_Specialist
ID: 23695008
We haven't had support for the server for a year, and I haven't had to touch it.  I tried stopping the iptables - nothing new.  I tried running the program while the server was off/rebooting and a completely different message was received, indicating that it must be starting automatically.  
In the /var/log directory, the logs that are recent are from late yesterday(prelink.log) and a few from today(rpmpkgs, dmesg, acpid, secure, maillog, boot.log, cron, wtmp lastlog, messages), but nothing from the night/morning when it went down.  
Looking in the messages log: around 9:29am (I believe shortly before I found out it was down)su(pam_unix)[21908]:session opened for user hotel by (uid=0)
                  su(pam_unix)[21908]:session closed for user hotel
                  shutdown:shutting down for system halt
                  init: Switching to runlevel: 0
                  cups:Cupsd shutdown succeeded
                  sshd:sshd -TERM succeeded
                  xinetd[1982]: Exiting
                  xinetd/acpid/crond: all shutdown succeeded
                  kernel: Kernel logging (proc) stopped/kernel log daemon terminating
                  syslog: klogd shutdown succeeded
                  exiting on signal 15
then at 10:18 (When I think I rebooted) syslogd 1.4.1:restart

Through the boot up: (ones that stood out as failing - same ones for each time tried to reboot)
     kernel: there is already a security framework initialized, register_security failed
     kernel: vesafb: probe of vesafb0 failed with error -6
     kernel: apm: BIOS not found
     kernel: sda: asking for cache data failed
When I try to access any of the other logs, it just repeats the string that I had typed, minus the ls, on the line below


0
 
LVL 19

Expert Comment

by:jools
ID: 23696036
I dont suppose it's possible to log onto the system just to have a look?

I wouldnt normally do this but it seems that most of the obvious stuff has been checked.

If you want me to have a look email my hotmail address ft00mch @ hotmail . com and I'll contact you using my proper account (I don't want to post that here).

The choice is yours.... I'll understand if you dont want to go ahead with it.

In the mean time, I'll rack my brains a little more.

J
0
 
LVL 19

Expert Comment

by:jools
ID: 23696068
Just a thought, Can you post more detail as to how exactly your users connect, what do they do, is there a special application involved?

0
 
LVL 7

Expert Comment

by:Morne Lategan
ID: 23696381
You said:

"After initially installing the program, setting up the host/port, they can click on the icon on their desktop to run the program."

What port are we talking about? Lets say its port 12345, try this on the Server:

Log in with root and run:

netstat -nlp |grep 12345

(or whatever the port is that you fill in on the client).

This will tell you what program is listening on that port (if any). If it returns nothing, the software on the server is not running. If it does return something, please post it here so that we can try to figure out what is listening on the port.
0
 

Author Comment

by:Technical_Specialist
ID: 23696891
For connecting, it's all that I mentioned before.  Once I or other techy installs the program, we set up a host (ip address of the server) and a port number.  After that there is an icon on the desktop, and all they have to do to connect is click it, enter their username/pass on the main window of the application, and they use the program freely.   I think ProvideX may somehow be involved, but not exactly sure how.  
0
 

Author Comment

by:Technical_Specialist
ID: 23696968
It comes back with:
tcp        0     0 0.0.0.:20000          0.0.0.0:*   List
EN 2372/pvx

Looks like part of the problem here.  
0
 
LVL 19

Expert Comment

by:jools
ID: 23696993
When I asked before about checking if the application was running; http://#23693786 you seemed to think it wasnt a problem; http://#23693982.

Can you post chkconfig --list and check to see if the application is shown.

Have you rebooted before successfully.... or was it once in a blue moon???
0
 
LVL 19

Expert Comment

by:jools
ID: 23697000
Run `lsof | grep 20000` to see what is using the port.

also run `ifconfig -a`
0
 

Author Comment

by:Technical_Specialist
ID: 23697113
Under chkconfig --list I don't see the program listed... Displays a decent size list with columns 0-6 with On/Off  (some all off; some 3 offs, 3 ons, 1 off;  some 2 offs, 4 ons, 1 off)
Xinetd based services:  all off except krb5-telnet

 As for rebooting before, it was once in a blue moon, and I never had to do anything more.  The last time was a year ago, when we moved the server to a new rack.  

lsof :   pvx     2372 root  6u   IPv4    5673     TCP *:20000(LISTEN)

ifconfig -a:  

eth0      Lists all network information; UP BROADCAST RUNNING MULTICAST; RX,TX packets:14361/10203 - no errors/dropped/overruns/frame/carrier/collisions; txqueuelen:10000; RX/TX Bytes; Base address/Memory

lo           List loopback information; RX/TX packets:2535/2535 - 0's for everything else

sit0        Link encap:IPv6-in-IPv4 --- 0's for all information
0
 
LVL 19

Expert Comment

by:jools
ID: 23697156
It seems that the pvx process is listening on port 20000. Is that right?

You can have a look in /etc/inittab to see if the process starts there or have a look in /etc/rc.d/init.d for something that mentions pvx.

Can you clarify what the users do to connect to the system, can you `su -` to a userid and see if it works?

0
 

Author Comment

by:Technical_Specialist
ID: 23697261
It lists inittab under etc, but says it is not a directory when I cd /etc/inittab

Under init.d, there is nothing that mentions pvx

When I try the su - 'username' , it says user 'username' does not exist
0
 
LVL 19

Expert Comment

by:jools
ID: 23697286
cat /etc/inittab
cd /etc/rc.d/init.d
grep pvx *

How do your users connect? They must be able to login somehow! username needs to be a valid username for example, look in /etc/passwd and find a userid that accesses the system then run `su - validuser`.

0
 

Author Comment

by:Technical_Specialist
ID: 23697485
running cat /etc/inittab ran through a bunch of stuff:

id:3:initdefault

system initialization.
si::sysinit:/etc/rc.d/rc.sysinit

l0:0:wait:/etc/rc.d/ rc 0 (through 6)

Trap C-A-D
ca::ctrlaltdel:/sbin/shutdown -t3 -r now

Run gettys in standard runlevels
1:2345:respawn:/sbin/mingetty tty1
2:2345:off:/sbin/rungetty tty2 -u hotel /mega/bin/interface T2 (through 7)
8:2345:respawn:/sbin/mingetty tty8

Run xdm in runlevel 5
x:6:respawn:/etc/X11/prefdm -nodaemon

Run megahost for client connections
mhs1:2345:respawn:/mega/bin/megahost <dev/null >/dev/null 2>&1
cred:2345:respawn:/mega/bin/interface CRED <dev/null >/dev/null 2>&1
cact:2345:respawn:/mega/bin/interface CACT <dev/null >/dev/null 2>&1
pbx:2345:respawn:/mega/bin/interface PBX <dev/null >/dev/null 2>&1
voic:2345:respawn:/mega/bin/interface VOIC <dev/null >/dev/null 2>&1

going to the init.d directory, grep pvx * returned nothing

I'm not sure how to look in that.  It says its not a valid directory, and when I'm in the etc directory and type passwd, it prompts to change the root password.

When using the su -, I am using the username that I would use to get into the program.  I've tried a couple others as well, along with admin, root, pam_unix (which was showing up in the messages log).  
 
I attached a file with images of the properties for the shortcut and the screen I get now.  Above in the postings is a PDF with what I should get when I run the program.
MegaPro.doc
0
 
LVL 19

Expert Comment

by:jools
ID: 23697601
OK,

For future reference;

This line looks like some sort of modem thing, it's disabled so you can ignore it.
> 2:2345:off:/sbin/rungetty tty2 -u hotel /mega/bin/interface T2 (through 7)

Theses lines actually run the services;
> mhs1:2345:respawn:/mega/bin/megahost <dev/null >/dev/null 2>&1
> cred:2345:respawn:/mega/bin/interface CRED <dev/null >/dev/null 2>&1
> cact:2345:respawn:/mega/bin/interface CACT <dev/null >/dev/null 2>&1
> pbx:2345:respawn:/mega/bin/interface PBX <dev/null >/dev/null 2>&1
> voic:2345:respawn:/mega/bin/interface VOIC <dev/null >/dev/null 2>&1

You could change them to the following which may display some useful diags;
mhs1:2345:respawn:/mega/bin/megahost
cred:2345:respawn:/mega/bin/interface CRED
cact:2345:respawn:/mega/bin/interface CACT
pbx:2345:respawn:/mega/bin/interface PBX
voic:2345:respawn:/mega/bin/interface VOIC

You did have something listening on port 20000 but I'm not sure if thats normal or if there is something missing, other ports/services etc.

Your users definately use an application, checking the target files can help you here too; I cant see all the info in the document but MegaPro.ini and MegaClient.pvx could be worth checking.

0
 
LVL 7

Expert Comment

by:Morne Lategan
ID: 23697923
This is what's running on port 20000: http://www.pvx.com/

"ProvideX is an exceptionally powerful, versatile, and intuitive programming language for building sophisticated business applications - applications that are ready to accommodate future technologies and a variety of host platforms."

Seems like some sort of application server/development framework. I suppose the system is built on top of it. And clients connect to the software through their middleware.
0
 
LVL 7

Expert Comment

by:Morne Lategan
ID: 23698001
I have the number of a guy at Megasys that you can contact to help you, or is that out of the question?
0
 

Author Comment

by:Technical_Specialist
ID: 23698035
If he is willing.  I tried to contact them yesterday and they said since we stopped paying for support fees a year ago they wouldn't help.
0
 
LVL 7

Expert Comment

by:Morne Lategan
ID: 23698084
Nah, then he probably wont help you :)

Error #12: File does not exist (or already exists)
Possible Reasons:
· Cannot create a file (DIRECT, INDEXED, SORT, etc.) if a file of the same given name
already exists.
· Filename does not exist ­ cannot ERASE, OPEN, LOAD, RUN, or CALL a
non-existent file.
· In [TCP] ­ can't OPEN, server is not listening.

That last bit seems likely???
0
 
LVL 7

Expert Comment

by:Morne Lategan
ID: 23698090
Can you telnet to port 20000 of the server from the workstation?

telnet <serverip> 20000
0
 
LVL 7

Expert Comment

by:Morne Lategan
ID: 23698117
In your inittab file, put # before this line:

#mhs1:2345:respawn:/mega/bin/megahost <dev/null >/dev/null 2>&1

Then save it, and from the console, run:

init q

That will stop the application.

Then from the command line, run it manually:

/mega/bin/megahost

Copy the output, and post it here.

Then take the hash out again, and run

init q

again
0
 

Author Comment

by:Technical_Specialist
ID: 23698119
If I do a telnet <serverip> I get:
   campus (Linux release 2.6.9-5.ELsmp #1 SMP Wed Jan 5 19:30:39 EST 2005) (0)
login:
(can't login with any user/root/admin though)
If I telnet <serverIP> 20000, I get a blank box, and anything I type get an OK, just about after every letter I type, or hit enter.
0
 
LVL 7

Expert Comment

by:Morne Lategan
ID: 23698156
Well, we can figure out what the comms are between the client and the server:

On the server, type:

netcat -l -p 12345

(that's the letter "l" for listen)

or

nc -l -p 12345

That will hook a listerner to port 12345. It will sit an wait for input.

Then point your client to port 12345 instead of 20000. And see what appears on the server netcat screen. Whenever the client sends something, write it down, and respond with OK and enter on the netcat screen.

Then reverse the process, from the client pc:

 telnet <serverip> 20000

And now send whatever it was that the client sent to your netcat session, and see what the server responds. Maybe it gives an error at some stage.

0
 
LVL 19

Expert Comment

by:jools
ID: 23698817
So at least telnet is running, check your /var/log/secure file for errors when you try to telnet in.

Telnetting to the open port looks OK but we already know the service is listening.

I'm not convinced all the services are running properly, can you modify the inittab file as I mentioned above and restart (or init -c / telinit q ??)
0
 

Author Comment

by:Technical_Specialist
ID: 23699922
How do you edit/access files?   Also for logs, I found the command 'ls <logfile>' to view the log, but only seemed to allow me to see one (messages) log and could not exit it.  I'm getting the nack of moving through the directories as it is very similiar to dos, but the rest of the commands are not.  I've found a few sites with basic commands, but which ones should I be using?
0
 
LVL 19

Assisted Solution

by:jools
jools earned 300 total points
ID: 23699963
cat | more - allow you to view files : eg/ cat <file>; cat <file> | more; more <file>
ls - lists the contents of a directory : eg/ ls -altr <directory>
vim - edit files, you need to know vi to do this!!! long story!
cd - change directory : eg/ cd <directory> ; cd ../../<directory
0
 
LVL 7

Expert Comment

by:Morne Lategan
ID: 23699990
The easiest commandline editor is nano. It is similar to notepad and the dos editor.

Please be sure to make a backup copy of any files first. ESPECIALLY important files like inittab that can bring your entire system to a halt:

Making a backup:
----------------------

Create a backup dir:
mkdir /root/backups

Copy inittab to it:
cd /etc
cp inittab /root/backups

Make sure its there:
ls /root/backups

Do the same for any other file you want to edit. I'm saying this because editors in Linux are not always as easy to use as notepad, and pressing the wrong button can cause havoc. Rather safe than sorry :)

Editing files:
---------------
Now edit your file:
nano /etc/inittab

Press CTRL-X once you're done and it will ask you to save.

If you don't have nano, you'll have to use vi, but before doing so, read an introduction to vi. Do NOT use vi without reading something first :)

Viewing files:
----------------

Viewing the file page-by-page:
more /path/to/file

Viewing the end of the file (usefull for logs):
tail -<numberoflines> file
e.g. tail -1000 file

Viewing the end of the file page by page:
tail -1000 file | more

Monitoring changes to a logfile:
tail -f file
(press CTRL-C to quit monitoring it)

0
 
LVL 7

Expert Comment

by:Morne Lategan
ID: 23700023
I you can provide remote access to the machine through vpn or ssh, I'll be glad to help you fix your problem. Mail me at uberpappa123 at gmail dot com and I'll give you the details of how to contact me via instant messaging.
0
 

Author Comment

by:Technical_Specialist
ID: 23710896
Editing the inittab file caused havoc.  Good thing for backups.  It kept giving the error message that would be received when starting the application.  Also it made it VERY difficult to log back in.  

In the secure log:
  Feb 19 09:31:28 campus sshd[1929]: Received signal 15; terminating
  Feb 19 10:18:52 campus sshd[1940]: Server listening on :: port22
  Feb 19 10:18:52 campus sshd[1940]: error: Bind to port 22 on 0.0.0.0 failed: Address already in use.

It had the same outputs 4 more times w/ different [#'s], then:
Feb 20 16:15:14 campus xinetd[2178]: START: telnet pid=2494 from=10.0.0.90  (x10 w/ different pid's)
Then 6 more of the above messages.

The netcat screen displayed the same error message the client gets.
0
 
LVL 7

Expert Comment

by:Morne Lategan
ID: 23711080
Just to clarify:

When you did netcat -l -p 12345 and configured the client to connect to the server ip and port 12345, it still gave the same error on the client?
0
 
LVL 7

Expert Comment

by:Morne Lategan
ID: 23711218
If so, what happens if you just completely plug the client out of the network, and then connect to the server?
0
 
LVL 19

Expert Comment

by:jools
ID: 23711328
> Editing the inittab file caused havoc.  Good thing for backups.  It kept giving the error message that would be received when starting the application.  Also it made it VERY difficult to log back in.  

Do the same again but turn them to `off`
mhs1:2345:off:/mega/bin/megahost
cred:2345:off:/mega/bin/interface CRED
cact:2345:off:/mega/bin/interface CACT
pbx:2345:off:/mega/bin/interface PBX
voic:2345:off:/mega/bin/interface VOIC

Then run the commands manually. Post the errors back here;

The offer is still open BTW....


0
 

Author Comment

by:Technical_Specialist
ID: 23723338
'When you did netcat -l -p 12345 and configured the client to connect to the server ip and port 12345, it still gave the same error on the client?'

Doing this, 'V2MEGAPROT0(this is the number 0):jeff' showed on the server.  On the workstation, it did nothing.  After I pressed Enter on the server, a Provide X box came up on the workstation - stating 'Unable to acquire license from server in time allowed'
0
 
LVL 19

Expert Comment

by:jools
ID: 23723440
> 'Unable to acquire license from server in time allowed'

I was wondering if it was a licensing problem.... has the license expired?
It would certainly point to the problem looking sort of OK on the server.

Unless of course it's because the workstation cannot connect to get the license information....

0
 

Author Comment

by:Technical_Specialist
ID: 23723508
Definitely could be expired.  It's been running un-manned for a year now, and they probably did not want to continue to purchase additional licenses if they were fading it out.  Kinda weird that it would just go without warning or notification of expiration.    
0
 
LVL 7

Accepted Solution

by:
Morne Lategan earned 1200 total points
ID: 23723645
It wont collect the license when you connect to the netcat.  When connecting to port 20000, this error did not come up, which means that a valid license was exchanged.

However this confirms that indeed the client is talking to the server on port 20000, the server does give back a valid license, and then the error occurs when the session is started.

So the problem lies deeper than simple networking issues. Its inside the system (on the server side) somewhere. Which sadly probably means that you will have to pay the vendor for results :(
0
 
LVL 19

Expert Comment

by:jools
ID: 23723913
have you tried changing the date on the server and workstations and restarting....
0
 

Author Closing Comment

by:Technical_Specialist
ID: 31549087
Thanks for the help.  Learned quite a bit about Linux with this, which might be good for the future, but still unable to resolve the problem.  We are trying to get the software company to take a look at it, but they still don't seem to want to help even with one time paid support.
0

Featured Post

New feature and membership benefit!

New feature! Upgrade and increase expert visibility of your issues with Priority Questions.

Question has a verified solution.

If you are experiencing a similar issue, please ask a related question

This article will explain how to establish a SSH connection to Ubuntu through the firewall and using a different port other then 22. I have set up a Ubuntu virtual machine in Virtualbox and I am running a Windows 7 workstation. From the Ubuntu vi…
I have written articles previously comparing SARDU and YUMI.  I also included a couple of lines about Easy2boot (easy2boot.com).  I have now been using, and enjoying easy2boot as my sole multiboot utility for some years and realize that it deserves …
Learn how to navigate the file tree with the shell. Use pwd to print the current working directory: Use ls to list a directory's contents: Use cd to change to a new directory: Use wildcards instead of typing out long directory names: Use ../ to move…
How to Install VMware Tools in Red Hat Enterprise Linux 6.4 (RHEL 6.4) Step-by-Step Tutorial
Suggested Courses
Course of the Month19 days, 22 hours left to enroll

873 members asked questions and received personalized solutions in the past 7 days.

Join the community of 500,000 technology professionals and ask your questions.

Join & Ask a Question