Link to home
Start Free TrialLog in
Avatar of David Brennan
David BrennanFlag for Ireland

asked on

Lots of Logons and Logoffs (straight away) in Event Viewer (Server 2003)

We've noticed an issue on one of our servers - an important service we need (IBSERVER, which is an Interbase Database Service) - keeps failing,  and it seems to coincide with multiple logons of the same logon which happens sporadically throughout the day. What makes matters worse is that the logon username has administrative rights and has been used throughout our environment (regionally) so I can't just reset the password or disable the account (please don't lecture me on this sort of thing - I completely understand Microsoft's recommendations for admin accounts but I can't speak for the rest of the national team).

Anyway to cut a long story short, the logons are promptly followed by a log off.. all within the same second we get the following event IDs - 540,576,538

There are numerous other 'normal' logons on the server all through the day but this logon is particularly worrying because it was happening a lot on a Sunday when no IT staff would be around, so it's either one of our own automated systems using those credentials, or it's a software interface (there are a few connecting back to that server), or (god forbid) it's something or someone nefarious (a virus or whatever).

So I'm wondering is there a way to check which hostname or IP these logons are coming from ? should I write a short logon script to log to a text file ? or is there a better way ? I need to figure out why this logon is happening so often in order to fix our other system.
Avatar of gheist
gheist
Flag of Belgium image

I have a feeling this is related fix:
http://support.microsoft.com/kb/923241/

Can you post a text copied from one representative error message?
Avatar of David Brennan

ASKER

as far as I can see that article is related to lsass.exe and domain controllers, and nothing to do with the question I asked. Any other ideas?
please post error message from event log, i cannot attribute any hotfix to particular problem if i do not see event.
latest netlogon hotfix includes all others which in turn may or may not fix your domain authentication issue. you need to apply it on machines where logon failures are logged, not more.
Are you sure it's an authentication issue ? it's not a domain controller, and the event IDs I listed above are information events, not error events. The user gets logged in and then logs off. There's no error message as such, except the event ID 281 with the Interbase Guardian service failing at more or less the same time as the logon event.
Dear thedavil!
Please post all error messages mentioned. Your chatting abput and around them does not constitute a valid technical information to resolve authentication problem.

Thank you in advance for your understanding and support.
ERROR - EVENT ID 281 (application log)
-------------------------------
The description for Event ID ( 281 ) in Source ( InterBase Guardian ) cannot be found. Either the component that raises this event is not installed on your local computer, or the installation is corrupted. You can install or repair the component on the local computer, or contact the component manufacturer for a newer version.
If the event was saved from another computer or forwarded from a remote computer, you might have to include display information with the events when saving them or when setting up the forwarding s Abnormal Termination: C:\Program Files\Borland\InterBase\bin\ibserver.exe: terminated abnormally (-1).



at the exact same time, three successful events appear in security log:
-----------------------------------------------------------------

event 540:
----------
Successful Network Logon:
       User Name:      obscuredforsecurity
       Domain:            obscuredforsecurity
       Logon ID:            (0x0,0xBC7ABCC)
       Logon Type:      3
       Logon Process:      Authz  
       Authentication Package:      Kerberos
       Workstation Name:      obscuredforsecurity
       Logon GUID:      -
       Caller User Name:      obscuredforsecurity$
       Caller Domain:      obscuredforsecurity
       Caller Logon ID:      (0x0,0x3E7)
       Caller Process ID: 884
       Transited Services: -
       Source Network Address:      -
       Source Port:      -

event 576:
----------
Special privileges assigned to new logon:
       User Name:      obscuredforsecurity
       Domain:            obscuredforsecurity
       Logon ID:            (0x0,0xBC7ABCC)
       Privileges:      SeSecurityPrivilege
                  SeBackupPrivilege
                  SeRestorePrivilege
                  SeTakeOwnershipPrivilege
                  SeDebugPrivilege
                  SeSystemEnvironmentPrivilege
                  SeLoadDriverPrivilege
                  SeImpersonatePrivilege
event 538:
----------
User Logoff:
       User Name:      backuptest
       Domain:            HQD01
       Logon ID:            (0x0,0xBC7ABCC)
       Logon Type:      3
Please use "copy" button in error details.
Error text without source makes little sense.

Authz is "kerberos client" aka impersonation.

I suspect hotfix http://support.microsoft.com/kb/890477 and not my initial guess is of help.
(or adding DB account to domain)
helps.

Here's the error below in more detail (I can't see what difference it makes)

Also can you explain why you think that the KB page link you sent me (event ID 529) is related in any way ?
---------------------------------------------------------------------------------------

Event Type:      Error
Event Source:      InterBase Guardian
Event Category:      None
Event ID:      281
Date:            20/02/2011
Time:            14:42:09
User:            N/A
Computer:      obscuredforsecurity
Description:
The description for Event ID ( 281 ) in Source ( InterBase Guardian ) cannot be found. Either the component that raises this event is not installed on your local computer, or the installation is corrupted. You can install or repair the component on the local computer, or contact the component manufacturer for a newer version.

If the event was saved from another computer or forwarded from a remote computer, you might have to include display information with the events when saving them or when setting up the forwarding s Abnormal Termination: C:\Program Files\Borland\InterBase\bin\ibserver.exe: terminated abnormally (-1).


authz is suspected source of event 540
thus hotfix describes that you may experience problems without it if impersonating user is outside domain.
interbase error makes little sense... is there any interbase log where you can find more details?
I will look into the interbase error - I'll see if I can find some log file somewhere.
as for hotfix, I will apply as soon as possible and see if it makes any difference. I forgot to mention that apparently this problem only started in the last few weeks. the error actually only happens every few days. I've attached a screenshot of the timing of the error. User generated image
Application hang is very suspicious
Double click that event, press "copy" button and paste it here.

This https://www.microsoft.com/downloads/en/details.aspx?FamilyID=1b286e6d-8912-4e18-b570-42470e2f3582 for (hopefully)  irrelevant userenv problem.

If you get past userenv problem try to paste more relevant errors.
Event Type:      Error
Event Source:      Application Hang
Event Category:      (101)
Event ID:      1002
Date:            15/02/2011
Time:            16:22:46
User:            N/A
Computer:      obscuredforsecurity
Description:
Hanging application IBConsole.exe, version 7.5.0.26, hang module hungapp, version 0.0.0.0, hang address 0x00000000.

For more information, see Help and Support Center at http://go.microsoft.com/fwlink/events.asp.
Data:
0000: 41 70 70 6c 69 63 61 74   Applicat
0008: 69 6f 6e 20 48 61 6e 67   ion Hang
0010: 20 20 49 42 43 6f 6e 73     IBCons
0018: 6f 6c 65 2e 65 78 65 20   ole.exe
0020: 37 2e 35 2e 30 2e 32 36   7.5.0.26
0028: 20 69 6e 20 68 75 6e 67    in hung
0030: 61 70 70 20 30 2e 30 2e   app 0.0.
0038: 30 2e 30 20 61 74 20 6f   0.0 at o
0040: 66 66 73 65 74 20 30 30   ffset 00
0048: 30 30 30 30 30 30         000000  



Event Type:      Error
Event Source:      Application Hang
Event Category:      (101)
Event ID:      1002
Date:            15/02/2011
Time:            16:22:23
User:            N/A
Computer:      obscuredforsecurity
Description:
Hanging application pmirep.exe, version 2.9.41.10, hang module hungapp, version 0.0.0.0, hang address 0x00000000.

For more information, see Help and Support Center at http://go.microsoft.com/fwlink/events.asp.
Data:
0000: 41 70 70 6c 69 63 61 74   Applicat
0008: 69 6f 6e 20 48 61 6e 67   ion Hang
0010: 20 20 70 6d 69 72 65 70     pmirep
0018: 2e 65 78 65 20 32 2e 39   .exe 2.9
0020: 2e 34 31 2e 31 30 20 69   .41.10 i
0028: 6e 20 68 75 6e 67 61 70   n hungap
0030: 70 20 30 2e 30 2e 30 2e   p 0.0.0.
0038: 30 20 61 74 20 6f 66 66   0 at off
0040: 73 65 74 20 30 30 30 30   set 0000
0048: 30 30 30 30               0000    



Event Type:      Error
Event Source:      Application Hang
Event Category:      None
Event ID:      1001
Date:            15/02/2011
Time:            16:21:09
User:            N/A
Computer:      OLDAT02
Description:
Fault bucket 214237803.

For more information, see Help and Support Center at http://go.microsoft.com/fwlink/events.asp.
Data:
0000: 42 75 63 6b 65 74 3a 20   Bucket:
0008: 32 31 34 32 33 37 38 30   21423780
0010: 33 0d 0a                  3..    


Event Type:      Error
Event Source:      Application Hang
Event Category:      (101)
Event ID:      1002
Date:            15/02/2011
Time:            16:20:55
User:            N/A
Computer:      OLDAT02
Description:
Hanging application IBConsole.exe, version 7.5.0.26, hang module hungapp, version 0.0.0.0, hang address 0x00000000.

For more information, see Help and Support Center at http://go.microsoft.com/fwlink/events.asp.
Data:
0000: 41 70 70 6c 69 63 61 74   Applicat
0008: 69 6f 6e 20 48 61 6e 67   ion Hang
0010: 20 20 49 42 43 6f 6e 73     IBCons
0018: 6f 6c 65 2e 65 78 65 20   ole.exe
0020: 37 2e 35 2e 30 2e 32 36   7.5.0.26
0028: 20 69 6e 20 68 75 6e 67    in hung
0030: 61 70 70 20 30 2e 30 2e   app 0.0.
0038: 30 2e 30 20 61 74 20 6f   0.0 at o
0040: 66 66 73 65 74 20 30 30   ffset 00
0048: 30 30 30 30 30 30         000000  


Event Type:      Error
Event Source:      Application Hang
Event Category:      (101)
Event ID:      1002
Date:            15/02/2011
Time:            16:20:22
User:            N/A
Computer:      OLDAT02
Description:
Hanging application IBConsole.exe, version 7.5.0.26, hang module hungapp, version 0.0.0.0, hang address 0x00000000.

For more information, see Help and Support Center at http://go.microsoft.com/fwlink/events.asp.
Data:
0000: 41 70 70 6c 69 63 61 74   Applicat
0008: 69 6f 6e 20 48 61 6e 67   ion Hang
0010: 20 20 49 42 43 6f 6e 73     IBCons
0018: 6f 6c 65 2e 65 78 65 20   ole.exe
0020: 37 2e 35 2e 30 2e 32 36   7.5.0.26
0028: 20 69 6e 20 68 75 6e 67    in hung
0030: 61 70 70 20 30 2e 30 2e   app 0.0.
0038: 30 2e 30 20 61 74 20 6f   0.0 at o
0040: 66 66 73 65 74 20 30 30   ffset 00
0048: 30 30 30 30 30 30         000000  
I've attached an interbase log file (partial) - all day today and all previous days there seem to be an error 10054 message coming up (it's appearing with every single citrix logon it would seem). After doing some quick research, it seems like a network issue maybe:

 INTERBASE-LOG-FOR-TODAY-PARTIAL-.TXT

also on the last day that we got the IBSERVER failures, the following appears in the interbase logs:

 SUNDAY-20TH-FEB-ERRORS.TXT
http://www.ib-aid.com/articles/item69

search in page for 10054 - contact technical support if could not find a problem on a network...
(or try firebird client driver ???)

Is your apps crashing interactively or behind the scenes? address 0x0 is usually manual "end task"

This may pont you to a test case for support:
http://tracker.firebirdsql.org/browse/CORE-1196

Please give feedback if userenv events went away....
Do you see more authz errors? Do they relate to hungapp events?
Do you want / are you able / are you allowed to debug hanging application(s)?

pmirem yields nothing on google, so it seems like in-house application, which you can ask original developer to debug....
Original developer is gone bust.... but we're in healthcare so we can't just discontinue using the app. We have a consultancy firm looking at it but to be honest they're leaving a lot of this on me, even though I'm not highly trained in any of this stuff. But I'm sure we'll probably figure it out soon. I will keep you posted with info re hotix etc
ASKER CERTIFIED SOLUTION
Avatar of gheist
gheist
Flag of Belgium image

Link to home
membership
This solution is only available to members.
To access this solution, you must be a member of Experts Exchange.
Start Free Trial
Sorry man, that's way above my head. I might leave it to the consultancy firm.. I'll push them to analyse the problem themselves instead of dumping it on me. Third party software is a pain in the &£^$&^$
Recompiling software in latest visual studio may help.
we don't have the source code. closed source commercial app.
Ask for compatible version?

Please use (1) on crashing binaries and apply compatibility mode as suggested.
If you still experience crashes consider (3) or leave for consultants

Dont give up so quickly....

Please please please ;)

I find your problem common and "interesting"...
1. Unfortunately I cannot analyse the app 'before running', as it runs 24 / 7 / 365.

2. Tasklist /m (should be same as process explorer) gives me the following dependencies for ibserver.exe

ibserver.exe                  1076 ntdll.dll, kernel32.dll, GDI32.dll,        
                                   USER32.dll, ADVAPI32.dll, RPCRT4.dll,      
                                   Secur32.dll, MSVCRT.dll, COMCTL32.dll,      
                                   WS2_32.dll, WS2HELP.dll, MPR.dll,          
                                   SHELL32.dll, SHLWAPI.dll, IMM32.DLL,        
                                   comctl32.dll, libborland_lm.dll,            
                                   apphelp.dll, msctfime.ime, ole32.dll,      
                                   DNSAPI.dll, mswsock.dll, winrnr.dll,        
                                   WLDAP32.dll, rasadhlp.dll, hnetcfg.dll,    
                                   wshtcpip.dll, gdsintl.dll, ksudflib.dll,    
                                   oleaut32.dll, gds32.dll

so no MSVCRT....

unfortunately I think I will have to leave number 3 to the consultants because I'm not a C++ or C# or even a Delphi developer ( I'll stick to PHP ) and I'm not sure how the program works.

At the moment there are no errors or warnings in the event log since at least the 24th of February. It must have been some problem at the specific time in the logs. Thanks for your help but I'm not sure I'm going to figure this out through looking at the app. I've a feeling the logons in the event viewer can be attributed to one or more of the other 'interfaces' that connect back to the firebird database. The original software company created a few of these 'interfaces' that pump data in and out of the database to integrate with other softwares. I should be able to track it down eventually. Your steps 1, 2 and 3 I would imagine will come in very useful for troubleshooting a lot of software problems, so thank you.

Also, do you have any idea how to answer my original question - i.e. is there any way to track past logons from the specific user that was appearing in the event viewer -  back to a hostname (point of origin)
apphelp.dll ?

So you ARE actually using compatibility mode.... (either by MS o

Check msvcrt.dll around
It is shared component ov visual C 4-6 (v6 provided also numbered component)
It is from times of NT4-W2000

Check if there are multiple versions of this dll on the system.

Rename those in program directories, best is in system32 which may not load compatibility mode.
3 tools i mentioned show actual file being loaded in process image... it is more informative.
Yes I realise your recommended tools are better but as I've said, I cannot restart the software while it is running, it is important to keeping our X-Ray department functioning. I think I'm going to have to just close this question off and give you the points. I'm not really allowed to start renaming DLLs etc on major systems - I'll leave it to the software consultants.
Wait
Do static analysis on binary with depends.exe
If you discover dual msvcrt you can plan 2x5min downtime to resolve and/or revert in case of failure.
That program crashing is real pain in the a--.
Can you repeat crash in non-production vmware? It could be a good starting point for debugger.
Once you make it stable everybody will thank you.
You can see the result immediatly - if app does not load apphelp.dll it runs in native mode (as microsoft wants it to be)
Give it a try... Ask some doctor for test case when it crashes, it might give you volunteer to attach debugger and test the solution.

Mistake was done by letting app in compatibility mode in production. You can only correct it. It cannot get worse.
BUT write down what you plan to do and ask somebody to watch if you do it right.
ok I'll take a look.
I think consultants outside microsoft or interbase official support cannot help much
Do paperwork - really, if after 2 years you will go for w2008 R3 you know what i mean...
I ask moderators to call more attention to this question.
Similar problem with succesfully avoiding DLL conflicts
https://www.experts-exchange.com/questions/26843009/Upgrading-SSL-need-how-to.html
Luckily did not get far enough to run debuggers, depends.exe and process explorer.