Solved

Lots of Logons and Logoffs (straight away) in Event Viewer (Server 2003)

Posted on 2011-02-23
30
852 Views
Last Modified: 2012-05-11
We've noticed an issue on one of our servers - an important service we need (IBSERVER, which is an Interbase Database Service) - keeps failing,  and it seems to coincide with multiple logons of the same logon which happens sporadically throughout the day. What makes matters worse is that the logon username has administrative rights and has been used throughout our environment (regionally) so I can't just reset the password or disable the account (please don't lecture me on this sort of thing - I completely understand Microsoft's recommendations for admin accounts but I can't speak for the rest of the national team).

Anyway to cut a long story short, the logons are promptly followed by a log off.. all within the same second we get the following event IDs - 540,576,538

There are numerous other 'normal' logons on the server all through the day but this logon is particularly worrying because it was happening a lot on a Sunday when no IT staff would be around, so it's either one of our own automated systems using those credentials, or it's a software interface (there are a few connecting back to that server), or (god forbid) it's something or someone nefarious (a virus or whatever).

So I'm wondering is there a way to check which hostname or IP these logons are coming from ? should I write a short logon script to log to a text file ? or is there a better way ? I need to figure out why this logon is happening so often in order to fix our other system.
0
Comment
Question by:David Brennan
  • 17
  • 13
30 Comments
 
LVL 61

Expert Comment

by:gheist
ID: 34968627
I have a feeling this is related fix:
http://support.microsoft.com/kb/923241/

Can you post a text copied from one representative error message?
0
 
LVL 3

Author Comment

by:David Brennan
ID: 34995750
as far as I can see that article is related to lsass.exe and domain controllers, and nothing to do with the question I asked. Any other ideas?
0
 
LVL 61

Expert Comment

by:gheist
ID: 34996420
please post error message from event log, i cannot attribute any hotfix to particular problem if i do not see event.
latest netlogon hotfix includes all others which in turn may or may not fix your domain authentication issue. you need to apply it on machines where logon failures are logged, not more.
0
 
LVL 3

Author Comment

by:David Brennan
ID: 34996518
Are you sure it's an authentication issue ? it's not a domain controller, and the event IDs I listed above are information events, not error events. The user gets logged in and then logs off. There's no error message as such, except the event ID 281 with the Interbase Guardian service failing at more or less the same time as the logon event.
0
 
LVL 61

Expert Comment

by:gheist
ID: 34996576
Dear thedavil!
Please post all error messages mentioned. Your chatting abput and around them does not constitute a valid technical information to resolve authentication problem.

Thank you in advance for your understanding and support.
0
 
LVL 3

Author Comment

by:David Brennan
ID: 34996641
ERROR - EVENT ID 281 (application log)
-------------------------------
The description for Event ID ( 281 ) in Source ( InterBase Guardian ) cannot be found. Either the component that raises this event is not installed on your local computer, or the installation is corrupted. You can install or repair the component on the local computer, or contact the component manufacturer for a newer version.
If the event was saved from another computer or forwarded from a remote computer, you might have to include display information with the events when saving them or when setting up the forwarding s Abnormal Termination: C:\Program Files\Borland\InterBase\bin\ibserver.exe: terminated abnormally (-1).



at the exact same time, three successful events appear in security log:
-----------------------------------------------------------------

event 540:
----------
Successful Network Logon:
       User Name:      obscuredforsecurity
       Domain:            obscuredforsecurity
       Logon ID:            (0x0,0xBC7ABCC)
       Logon Type:      3
       Logon Process:      Authz  
       Authentication Package:      Kerberos
       Workstation Name:      obscuredforsecurity
       Logon GUID:      -
       Caller User Name:      obscuredforsecurity$
       Caller Domain:      obscuredforsecurity
       Caller Logon ID:      (0x0,0x3E7)
       Caller Process ID: 884
       Transited Services: -
       Source Network Address:      -
       Source Port:      -

event 576:
----------
Special privileges assigned to new logon:
       User Name:      obscuredforsecurity
       Domain:            obscuredforsecurity
       Logon ID:            (0x0,0xBC7ABCC)
       Privileges:      SeSecurityPrivilege
                  SeBackupPrivilege
                  SeRestorePrivilege
                  SeTakeOwnershipPrivilege
                  SeDebugPrivilege
                  SeSystemEnvironmentPrivilege
                  SeLoadDriverPrivilege
                  SeImpersonatePrivilege
event 538:
----------
User Logoff:
       User Name:      backuptest
       Domain:            HQD01
       Logon ID:            (0x0,0xBC7ABCC)
       Logon Type:      3
0
 
LVL 61

Expert Comment

by:gheist
ID: 34996820
Please use "copy" button in error details.
Error text without source makes little sense.

Authz is "kerberos client" aka impersonation.

I suspect hotfix http://support.microsoft.com/kb/890477 and not my initial guess is of help.
(or adding DB account to domain)
helps.

0
 
LVL 3

Author Comment

by:David Brennan
ID: 34996850
Here's the error below in more detail (I can't see what difference it makes)

Also can you explain why you think that the KB page link you sent me (event ID 529) is related in any way ?
---------------------------------------------------------------------------------------

Event Type:      Error
Event Source:      InterBase Guardian
Event Category:      None
Event ID:      281
Date:            20/02/2011
Time:            14:42:09
User:            N/A
Computer:      obscuredforsecurity
Description:
The description for Event ID ( 281 ) in Source ( InterBase Guardian ) cannot be found. Either the component that raises this event is not installed on your local computer, or the installation is corrupted. You can install or repair the component on the local computer, or contact the component manufacturer for a newer version.

If the event was saved from another computer or forwarded from a remote computer, you might have to include display information with the events when saving them or when setting up the forwarding s Abnormal Termination: C:\Program Files\Borland\InterBase\bin\ibserver.exe: terminated abnormally (-1).


0
 
LVL 61

Expert Comment

by:gheist
ID: 34996870
authz is suspected source of event 540
thus hotfix describes that you may experience problems without it if impersonating user is outside domain.
interbase error makes little sense... is there any interbase log where you can find more details?
0
 
LVL 3

Author Comment

by:David Brennan
ID: 34997189
I will look into the interbase error - I'll see if I can find some log file somewhere.
as for hotfix, I will apply as soon as possible and see if it makes any difference. I forgot to mention that apparently this problem only started in the last few weeks. the error actually only happens every few days. I've attached a screenshot of the timing of the error. Warnings + Errors for last while
0
 
LVL 61

Expert Comment

by:gheist
ID: 34997222
Application hang is very suspicious
Double click that event, press "copy" button and paste it here.

This https://www.microsoft.com/downloads/en/details.aspx?FamilyID=1b286e6d-8912-4e18-b570-42470e2f3582 for (hopefully)  irrelevant userenv problem.

If you get past userenv problem try to paste more relevant errors.
0
 
LVL 3

Author Comment

by:David Brennan
ID: 34997261
Event Type:      Error
Event Source:      Application Hang
Event Category:      (101)
Event ID:      1002
Date:            15/02/2011
Time:            16:22:46
User:            N/A
Computer:      obscuredforsecurity
Description:
Hanging application IBConsole.exe, version 7.5.0.26, hang module hungapp, version 0.0.0.0, hang address 0x00000000.

For more information, see Help and Support Center at http://go.microsoft.com/fwlink/events.asp.
Data:
0000: 41 70 70 6c 69 63 61 74   Applicat
0008: 69 6f 6e 20 48 61 6e 67   ion Hang
0010: 20 20 49 42 43 6f 6e 73     IBCons
0018: 6f 6c 65 2e 65 78 65 20   ole.exe
0020: 37 2e 35 2e 30 2e 32 36   7.5.0.26
0028: 20 69 6e 20 68 75 6e 67    in hung
0030: 61 70 70 20 30 2e 30 2e   app 0.0.
0038: 30 2e 30 20 61 74 20 6f   0.0 at o
0040: 66 66 73 65 74 20 30 30   ffset 00
0048: 30 30 30 30 30 30         000000  



Event Type:      Error
Event Source:      Application Hang
Event Category:      (101)
Event ID:      1002
Date:            15/02/2011
Time:            16:22:23
User:            N/A
Computer:      obscuredforsecurity
Description:
Hanging application pmirep.exe, version 2.9.41.10, hang module hungapp, version 0.0.0.0, hang address 0x00000000.

For more information, see Help and Support Center at http://go.microsoft.com/fwlink/events.asp.
Data:
0000: 41 70 70 6c 69 63 61 74   Applicat
0008: 69 6f 6e 20 48 61 6e 67   ion Hang
0010: 20 20 70 6d 69 72 65 70     pmirep
0018: 2e 65 78 65 20 32 2e 39   .exe 2.9
0020: 2e 34 31 2e 31 30 20 69   .41.10 i
0028: 6e 20 68 75 6e 67 61 70   n hungap
0030: 70 20 30 2e 30 2e 30 2e   p 0.0.0.
0038: 30 20 61 74 20 6f 66 66   0 at off
0040: 73 65 74 20 30 30 30 30   set 0000
0048: 30 30 30 30               0000    



Event Type:      Error
Event Source:      Application Hang
Event Category:      None
Event ID:      1001
Date:            15/02/2011
Time:            16:21:09
User:            N/A
Computer:      OLDAT02
Description:
Fault bucket 214237803.

For more information, see Help and Support Center at http://go.microsoft.com/fwlink/events.asp.
Data:
0000: 42 75 63 6b 65 74 3a 20   Bucket:
0008: 32 31 34 32 33 37 38 30   21423780
0010: 33 0d 0a                  3..    


Event Type:      Error
Event Source:      Application Hang
Event Category:      (101)
Event ID:      1002
Date:            15/02/2011
Time:            16:20:55
User:            N/A
Computer:      OLDAT02
Description:
Hanging application IBConsole.exe, version 7.5.0.26, hang module hungapp, version 0.0.0.0, hang address 0x00000000.

For more information, see Help and Support Center at http://go.microsoft.com/fwlink/events.asp.
Data:
0000: 41 70 70 6c 69 63 61 74   Applicat
0008: 69 6f 6e 20 48 61 6e 67   ion Hang
0010: 20 20 49 42 43 6f 6e 73     IBCons
0018: 6f 6c 65 2e 65 78 65 20   ole.exe
0020: 37 2e 35 2e 30 2e 32 36   7.5.0.26
0028: 20 69 6e 20 68 75 6e 67    in hung
0030: 61 70 70 20 30 2e 30 2e   app 0.0.
0038: 30 2e 30 20 61 74 20 6f   0.0 at o
0040: 66 66 73 65 74 20 30 30   ffset 00
0048: 30 30 30 30 30 30         000000  


Event Type:      Error
Event Source:      Application Hang
Event Category:      (101)
Event ID:      1002
Date:            15/02/2011
Time:            16:20:22
User:            N/A
Computer:      OLDAT02
Description:
Hanging application IBConsole.exe, version 7.5.0.26, hang module hungapp, version 0.0.0.0, hang address 0x00000000.

For more information, see Help and Support Center at http://go.microsoft.com/fwlink/events.asp.
Data:
0000: 41 70 70 6c 69 63 61 74   Applicat
0008: 69 6f 6e 20 48 61 6e 67   ion Hang
0010: 20 20 49 42 43 6f 6e 73     IBCons
0018: 6f 6c 65 2e 65 78 65 20   ole.exe
0020: 37 2e 35 2e 30 2e 32 36   7.5.0.26
0028: 20 69 6e 20 68 75 6e 67    in hung
0030: 61 70 70 20 30 2e 30 2e   app 0.0.
0038: 30 2e 30 20 61 74 20 6f   0.0 at o
0040: 66 66 73 65 74 20 30 30   ffset 00
0048: 30 30 30 30 30 30         000000  
0
 
LVL 3

Author Comment

by:David Brennan
ID: 34997725
I've attached an interbase log file (partial) - all day today and all previous days there seem to be an error 10054 message coming up (it's appearing with every single citrix logon it would seem). After doing some quick research, it seems like a network issue maybe:

 INTERBASE-LOG-FOR-TODAY-PARTIAL-.TXT

also on the last day that we got the IBSERVER failures, the following appears in the interbase logs:

 SUNDAY-20TH-FEB-ERRORS.TXT
0
 
LVL 61

Expert Comment

by:gheist
ID: 34998865
http://www.ib-aid.com/articles/item69

search in page for 10054 - contact technical support if could not find a problem on a network...
(or try firebird client driver ???)

Is your apps crashing interactively or behind the scenes? address 0x0 is usually manual "end task"

This may pont you to a test case for support:
http://tracker.firebirdsql.org/browse/CORE-1196

Please give feedback if userenv events went away....
Do you see more authz errors? Do they relate to hungapp events?
Do you want / are you able / are you allowed to debug hanging application(s)?

0
 
LVL 61

Expert Comment

by:gheist
ID: 34998879
pmirem yields nothing on google, so it seems like in-house application, which you can ask original developer to debug....
0
 
LVL 3

Author Comment

by:David Brennan
ID: 34998899
Original developer is gone bust.... but we're in healthcare so we can't just discontinue using the app. We have a consultancy firm looking at it but to be honest they're leaving a lot of this on me, even though I'm not highly trained in any of this stuff. But I'm sure we'll probably figure it out soon. I will keep you posted with info re hotix etc
0
 
LVL 61

Accepted Solution

by:
gheist earned 500 total points
ID: 34999014
So it is about to support legacy app

Get your tools ready:
http://www.dependencywalker.com/ - to analyze binary BEFORE running
http://technet.microsoft.com/en-gb/sysinternals/bb896653 - to see what modules are loaded
http://www.ollydbg.de/ - to catch module in which app crashes

1st tool:
MSVCRT 70 may point to w2000 compatibility mode
71 to XP
60 to NT4
i.e see what modules are being loaded and if there are no two versions of same module in path
2nd tool:
you cn see in which module there is a stuck thread, analyze with (1) again
3rd
catches crash and you can find crashing module


e.g:
app x loads DB driver dll which has msvcrt in its own dir and then uses same module from system dir - memory image content is non-determinable, and app crashes sooner or later.

Good luck in debugging, tell me if you need more assistance.

general advice is to make single application use same copy of same .dll at all times, preferably one in system32 directory.

0
 
LVL 3

Author Comment

by:David Brennan
ID: 34999123
Sorry man, that's way above my head. I might leave it to the consultancy firm.. I'll push them to analyse the problem themselves instead of dumping it on me. Third party software is a pain in the &£^$&^$
0
 
LVL 61

Expert Comment

by:gheist
ID: 34999157
Recompiling software in latest visual studio may help.
0
 
LVL 3

Author Comment

by:David Brennan
ID: 34999169
we don't have the source code. closed source commercial app.
0
 
LVL 61

Expert Comment

by:gheist
ID: 35005616
Ask for compatible version?

Please use (1) on crashing binaries and apply compatibility mode as suggested.
If you still experience crashes consider (3) or leave for consultants

Dont give up so quickly....

Please please please ;)

I find your problem common and "interesting"...
0
 
LVL 3

Author Comment

by:David Brennan
ID: 35005748
1. Unfortunately I cannot analyse the app 'before running', as it runs 24 / 7 / 365.

2. Tasklist /m (should be same as process explorer) gives me the following dependencies for ibserver.exe

ibserver.exe                  1076 ntdll.dll, kernel32.dll, GDI32.dll,        
                                   USER32.dll, ADVAPI32.dll, RPCRT4.dll,      
                                   Secur32.dll, MSVCRT.dll, COMCTL32.dll,      
                                   WS2_32.dll, WS2HELP.dll, MPR.dll,          
                                   SHELL32.dll, SHLWAPI.dll, IMM32.DLL,        
                                   comctl32.dll, libborland_lm.dll,            
                                   apphelp.dll, msctfime.ime, ole32.dll,      
                                   DNSAPI.dll, mswsock.dll, winrnr.dll,        
                                   WLDAP32.dll, rasadhlp.dll, hnetcfg.dll,    
                                   wshtcpip.dll, gdsintl.dll, ksudflib.dll,    
                                   oleaut32.dll, gds32.dll

so no MSVCRT....

unfortunately I think I will have to leave number 3 to the consultants because I'm not a C++ or C# or even a Delphi developer ( I'll stick to PHP ) and I'm not sure how the program works.

At the moment there are no errors or warnings in the event log since at least the 24th of February. It must have been some problem at the specific time in the logs. Thanks for your help but I'm not sure I'm going to figure this out through looking at the app. I've a feeling the logons in the event viewer can be attributed to one or more of the other 'interfaces' that connect back to the firebird database. The original software company created a few of these 'interfaces' that pump data in and out of the database to integrate with other softwares. I should be able to track it down eventually. Your steps 1, 2 and 3 I would imagine will come in very useful for troubleshooting a lot of software problems, so thank you.

Also, do you have any idea how to answer my original question - i.e. is there any way to track past logons from the specific user that was appearing in the event viewer -  back to a hostname (point of origin)
0
 
LVL 61

Expert Comment

by:gheist
ID: 35006181
apphelp.dll ?

So you ARE actually using compatibility mode.... (either by MS o

Check msvcrt.dll around
It is shared component ov visual C 4-6 (v6 provided also numbered component)
It is from times of NT4-W2000

Check if there are multiple versions of this dll on the system.

Rename those in program directories, best is in system32 which may not load compatibility mode.
0
 
LVL 61

Expert Comment

by:gheist
ID: 35006189
3 tools i mentioned show actual file being loaded in process image... it is more informative.
0
 
LVL 3

Author Comment

by:David Brennan
ID: 35006415
Yes I realise your recommended tools are better but as I've said, I cannot restart the software while it is running, it is important to keeping our X-Ray department functioning. I think I'm going to have to just close this question off and give you the points. I'm not really allowed to start renaming DLLs etc on major systems - I'll leave it to the software consultants.
0
 
LVL 61

Expert Comment

by:gheist
ID: 35006585
Wait
Do static analysis on binary with depends.exe
If you discover dual msvcrt you can plan 2x5min downtime to resolve and/or revert in case of failure.
That program crashing is real pain in the a--.
Can you repeat crash in non-production vmware? It could be a good starting point for debugger.
Once you make it stable everybody will thank you.
0
 
LVL 61

Expert Comment

by:gheist
ID: 35006614
You can see the result immediatly - if app does not load apphelp.dll it runs in native mode (as microsoft wants it to be)
Give it a try... Ask some doctor for test case when it crashes, it might give you volunteer to attach debugger and test the solution.

Mistake was done by letting app in compatibility mode in production. You can only correct it. It cannot get worse.
BUT write down what you plan to do and ask somebody to watch if you do it right.
0
 
LVL 3

Author Comment

by:David Brennan
ID: 35006805
ok I'll take a look.
0
 
LVL 61

Expert Comment

by:gheist
ID: 35006842
I think consultants outside microsoft or interbase official support cannot help much
Do paperwork - really, if after 2 years you will go for w2008 R3 you know what i mean...
I ask moderators to call more attention to this question.
0
 
LVL 61

Expert Comment

by:gheist
ID: 35012406
Similar problem with succesfully avoiding DLL conflicts
http://www.experts-exchange.com/Software/Server_Software/Web_Servers/Apache/Q_26843009.html
Luckily did not get far enough to run debuggers, depends.exe and process explorer.

0

Join & Write a Comment

Suggested Solutions

Title # Comments Views Activity
LDAP and ADFS 1 25
ACTIVE DIRECTORY 3 29
active directory 11 25
GPO Delegation 4 16
Installing a printer using group policy preferences is not that hard let’s take a look at it. First lets open up your group policy console and edit the policy you want to add it to. I recommend creating a new policy for each printer makes it a l…
[b]Ok so now I will show you how to add a user name to the description at login. [/b] First connect to your DC (Domain Controller / Active Directory Server) SET PERMISSIONS FOR SCRIPT TO UPDATE COMPUTER DESCRIPTION TO USERNAME 1. Open Active …
This tutorial will walk an individual through the process of transferring the five major, necessary Active Directory Roles, commonly referred to as the FSMO roles from a Windows Server 2008 domain controller to a Windows Server 2012 domain controlle…
This tutorial will walk an individual through the process of transferring the five major, necessary Active Directory Roles, commonly referred to as the FSMO roles to another domain controller. Log onto the new domain controller with a user account t…

760 members asked questions and received personalized solutions in the past 7 days.

Join the community of 500,000 technology professionals and ask your questions.

Join & Ask a Question

Need Help in Real-Time?

Connect with top rated Experts

21 Experts available now in Live!

Get 1:1 Help Now