July 6, 2008 02:48pm pdt

1. lnkevin 308,533
2. andyalder 162,052
3. leew 71,459
4. KCTS 52,801
5. rindi 51,081
6. RobWill 44,254
7. meyersd 41,397
8. PUNKY 38,733
9. sifuedition 34,898
10. tigermatt 34,020
11. akirhol 28,635
12. SysExpert 28,513
13. ryansoto 26,805
14. Callandor 23,698
15. TechSoEasy 23,525

1. lnkevin 537,184
2. andyalder 403,060
3. leew 137,270
4. rindi 112,488
5. KCTS 108,735
6. SysExpert 88,566
7. PUNKY 70,283
8. meyersd 65,697
9. 65td 62,400
10. RobWill 59,454
11. JjcampNR 50,660
12. TechSoEasy 49,843
13. jamietoner 47,847
14. tigermatt 38,560
15. Callandor 37,538

08.22.2007 at 06:56AM PDT, ID: 22779445

	[x] Attachment Details

[x]

The Solution Rating System

With so many solutions, how can you tell which solutions are most likely to help you and which ones are not? To provide you with a tool to use, we rate our solutions based on various elements that most accurately determine if a solution is a quality solution. To explain what factors affect the solution rating, here are the elements we take into consideration when formulating our solution rating.

The Grade of the Solution
The Zone Rank of the Expert Providing the Solution
The Number of Author and Expert Comments
The Number of Experts Contributing
The Feedback of the Community

Your Input Matters
Because of the way the system is set up, the most important variable in this equation is you. As a member of Experts Exchange, you are able to cast your vote on the quality of the solutions in regard to how complete, accurate, helpful and easy to understand each solution is. When you provide your feedback, each rating is adjusted accordingly. So, if you see a solution that has a poor rating that you think is a good solution, let us know by rating it. As you do, the rating will be adjusted and will become more accurate for other members of our site.

If you have any suggestions that you would like to make for our rating system, please ask a question in the Suggestions Zone of Community Support.

Thank you!

7.7

Memory Leak

Zones: Computer Servers, Windows 2003 Server, Backup Exec

Tags: memory, obci, leak

Hi,

Hope you can help as this problem has been ongoing now for a number of months, I have joined this site specifically to fix this problem. As it is slowly driving me insane !

I have a HP Prolient DL360 G5 server that is constantly crashing and becoming unstable until rebooted. I have noticed that the server appears to have a memory leak I think being caused from backupexec.

The server is used for our main File and Print server, it has on it Windows 2003 Std R2 SP2. Also on the server is AVG Anti Malware Edition, Backupexec remote client, several printer drivers. Attached to the server is a MSA30 that houses 14 x 300GB HDDs. This is split up into two partitions (Partition A = 2TB) (Partition B = 1.6TB)

The problem suddenly started occurring one weekend after I noticed the server had stopped responding after the weekly fulls had run. Communication errors where all over backupexec (which is run from a different server) the event log reported eventid errors 2019 (The server was unable to allocate from the system nonpaged pool because the pool was empty). Once the server is rebooted all is fine. Also we only get this problem when we backup a certain area on Partition B.

I suspected one of the HDDs in the MSA may have been faulty so I moved the data from Partition B to Partition A. I then retried the backup with the same problem 2019 errors.
The area that has the problem contains 5,373,480 files in 56,860 folders and is 163GB in size.

I didnt want to have too much downtime and I had a spare HP DL360 G5 so I thought over the weekend I would build that to be the new file and print server.
OK new server in place with all upto date drivers. I transferred the array card and attached the MSA30 to the new server. The new server has all the same software installed and I used print migrator to move over the printers and drivers.

Tested a backup of the same area, same problem ! When I run the utility poolmon I get these results after I have run a backup job which backed up 36GB of data from the affected area taking 2hours to run&

Memory: 2094964K Avail: 730620K PageFlts: 2 InRam Krnl: 3088K P:44888K
Commit:1198724K Limit:6208524K Peak:1238336K Pool N:104436K P:46000K
System pool information
Tag Type Allocs Frees Diff Bytes Per Alloc

ObCi Nonp 975195 ( 0) 46 ( 0) 975149 54608344 ( 0) 56
CMVa Paged 207932 ( 0) 200549 ( 0) 7383 420896 ( 0) 57
File Nonp 19263397 ( 40) 19258697 ( 40) 4700 716128 ( 0) 152
Even Nonp 232251 ( 1) 228670 ( 1) 3581 175008 ( 0) 48
Vad Nonp 1036097 ( 0) 1032545 ( 0) 3552 170496 ( 0) 48
Ntfr Nonp 19519 ( 0) 16050 ( 0) 3469 222984 ( 0) 64
CMDa Paged 30741 ( 0) 27448 ( 0) 3293 554320 ( 0) 168
Mdl Nonp 17544 ( 0) 14320 ( 0) 3224 438464 ( 0) 136
CMNb Paged 45037667 ( 118) 45035315 ( 118) 2352 115256 ( 0) 49
CMVI Paged 21171 ( 0) 19046 ( 0) 2125 77320 ( 0) 36
Ntfn Nonp 20023 ( 0) 17962 ( 0) 2061 83328 ( 0) 40
NtFs Nonp 1610605 ( 1) 1608569 ( 0) 2036 85184 ( 40) 41
VadS Nonp 103666 ( 1) 101688 ( 1) 1978 63296 ( 0) 32
TCPc Nonp 55505 ( 0) 53730 ( 0) 1775 85200 ( 0) 48
TCPC Nonp 3851 ( 0) 2164 ( 0) 1687 143272 ( 0) 84
ReTa Nonp 8673 ( 0) 7007 ( 0) 1666 66992 ( 0) 40
FSfm Nonp 17063 ( 0) 15450 ( 0) 1613 64520 ( 0) 40
MmSt Paged 1059985 ( 0) 1058378 ( 0) 1607 10735368 ( 0) 6680
FSim Paged 16539 ( 0) 15012 ( 0) 1527 195456 ( 0) 128
ObNm Paged 3624368 ( 9) 3622859 ( 9) 1509 76968 ( 0) 51
ObDi Paged 24821 ( 0) 23337 ( 0) 1484 35616 ( 0) 24
IoNm Paged 8385727 ( 14) 8384258 ( 14) 1469 173632 ( 0) 118
ReEv Nonp 6505 ( 0) 5111 ( 0) 1394 33456 ( 0) 24
Key Paged 47649884 ( 123) 47648494 ( 123) 1390 144488 ( 0) 103
Irp Nonp 9651002 ( 0) 9649654 ( 0) 1348 556208 ( 0) 412
Vadl Nonp 168666 ( 5) 167351 ( 5) 1315 84160 ( 0) 64
Sema Nonp 2666 ( 0) 1362 ( 0) 1304 73344 ( 0) 56
MmCm Nonp 49456 ( 0) 48280 ( 0) 1176 17381168 ( 0) 14779

&As you can see the ObCi is taking up a lot of memory and never frees it up. Even when the backup job completes. I can stop and start the backupexec remote client service and the memory is still not released. The only way to release the memory is to reboot the server. I am not sure what ObCi is to be honest.

I have a call logged with HP but to be honest they just keep sending me round in circles.

I hope I have provided enough information for you to help, if you require and further information please let me know what you need.

Thank you in advance for your help with this.

Lee.

Start your free trial to view this solution

Question Stats

Zone: Computer Hardware

Question Asked By: lee_manton

Solution Provided By: impactuser

Participating Experts: 9

Solution Grade: A

Translate:

Loading Advertisement...

Microsoft

Internet Protocols
Applications

Development
OS

Hardware
Windows Security

Apple

Operating Systems
Hardware

Programming
Networking

Software

Internet

Search Engines
File Sharing
WebTrends / Stats
Spy / Ad Blockers

Web Browsers
New Net Users
Web Development
Chat / IM

Anti Spam
Web Servers
Anti-Virus
Email Clients

Gamers

Tips
Online / MMORPG
Puzzle
Emulators

Action / Adventure
Role Playing
Consoles
Game Programming

Strategy
Sports
Misc
Computer Games

Digital Living

Hardware
New Net Users
New Users

Software
Digital Music
Gaming World

Home Security
Apple
Networking Hardware

Virus & Spyware

Vulnerabilities
IDS
Encryption
Anti-Virus

Operating Systems Security
Software Firewalls
WebApplications
Cell Phones

Operating Systems
Internet
Hardware Firewalls

Hardware

Handhelds / PDAs
Displays / Monitors
Components
Networking Hardware

Peripherals
Laptops/Notebooks
Storage
Servers

Desktops
New Users
Misc
Apple

Software

System Utilities
Industry Specific
Network Management
Photos / Graphics
Page Layout
VMWare
Misc
Web Development

OS
CYGWIN
Voice Recognition
Message Queue
Quality Assurance
Security
Firewalls
MultiMedia Applications

Development
Database
Office / Productivity
Business Management
OS/2 Apps
Server Software
Internet / Email

ITPro

OS
Storage
Encryption
Operating Systems Security
Apple Hardware
Laptops & Notebooks
Servers
Networking Hardware
Peripherals

Devices
Displays / Monitors
WebTrends / Stats
Search Engines
Firewalls
WebApplications
IDS
Vulnerabilities
Email Clients

File Sharing
Spy / Ad Blockers
Web Browsers
Web Servers
Networking
Anti-Virus
Chat / IM
Anti Spam

Developer

Web Servers
Web Browsers
Game Programming
Dev Tools
Industry Specific
Office / Productivity

Database
CYGWIN
Web Development
Search Engines
File Sharing
WebTrends / Stats

Programming
Content Management
Application Servers
Protocols

Storage

Removable Backup Media
Storage Technology
Servers

Grid
Remote Access
Backup / Restore

Misc
Hard Drives

Miscellaneous
Security
Development
Linux
VMWare

MainFrame OS
Unix
Apple
OS / 2
AS / 400

BeOS
Microsoft
VMS / OpenVMS

Database

Oracle
Miscellaneous
MySQL
Software
Sybase
Contact Management
PostgreSQL
Data Manipulation
Clarion
InterSystems Cache

Siebel
MUMPS
OLAP
SQLBase
SAS
GIS & GPS
4GL
Berkeley DB
DB2
Informix

Interbase / Firebird
FoxPro
Reporting
LDAP
Filemaker Pro
MS SQL Server
dBase
MS Access

Security

Misc
Web Browsers
Software Firewalls
Operating Systems Security
File Sharing

Spy / Ad Blockers
Vulnerabilities
WebApplications
IDS
Anti-Virus

Encryption
Anti Spam
Email Clients
VPN
Chat / IM

Programming

Editors IDEs
Installation
Handhelds / PDAs
Multimedia Programming
System / Kernel

Algorithms
Game
Signal Processing
Project Management
Open Source

Database
Misc
Languages
Processor Platforms
Theory

Web Development

Scripting
Blogs
Web Servers
Software
Search Engines
Web Graphics
Images

Internet Marketing
Images and Photos
Components
Document Imaging
Web Languages/Standards
Illustration
WebApplications

Fonts
WebTrends / Stats
Authoring
Digital Camera Software
Miscellaneous

Networking

Protocols
Apple Networking
Network Management
Message Queue
Application Servers
Content Management
File Servers
Email Servers
Misc
Java Editors & IDEs

Wireless
Networking Hardware
Backup / Restore
System Utilities
ISPs & Hosting
Web Servers
Storage Technology
Removable Backup Media
Servers
Broadband

Grid
OS / 2
Novell Netware
Unix Networking
Windows Networking
Security
Telecommunications
Operating Systems
Linux Networking

Other

Community Advisor
Lounge
Community Support
New Net Users

Philosophy / Religion
Math / Science
Miscellaneous
URLs

Expert Lounge
Politics
Puzzles / Riddles

Community Support

Suggestions
New to EE
New Topics
Community Advisor

CleanUp
Announcements
General

Feedback
Input
EE Bugs

08.22.2007 at 07:36AM PDT, ID: 19746509

Rank: Master

ocon827679:

The only thing I found on ObCi is non-paged object creation memory tag.
Try stopping your backup exec services and see if the memory leak halts. Maybe try stopping services one at a time to see if you get a response if the backup exec services don't show anything.

08.22.2007 at 07:41AM PDT, ID: 19746564

lee_manton:

OK thanks,

I have tryed stopping the backupexec services as stated. Once the backupexec services are stopped ObCi stops going up but never releases the memory.I will try stopping other services to see if the memory is returned. Unfortunatly as this is a live server I will have to wait until after 5pm to do this testing.

Thank you for the swift reply.

Kind Regards.

08.22.2007 at 07:45AM PDT, ID: 19746624

Rank: Master

ocon827679:

Sounds like backup exec is your problem. You may want to check the Veritas support site (or give them a call) and see if they have a fix.

08.22.2007 at 07:53AM PDT, ID: 19746684

lee_manton:

I have been through the veritas support site with no joy. I will log a call with them now.

08.22.2007 at 09:24AM PDT, ID: 19747599

chafer:

I think I may see what's happening. My understanding is that for every file that Backup Exec backs up, it needs to track that file in RAM. ObCi is the "object creation lookaside list" which I believe is tracking objects created (for example pointers to each file). While the 163 GB of data is not that large, over 5 million files is massive. It's not the volume of data causing the problem, it's the number of files (kind of like the old file handles problem back in DOS, but more complicated). The only answer I see for backing up that many files is to add RAM. Unless I'm misreading things, you have 2GB of RAM on your server. For Backups of that size, that's very tight. Backup Exec needs RAM just for temporary space and to track things. I'd up the RAM to 4GB and see what happens. I do not believe that this is a memory leak in the classic sense of the word. Memory leaks are more than just taking up big blocks of RAM and not releasing it, they also cumulatively continue to add to the RAM used.

08.22.2007 at 02:01PM PDT, ID: 19749653

lee_manton:

Hi Chafer,

I suspected as much, I will order more RAM ASAP.Do you think its worth taking the box to 6GB of RAM as it has the space ? Thanks for your advice I think you have me going in the right direction now.

Cheers !!!

08.22.2007 at 02:09PM PDT, ID: 19749710

chafer:

6GB is great, but what version of Windows are you using. Windows 2003 Standard version only supports 4GB. To go above that, you need Windows 2003 Enterprise. If you already have Enterprise, then definately go to 6GB. If you don't you are looking at a two to three thousand dollar upgrade to purchase Enterprise.

08.22.2007 at 02:11PM PDT, ID: 19749727

chafer:

I just took a look at your post again. You did say that you were using Standard - I just missed that. Unless you upgrade to Enterprise, you are stuck at 4GB. I suspect that 4GB will be sufficient. Make sure that you have a 4GB paging file as well. I'd recommend setting the top and bottom numbers both to 4095 (or 4092 if it won't take 4095).

Considering the price of RAM versus the price of an Enterprise upgrade, I'd start with the RAM upgrade and see if it works.

08.23.2007 at 12:41AM PDT, ID: 19752192

lee_manton:

OK no problem thanks. I'll get another 2GB of RAM ordered today. Paging file is already set to 4096.

09.03.2007 at 02:32AM PDT, ID: 19819817

lee_manton:

Hi,

Just to keep you updated...

I have added another 2GB of RAM and re-ran the backup jobs over the weekend. Unfortunatly I am still getting the same errors. event id 2019 in the system log. and the server stops responding and needs to be rebooted to get it back online. Any other idea's anyone ? Thanks in advance.

09.04.2007 at 05:27AM PDT, ID: 19824643

Rank: Master

ocon827679:

Did you ever get a hold of Veritas support? Have you tried reloading Backup Exec?

09.04.2007 at 05:32AM PDT, ID: 19824669

lee_manton:

Veritas could not sugest anything, the call is still open. I have tried removing backupexec and reinstalling.
Backupexec is controlled from a different server its only the remote agent thats on the effected server. I have even upgraded backupexec from version 10d to 11d to see if this helped the problem. It didn't :-(

09.04.2007 at 10:41AM PDT, ID: 19826885

Rank: Sage

honmapog:

Do you have any anti-virus software running?
If you have, it may be a good idea to disable it during the backup to see what happens.

09.05.2007 at 12:38AM PDT, ID: 19830530

lee_manton:

I have AVG Anti-Malware edition running but I will remove it from the backup server and the file server while I run a scan. However this Anti virus software has been on the server long before the problem started happening. Will give it a try tho, thanks. Will let you know.

11.02.2007 at 06:00PM PDT, ID: 20205522

seeimred:

Lee, I have also been battling a plague of nonpaged pool errors on a variety of HP proliant models & configurations. 2 cluster pods, several stand alone, all exchange 2003 mailboxes & public folders, etc. This has been going on for months and since day one of install. I really understand your pain 10 fold. In my quest, I've worked with MS PSS but mostly on my own and here's somethings I have learned that I think will help you. Poolmon is good. When you start it, hit P to filter by paged/nonpaged, then B to sort by bytes. This will give you the most likely culprit at the top. In our case it's always MMCM. On our PF servers, I got the most improvement when I dissolved the NIC team for the dedicated netbackup connection. We also dissolved the backup team on a cluster pod that could not keep the HTTP resources alive due to the NPP depletion. It also helped to bring down the NPP to the 95 Mb range where before it was at 109+ Mb.

Now I also noticed that you are running SP2 for Windows 2003. Well, were you aware of the TCPChimney Feature introduced in the Scalable Network Pack which was added in to SP2? If not, I have a feeling that you are experiencing the impact. You may find http://support.microsoft.com/kb/944011 very enlightening to your situation. HP Proliants are very impacted as they have usually have the Broadcom adapters with drivers that enable hardware Rx/Tx TCP Offloading. So, in a nutshell, if you have Windows still performing this offloading as well as the NIC drivers, the nonpaged pool is going to get hammered.

Another area you should check is again on the NIC settings. HP released a customer advisory not too long ago describing a problem with Proliants & NPP errors when you use Jumbo frames. They also recommend you change the driver for the ATI display adapter to use the generic Windows Standard VGA driver.

Hope this gives you some direction and a glimmer of hope.

11.03.2007 at 08:07PM PDT, ID: 20209096

TreborNimar:

I am having virtually the exact same problem on one of my clients' servers. The server is custom-built by me using an Intel motherboard, and I am running both AVG AntiMalware and Backup Exec on Windows 2003 Server sp2. The problem started around a week after I replaced Symantec Antivirus CE 10.1 with AVG. Now, every 7-10 days the server stops authenticating users and I get Event ID 2019s. Rebooting resolves the problem for another 7-10 days.

Using poolmon.exe I too found ObCi slowly accumulating non-paged kernel memory and giving up almost nothing in return. After eliminating all non-essential services (including Backup Exec) I found that uninstalling AVG was the only way to stop the memory leak. I've got a support ticket in with AVG, but they are less than helpful, and insist on me sending them a dmp file, even though I've explained that the server never actually crashes and produces the dmp file. I am currently running AVG in debug mode and will send them the log after letting the server crash, although that won't make my client very happy. Disabling BackupExec for several days did not seems to slow the memory leak down at all.

I install AVG on almost all of the servers I make so I checked the status of a group of them and found that in half of them, ObCi was far and away the highest non-paged kernel memory hog, with almost no frees compared with allocs. However as far as I know, none of these other servers are crashing due to this problem.

So this issue may not be related to your server being an HP, but may have more to do with AVG AntiMalware. I hope some of this info helps.

11.05.2007 at 10:50PM PST, ID: 20221826

Antarctic_Tech:

I'll join this group with similar issues.......2019 errors & reboot every three days.
Running Mail & Web Marshal, Sophos Antivirus and Shadow Protect.

My service providers logged a job with Microsoft APAC who pointed the finger at network drivers
after getting me to run some utils from tjeir site and me sending the associated logs.

"These problems that you have been seeing are due to a clear leak in the Non-Paged Pool tag NTID.
This tag has previously been associated with the driver cpqteam.sys, which is HPs Network Teaming Intermediate Driver (NTID), and this issue has been previously seen in the version that you have (dated 7th March 2007)."

Problem is downloading HP's latest drivers completely blows away the network cards and I'm currently trawling for any idea's on this site. (with the server dead) Are any of you running the network cards in a teamed environment ? - if so, try dropping the teaming config

My OLD (& apparently faulty) versions are:
HP NC373i Multifunction Gigabit Server Adapter - driver 3.0.5.0 22/12/2006
HP Network Team #1 - 8.60.0.0 7/3/2007
(that's DD/MM/YYYY format for you US folks!)

11.05.2007 at 10:53PM PST, ID: 20221831

Antarctic_Tech:

P.S - That was a Proliant DL360 G5 running Windows 2003 SP2 with 4 Gig RAM