Solved

Error message on Cisco 871w router, what does it mean and how can i fix it?

Posted on 2010-09-16
28
2,481 Views
Last Modified: 2012-05-10
It had been running fine until today when I got this error. I reloaded the router and I got a connection back but it was running very slow.

Thanks.  
*Apr  5 22:05:55.761: %SYS-2-MALLOCFAIL: Memory allocation of 65536 bytes failed
 from 0x8004F030, alignment 16
Pool: Processor  Free: 72632  Cause: Memory fragmentation
Alternate Pool: None  Free: 0  Cause: No Alternate pool
 -Process= "IP ARP Adjacency", ipl= 0, pid= 76,  -Traceback= 0x807D7100 0x8003CA
D0 0x80042968 0x8004F034 0x80050100 0x80575D6C 0x809A1DC4 0x809AECCC 0x805EF2CC
0x809DBC3C 0x80C2F330 0x800225CC 0x80025710
*Apr  5 22:05:55.761: %FIB-3-NOMEM: Malloc Failure, disabling CEF,  -Traceback=
0x807D7100 0x809A1F48 0x809AECCC 0x805EF2CC 0x809DBC3C 0x80C2F330 0x800225CC 0x8
0025710

Open in new window

0
Comment
Question by:fratapia
  • 13
  • 13
  • 2
28 Comments
 
LVL 3

Expert Comment

by:kf4zmt
ID: 33697944
Here is a web page on Cisco's web site that may be helpful for you.

http://www.cisco.com/en/US/products/sw/iosswrel/ps1831/products_tech_note09186a00800a6f3a.shtml
0
 
LVL 3

Expert Comment

by:kf4zmt
ID: 33697951
Cisco also has this to say about memory fragmentation:

Memory Fragmentation Problem or Bug

This situation means that a process has consumed a large amount of processor memory and then released most or all of it, leaving fragments of memory still allocated either by this process, or by other processes that allocated memory during the problem. If the same event occurs several times, the memory may fragment into very small blocks, to the point where all processes requiring a larger block of memory cannot get the amount of memory that they need. This may affect router operation to the extent that you cannot connect to the router and get a prompt if the memory is badly fragmented.

This problem is characterized by a low value in the "Largest" column (under 20,000 bytes) of the show memory command, but a sufficient value in the "Freed" column (1MB or more), or some other wide disparity between the two columns. This may happen when the router gets very low on memory, since there is no defragmentation routine in the IOS.

If you suspect memory fragmentation, shut down some interfaces. This may free the fragmented blocks. If this works, the memory is behaving normally, and all you have to do is add more memory. If shutting down interfaces doesn't help, it may be a bug. The best course of action is to contact your Cisco support representative with the information you have collected.
0
 
LVL 12

Expert Comment

by:acl-puzz
ID: 33700045
it seems like memory is malfunctioning try different memory!
0
 

Author Comment

by:fratapia
ID: 33700372
@kf4zmt  
all the interfaces are on but only the fa4 and fa0 are actually used.  do you think it will make a difference if i shut down fa1 - fa3.

@acl-puzz
is it the dram i must change?

0
 
LVL 3

Expert Comment

by:kf4zmt
ID: 33700638
If they aren't being used, go ahead and shut them down.  Maybe it will help.  However, I have never seen the problem you are describing.  I think you probably have a hardware problem. Maybe some bad RAM.  It could also be a bug in the Cisco IOS.  You might want to consider upgrading the IOS to a new version.  You could also swap out the RAM.
0
 
LVL 12

Expert Comment

by:acl-puzz
ID: 33701619
yeah just try using another if u have or use another IOS image it may corrupted as well or just  asking for update but before anything  backup everything!!

Link provided in first comment is very informative.
0
 

Author Comment

by:fratapia
ID: 33701709
Do i have to buy an upgrade of the IOS from Cisco?  
How do i reflash the same version of IOS?
0
 
LVL 3

Expert Comment

by:kf4zmt
ID: 33701753
Do you have a Cisco CCO login?  If so, you can download the latest IOS for free.  Alternatively, if you have another router that runs a compatible IOS, you could copy it from there.
0
 

Author Comment

by:fratapia
ID: 33704500
i do, there is a bunch that say 12.4 for 871w, not sure which is the right one?
0
 
LVL 3

Expert Comment

by:kf4zmt
ID: 33704533
That depends on what functionality you need.  If you run the "sh ver" command, it will show you the version of the IOS you already have.  It might look something like this...

System image file is "flash:c2600-ik9o3s3-mz.123-26.bin"

In this example, the IOS is version 12.3  (123-26).  The "ik9o3s3" part indicates various features of the IOS.  By looking at the version you already have, you should be able to find a file name that has similar details on the CCO website.

Just make sure that you have enough RAM & flash memory to support whatever IOS you download.
0
 

Author Comment

by:fratapia
ID: 33718006
This is kinda funny, not really.  
So I had a new cisco 881w that i was going to install in a new office.  Instead i installed it instead of the 871w because i needed this place working on the weekend.  This morning I get a call, "the internet is down".
I come into the office and bring it up the the terminal and see this error.  


*Sep 20 15:43:30.669: %SYS-2-MALLOCFAIL: Memory allocation of 1080 bytes failed
from 0x802EC664, alignment 0
Pool: Processor  Free: 551832  Cause: Memory fragmentation
Alternate Pool: I/O  Free: 5624  Cause: Memory fragmentation
 -Process= "Pool Manager", ipl= 0, pid= 5,  -Traceback= 0x81EC3F4Cz 0x81EE4870z
0x82C79250z 0x802EC668z 0x81F023FCz 0x81F026B0z 0x8045F1ECz 0x80462214z
comped>
*Sep 20 15:44:00.893: %SYS-2-MALLOCFAIL: Memory allocation of 1080 bytes failed
from 0x802EC664, alignment 0
Pool: Processor  Free: 551832  Cause: Memory fragmentation
Alternate Pool: I/O  Free: 5624  Cause: Memory fragmentation
 -Process= "Pool Manager", ipl= 0, pid= 5,  -Traceback= 0x81EC3F4Cz 0x81EE4870z
0x82C79250z 0x802EC668z 0x81F023FCz 0x81F026B0z 0x8045F1ECz 0x80462214z


I shut the down the terminals I don't use like you guys had suggested.  Can you guys look over my config and help me find if there is something that should not be, or that should?

Also, besides the log on the cisco.  What can I use to monitor the connection in the office, something that can maybe notify me if the router encounters an error?

Oh, also, this is the weird thing. I was able to remote desktop to my computer in the office all along.  From my house and even my cell phone.
comped#show running-config
Building configuration...

Current configuration : 5108 bytes
!
! Last configuration change at 16:20:44 UTC Mon Sep 20 2010
!
version 15.0
no service pad
service timestamps debug datetime msec
service timestamps log datetime msec
no service password-encryption
!
hostname comped
!
boot-start-marker
boot-end-marker
!
no logging buffered
enable secret 5 $1$V7Ju$OUFJaxzx30cdFcC.Pdvea1
enable password 
!
no aaa new-model
memory-size iomem 10
!
crypto pki trustpoint TP-self-signed-1375774997
 enrollment selfsigned
 subject-name cn=IOS-Self-Signed-Certificate-1375774997
 revocation-check none
 rsakeypair TP-self-signed-1375774997
!
!
crypto pki certificate chain TP-self-signed-1375774997
 certificate self-signed 01
  3082023E 308201A7 A0030201 02020101 300D0609 2A864886 F70D0101 04050030
  31312F30 2D060355 04031326 494F532D 53656C66 2D536967 6E65642D 43657274
  69666963 6174652D 31333735 37373439 3937301E 170D3130 30393230 31363034
  35395A17 0D323030 31303130 30303030 305A3031 312F302D 06035504 03132649
  4F532D53 656C662D 5369676E 65642D43 65727469 66696361 74652D31 33373537
  37343939 3730819F 300D0609 2A864886 F70D0101 01050003 818D0030 81890281
  8100E3E6 C4A57149 5FB0999F E96D5135 88C84E09 38626924 318598F9 EBCF179E
  82B152C3 2DB5034D 48FC49AC 5C038C65 938A9AE0 3893BF48 30C90477 C7B5A93E
  367EB786 6BCCE2CA 7B23FABE B2D1874B 39740429 5EB81833 1AC15FF7 729D36DB
  20FFF44A 36FBA654 1888AB06 20E552FF 1F4D4A9A 9477AEBA 1AC3C929 FC1D4F06
  2BCD0203 010001A3 66306430 0F060355 1D130101 FF040530 030101FF 30110603
  551D1104 0A300882 06636F6D 70656430 1F060355 1D230418 30168014 D173BF31
  A24DDD1B 94364144 6AAD0B2C 31AB0FCA 301D0603 551D0E04 160414D1 73BF31A2
  4DDD1B94 3641446A AD0B2C31 AB0FCA30 0D06092A 864886F7 0D010104 05000381
  81008D30 AA1FF20A EF6EAC44 1602992F D57D1D74 8FD4D477 40665FDB 061B0E00
  B905212C 81A81F7D 08E38B8C 5F8501AB 618AA3AE 55FFD1EA B3FD4048 2BE070F3
  A59F7FF3 7E69B4FD 033043B4 B1BE6FF1 0E64C69E E6D65B1B 3CBE2E2E 1CAB7BDF
  E476BCF1 A2DE9B17 16EDC4D9 F02E437C 29FDD0E7 DD09FE75 4FDF0B9C 9BB4AE82 ED6A
        quit
ip source-route
!
!
!
!
ip cef
ip name-server 167.206.112.138
ip name-server 167.206.7.4
no ipv6 cef
!
!
license udi pid CISCO881W-GN-A-K9 sn FTX143081BU
!
!
username admin privilege 15 password 0 
!
!
!
!
crypto isakmp policy 1
 encr 3des
 authentication pre-share
 group 2
crypto isakmp key  address 
crypto isakmp key  address 
!
!
crypto ipsec transform-set ESP-3DES-SHA esp-3des esp-sha-hmac
crypto ipsec transform-set ESP-3DES-SHA1 esp-3des esp-sha-hmac
crypto ipsec transform-set ESP-3DES-SHA2 esp-3des esp-sha-hmac
!
crypto map SDM_CMAP_1 1 ipsec-isakmp
 description Tunnel to
 set peer 
 set transform-set ESP-3DES-SHA1
 match address 102
crypto map SDM_CMAP_1 2 ipsec-isakmp
 description Tunnel to
 set peer 
 set transform-set ESP-3DES-SHA2
 match address 103
!
!
!
!
!
interface FastEthernet0
!
interface FastEthernet1
 shutdown
!
interface FastEthernet2
 shutdown
!
interface FastEthernet3
 shutdown
!
interface FastEthernet4
 ip address  255.255.255.248
 ip nat outside
 ip virtual-reassembly
 duplex auto
 speed auto
 crypto map SDM_CMAP_1
!
interface wlan-ap0
 description Service module interface to manage the embedded AP
 no ip address
 arp timeout 0
!
interface Wlan-GigabitEthernet0
 description Internal switch interface connecting to the embedded AP
!
interface Vlan1
 ip address 192.168.1.1 255.255.255.0
 ip nat inside
 ip virtual-reassembly
 ip tcp adjust-mss 1452
!
ip forward-protocol nd
ip http server
ip http authentication local
ip http secure-server
!
ip nat inside source route-map SDM_RMAP_1 interface FastEthernet4 overload
ip nat inside source static tcp 192.168.1.91 3389 96.56.178. 3395 extendable
ip nat inside source static tcp 192.168.1.92 3389 96.56.178. 3396 extendable
ip route 0.0.0.0 0.0.0.0 FastEthernet4 permanent
!
access-list 1 remark CCP_ACL Category=16
access-list 1 permit 192.168.1.0 0.0.0.255
access-list 100 remark CCP_ACL Category=4
access-list 100 remark IPSec Rule
access-list 100 permit ip 192.168.1.0 0.0.0.255 192.168.1.0 0.0.0.255
access-list 101 remark CCP_ACL Category=2
access-list 101 remark IPSec Rule
access-list 101 deny   ip 192.168.1.0 0.0.0.255 192.168.3.0 0.0.0.255
access-list 101 remark IPSec Rule
access-list 101 deny   ip 192.168.1.0 0.0.0.255 192.168.2.0 0.0.0.255
access-list 101 permit ip 192.168.1.0 0.0.0.255 any
access-list 102 remark CCP_ACL Category=4
access-list 102 remark IPSec Rule
access-list 102 permit ip 192.168.1.0 0.0.0.255 192.168.2.0 0.0.0.255
access-list 103 remark CCP_ACL Category=4
access-list 103 remark IPSec Rule
access-list 103 permit ip 192.168.1.0 0.0.0.255 192.168.3.0 0.0.0.255
!
!
!
!
route-map SDM_RMAP_1 permit 1
 match ip address 101
!
snmp-server community public RO
!
control-plane
!
!
line con 0
 no modem enable
line aux 0
line 2
 no activation-character
 no exec
 transport preferred none
 transport input all
line vty 0 4
 privilege level 15
 password 
 login local
 transport input telnet ssh
!
scheduler max-task-time 5000
end

Open in new window

0
 
LVL 3

Accepted Solution

by:
kf4zmt earned 500 total points
ID: 33718364
The "memory-size iomem 10" looks a bit suspicious.   Is there a reason it is set to 10%?   Here is a quote from Cisco's website.   According to this, the default is 25%.


memory-size iomem

To reallocate the percentage of DRAM to use for I/O memory and processor memory on Cisco 3600 series routers, use the memory-size iomem command in global configuration mode. To revert to the default memory allocation, use the no form of this command.

memory-size iomem i/o-memory-percentage

no memory-size iomem  i/o-memory-percentage
Syntax Description

i/o-memory-percentage
      

The percentage of DRAM allocated to I/O memory. The values permitted are 10, 15, 20, 25, 30, 40, and 50. A minimum of 4 MB of memory is required for I/O memory.

Defaults

The default memory allocation is 25 percent I/O memory and 75 percent processor memory.
0
 

Author Comment

by:fratapia
ID: 33719020
It had been set automatically when I programmed the router.  I did not set it.

Should I 'no' it and remove it?
0
 
LVL 3

Expert Comment

by:kf4zmt
ID: 33719060
I would.
0
Zoho SalesIQ

Hassle-free live chat software re-imagined for business growth. 2 users, always free.

 

Author Comment

by:fratapia
ID: 33719095
Done.

How can I see that it has made a diffrence or not?

How can I track it and is there a way for me to see if the router goes down and get notified?
0
 

Author Comment

by:fratapia
ID: 33721170
The router didn't crash but I got this error.  I was looking at the memory used on the Cisco CP and it was around 79% used when I notinced the error and then it was 40% but it didn't crash.  I could still get an internet connection.
Right now it's use is very low because nobody is here.


Do you think I should upgrade DRAM, that is what I'm getting from the troubleshooting page that I should do.

Where would it be best to get it?, which is the memory I should use.

0
 
LVL 3

Expert Comment

by:kf4zmt
ID: 33721177
Did you forget to include the error message?  I don't see it in your comments.
0
 

Author Comment

by:fratapia
ID: 33721185
I forget to attach the error


*Sep 20 19:56:27.915: %SYS-6-USING_RESERVE_MEMORY: Reserve memory is being used
*Sep 20 19:56:27.915: %SYS-6-RESERVE_MEMORY_CREATED: Reserve memory is being cre
ated
*Sep 20 19:56:27.955: %SYS-2-MALLOCFAIL: Memory allocation of 65536 bytes failed
 from 0x81EFD2D8, alignment 8
Pool: Processor  Free: 103408  Cause: Memory fragmentation
Alternate Pool: I/O  Free: 3348  Cause: Not enough free memory
 -Process= "Chunk Manager", ipl= 4, pid= 1,  -Traceback= 0x81EC3F4Cz 0x81EE603Cz
 0x81EFD2DCz 0x81EC2E20z 0x81EC2FACz 0x8045F1ECz 0x80462214z
*Sep 20 19:56:27.955: %SYS-2-CHUNKEXPANDFAIL: Could not expand chunk pool for ip
nat entry. No memory available -Process= "Chunk Manager", ipl= 4, pid= 1,  -Trac
eback= 0x81EC2FF4z 0x8045F1ECz 0x80462214z
*Sep 20 19:56:37.983: %SYS-2-CHUNKEXPANDFAIL: Could not expand chunk pool for ip
nat node. No memory available -Process= "Chunk Manager", ipl= 4, pid= 1,  -Trace
back= 0x81EC2FF4z 0x8045F1ECz 0x80462214z
*Sep 20 19:57:00.383: %SYS-2-MALLOCFAIL: Memory allocation of 21840 bytes failed
 from 0x82ADF11C, alignment 0
Pool: Processor  Free: 107820  Cause: Memory fragmentation
Alternate Pool: I/O  Free: 3872  Cause: Not enough free memory
 -Process= "Licensing Auto Update Process", ipl= 0, pid= 10,  -Traceback= 0x81EC
3F4Cz 0x81EE4870z 0x82C79250z 0x82ADF120z 0x82B17AB8z 0x82AD8FE0z 0x82ACF9FCz 0x
82AD6B84z 0x82AD75A8z 0x82B0F65Cz 0x82AD8850z 0x8004D704z 0x8045F1ECz 0x80462214
z
*Sep 20 19:57:00.383: %LICENSE-2-VLS_ERROR: 'VLSnotifyBirthAndExpiryEvents' fail
ed with an error - rc = 25 - 'Error[25]: Could not understand the message receiv
ed from license server on host "no-net".
Client-server version mismatch?
',  -Traceback= 0x8004D7B8z 0x8045F1ECz 0x80462214z
*Sep 20 19:57:01.031: %SYS-2-CHUNKEXPANDFAIL: Could not expand chunk pool for ip
nat node. No memory available -Process= "Chunk Manager", ipl= 4, pid= 1,  -Trace
back= 0x81EC2FF4z 0x8045F1ECz 0x80462214z
*Sep 20 19:57:40.235: %SYS-2-MALLOCFAIL: Memory allocation of 65536 bytes failed
 from 0x81EFE584, alignment 16
Pool: Processor  Free: 118656  Cause: Memory fragmentation
Alternate Pool: I/O  Free: 3872  Cause: Not enough free memory
 -Process= "IP ARP Adjacency", ipl= 4, pid= 96,  -Traceback= 0x81EC3F4Cz 0x81EE6
03Cz 0x81EFE588z 0x80EE9264z 0x80EEB33Cz 0x80EEB94Cz 0x80EE4374z 0x80EAAA9Cz 0x8
0E70480z 0x80EAE714z 0x80E65F54z 0x80E660ECz 0x80EF3180z 0x80EF4C04z 0x80EF4F58z
 0x80F43D58z
*Sep 20 19:57:40.239: %COMMON_FIB-3-NOMEM: Memory allocation failure for validat
ing prefix in IPv4 CEF [0x80E54E54] (fatal) (0 subsequent failures).
*Sep 20 19:57:40.239: %COMMON_FIB-4-DISABLING: IPv4 CEF is being disabled due to
 a fatal error.

Open in new window

0
 
LVL 3

Assisted Solution

by:kf4zmt
kf4zmt earned 500 total points
ID: 33721253
How much RAM do you have in it now?  Perhaps adding more would help, but I can't say for sure.  Do you have a support contract with Cisco?  If so, you might want to consider opening a TAC case.



Error Message    

%SYS-6-USING_RESERVE_MEMORY : Reserve memory is being used

Explanation    The system is low in memory. Memory is reserved for the console process, and that memory is being used by the console.

Recommended Action    Reduce other system activity to ease memory demands. If conditions warrant, upgrade to a larger memory configuration. If this message recurs, copy the error message exactly as it appears on the console or in the system log. Issue the show tech-support command to gather data that may help identify the nature of the error. If you cannot determine the nature of the error from the error message text or from the show tech-support command output, contact your Cisco technical support representative and provide the representative with the information you have gathered.
0
 

Author Comment

by:fratapia
ID: 33721319
I don't have a support contract with cisco.

What could I remove to make it run smoother?
It has 256M
Do you think maybe I need a different router for my needs? (15 computers and 5 network printers, 2 vpns and 4 computers at each site, traffic that travels on vpn is not heavy)
I never noticed the processor use being very high.
I think I'll give the memory upgrade a try. What is the best place to get it?
0
 
LVL 3

Expert Comment

by:kf4zmt
ID: 33725461
I noticed that there is a lot of crypto in this config.  That requires a lot of overhead.  Is it possible this router is simply being over utilized?
0
 

Author Comment

by:fratapia
ID: 33727559
I think that too, maybe it can't handle it as it is configured.  

What I can conclude from link to the cisco site that you gave me, is that I should add memory.

Do you think a memory upgrade will help with the fragmentation error?  I never noticed processor load to go past 12% but memory load hit 79% before I got that last error yesterday afternoon and then it dumped everything and went down to 40%.
0
 
LVL 3

Expert Comment

by:kf4zmt
ID: 33727791
256 MB is a lot of RAM for a router, especially a small router like this one.  According to this Cisco doc, you are already maxed out on RAM in the 871.

http://www.cisco.com/en/US/docs/routers/access/800/hardware/notes/800upgrd.html#wp41897

0
 

Author Comment

by:fratapia
ID: 33727994
I think I had mentioned it.  I had switched the router to a Cisco 881W on Friday.  The 881w is the one that received the error that I posted yesterday and then changed that setting on the memory that you had mentioned. Sorry maybe i should have closed that question and opened a new one.

881w has a max of 768.  

right now for example processor load is 6% and memory is 76%
0
 
LVL 3

Expert Comment

by:kf4zmt
ID: 33728134
Oops.  I remember that now.  

Well, it certainly won't hurt anything to add some RAM.  The only downside is that if it doesn't help, you will be out the cost of the memory.  
0
 

Author Comment

by:fratapia
ID: 33728213
Where do you recommend I purchase it?
0
 
LVL 3

Expert Comment

by:kf4zmt
ID: 33728344
I've never needed to purchase much RAM for routers over the years.  The last time I did was for my own personal use and I got it from a seller on eBay.  
0
 

Author Closing Comment

by:fratapia
ID: 33744501
After removing 'memory size iomem' from the configuration the router did not suffer the memory error.  Thant was on Monday, but it was noticeable that it was a bit slow, I added an extra 512Mb and it is running like a champ
 
Thanks very much for your help!
0

Featured Post

Find Ransomware Secrets With All-Source Analysis

Ransomware has become a major concern for organizations; its prevalence has grown due to past successes achieved by threat actors. While each ransomware variant is different, we’ve seen some common tactics and trends used among the authors of the malware.

Join & Write a Comment

In a WLAN, anything you broadcast over the air can be intercepted.  By default a wireless network is wide open to all until security is configured. Even when security is configured information can still be intercepted! It is very important that you …
While it is possible to put two routes in place with the secondary having a higher metric, this may not always work. In the event of a failure that does not bring down the physical interface on the router the primary route is not removed. There is a…
After creating this article (http://www.experts-exchange.com/articles/23699/Setup-Mikrotik-routers-with-OSPF.html), I decided to make a video (no audio) to show you how to configure the routers and run some trace routes and pings between the 7 sites…
After creating this article (http://www.experts-exchange.com/articles/23699/Setup-Mikrotik-routers-with-OSPF.html), I decided to make a video (no audio) to show you how to configure the routers and run some trace routes and pings between the 7 sites…

758 members asked questions and received personalized solutions in the past 7 days.

Join the community of 500,000 technology professionals and ask your questions.

Join & Ask a Question

Need Help in Real-Time?

Connect with top rated Experts

13 Experts available now in Live!

Get 1:1 Help Now