Solved

Cisco %TRACKING-5-STATE Not Appearing In Logs

Posted on 2014-02-20
26
2,129 Views
Last Modified: 2014-03-18
Hello Experts,

Can someone please let me know why I'm seeing the %TRACKING-5-STATE in my logs.

I have the following configured:

logging trap debugging
logging facility syslog
logging source-interface Vlan1
logging 10.44.108.79
logging 10.44.97.254
logging 10.45.32.66


ip sla 153
 icmp-echo 195.59.159.29 source-ip 194.75.202.234
 request-data-size 24
 tos 30
 timeout 60000
 threshold 200
 owner NetFlow
ip sla reaction-configuration 153 react rtt threshold-value 200 150 threshold-type immediate action-type trapOnly
ip sla reaction-configuration 153 react connectionLoss threshold-type immediate action-type trapOnly
ip sla reaction-configuration 153 react timeout threshold-type immediate action-type trapOnly
ip sla reaction-configuration 153 react verifyError threshold-type immediate action-type trapOnly
ip sla schedule 153 life forever start-time pending ageout 60



Track 153
  Response Time Reporter 153 state
  State is Down
    89 changes, last change 00:04:11
  Latest operation return code: Over threshold
  Latest RTT (millisecs) 8256

Open in new window



So, when the state changes I should get following syslog message %TRACKING-5-STATE:

However, its not showing ... any ideas?

Cheers

Carlton
0
Comment
Question by:cpatte7372
  • 13
  • 6
  • 3
  • +1
26 Comments
 
LVL 26

Expert Comment

by:Soulja
ID: 39873654
Do you have
ip sla logging traps configured?
0
 

Author Comment

by:cpatte7372
ID: 39873731
Soulja,

I didn't have it configured. However, I have now configured it with ip sla logging traps, but still nothing appears in the log.. any other ideas?
0
 
LVL 26

Expert Comment

by:Soulja
ID: 39874137
Are you able to receive any other logs from this device?
0
 
LVL 28

Expert Comment

by:mikebernhardt
ID: 39874382
Is it showing up locally? Add "logging buffered 65536 debug" and check (that allows for a LOT of buffer but you won't miss anything).

That will help determine whether the problem is with the logging configuration or not.
0
 

Author Comment

by:cpatte7372
ID: 39876173
Mike,

Thanks for responding.

I added "logging buffered 65536 debug", but the %TRACKING-5-STATE: syslog message still isn't appearing.

Soulja

I am getting other log messages.

Any other ideas?

Cheers
0
 
LVL 26

Accepted Solution

by:
Soulja earned 500 total points
ID: 39877070
See if this link helps. I see several commands in the posts that might be needed.

https://supportforums.cisco.com/thread/2018061
0
 

Author Comment

by:cpatte7372
ID: 39879796
Soulja,

I've tried everything from the link you provided and I still can't see the syslog message appear .. I don't understand..
0
 

Author Comment

by:cpatte7372
ID: 39880431
Experts,

Do you have any other ideas please?
0
 
LVL 32

Expert Comment

by:harbor235
ID: 39882942
What platform and IOS are you using?

Can you display the output of the following command

show logging


Can the device in question route to the syslog server?


You never see the state change in the local log?  How are you inducing a state change?

Is this IP bound to an interface on the device in question? 194.75.202.234?

Can you post the entire sanitized config


harbor235 ;}
0
 

Author Comment

by:cpatte7372
ID: 39883035
Hi Harbor,

The full version is as follows:

MX_DMVPN_HUB6#show version
Cisco IOS Software, C181X Software (C181X-ADVIPSERVICESK9-M), Version 12.4(6)T7, RELEASE SOFTWARE (fc5)
Technical Support: http://www.cisco.com/techsupport
Copyright (c) 1986-2007 by Cisco Systems, Inc.
Compiled Thu 29-Mar-07 02:54 by khuie

ROM: System Bootstrap, Version 12.3(8r)YH6, RELEASE SOFTWARE (fc1)

MX_DMVPN_HUB6 uptime is 29 weeks, 4 days, 17 hours, 50 minutes
System returned to ROM by error - a SIGTRAP exception, PC 0x800C694C at 00:00:51 GMT Wed Oct 29 2008
System restarted at 23:53:09 BST Wed Jul 31 2013
System image file is "flash:c181x-advipservicesk9-mz.124-6.T7.bin"


This product contains cryptographic features and is subject to United
States and local country laws governing import, export, transfer and
use. Delivery of Cisco cryptographic products does not imply
third-party authority to import, export, distribute or use encryption.
Importers, exporters, distributors and users are responsible for
compliance with U.S. and local country laws. By using this product you
agree to comply with applicable laws and regulations. If you are unable
to comply with U.S. and local laws, return this product immediately.

A summary of U.S. laws governing Cisco cryptographic products may be found at:
http://www.cisco.com/wwl/export/crypto/tool/stqrg.html

If you require further assistance please contact us by sending email to
export@cisco.com.

Cisco 1812 (MPC8500) processor (revision 0x400) with 118784K/12288K bytes of memory.
Processor board ID FCZ10462090, with hardware revision 0000

10 FastEthernet interfaces
1 ISDN Basic Rate interface
31360K bytes of ATA CompactFlash (Read/Write)

Configuration register is 0x2102

MX_DMVPN_HUB6#



The logging is as following:


MX_DMVPN_HUB6#show logging
Syslog logging: enabled (1 messages dropped, 50 messages rate-limited,
                0 flushes, 0 overruns, xml disabled, filtering disabled)
    Console logging: level debugging, 216008 messages logged, xml disabled,
                     filtering disabled
    Monitor logging: level debugging, 1374 messages logged, xml disabled,
                     filtering disabled
    Buffer logging: level debugging, 86 messages logged, xml disabled,
                    filtering disabled
    Logging Exception size (4096 bytes)
    Count and timestamp logging messages: disabled

No active filter modules.

ESM: 0 messages dropped

    Trap logging: level debugging, 215914 message lines logged
        Logging to 10.44.108.79(global) (udp port 514, audit disabled, link up), 215914 message lines logged, xml disabled,
               filtering disabled
        Logging to 10.44.97.254(global) (udp port 514, audit disabled, link up), 215914 message lines logged, xml disabled,
               filtering disabled
        Logging to 10.45.32.66(global) (udp port 514, audit disabled, link up), 1787 message lines logged, xml disabled,
               filtering disabled
         
Log Buffer (32656 bytes):


The device can definitely route to the server...

I'm testing the log with a an Interface loopback 0 interface 9.9.9.9. So I've created an ip sla 155, and a track 155. I test the syslog by shutting down the loopback 0 interface. The corresponding track 155 will go down, but no corresponding syslog message will appear.

Please find attached the configs

I hope that helps you to help me.


Cheers

Carlton
16-55-42--MX-DMVPN-HUB6-10.44.11.txt
0
 
LVL 32

Expert Comment

by:harbor235
ID: 39883103
So what are we looking for a syslog message or the trap? You stated you tried commands from cisco support Forums but i do not see this in your config -  
ip sla monitor logging traps?

If you looking for a syslog message your config will not accomplish this, it will send a trap. Does your syslog box listen on udp 162 to receive the traps? At the end of each of your "ip sla" commands you specify traponly.

So you will not see a syslog message, you will generate a trap

Soulja sent you a really good link describing what I said above, read the entire thread there is some good stuff in there. The y key however is the difference between a trap and a syslog message.

https://supportforums.cisco.com/thread/2018061


harbor235 ;}


harbor235 ;}
0
 

Author Comment

by:cpatte7372
ID: 39883712
Harbor,

Thanks for responding. As my question states, I'm trying to get a syslog message appear, not a trap.

I've gone through every singe command in the link provided by the Expert Soulja, however I'm still not seeing %TRACKING-5-STATE appear in syslog messages.

Can you or any Expert let me know how I can get the log message '%TRACKING-5-STATE' appear in the syslog.

I'm really surprised an Expert doesn't know the answer to this question.

Harbor,

Is there anything missing from the attached configs that would make the %TRACKING-5-STATE message appear in the syslog?

Carlton
0
What Is Threat Intelligence?

Threat intelligence is often discussed, but rarely understood. Starting with a precise definition, along with clear business goals, is essential.

 

Author Comment

by:cpatte7372
ID: 39883725
Harbor,

You mentioned
If you looking for a syslog message your config will not accomplish this, it will send a trap.


Do you know what will generate a syslog message?

Regards
0
 
LVL 32

Expert Comment

by:harbor235
ID: 39883828
You can use EEM scripting to perform the ip sla function and then generate a syslog message when certain criteria are met.

EEM Script:
action 1.0 syslog msg "My_Message_Here"


You will need to
0
 

Author Comment

by:cpatte7372
ID: 39884294
Harbor

I appreciate your continued efforts, but the answer doesn't really help explain why I'm unable to ge the syslog message in the first place.

I'm very familiar with EEM, and it would take much more than what your suggesting to make EEM work with IP SLA.

The fact is that I'm not getting the syslog message on this router when I'm getting all othe syslog messages. Furthermore, I'm getting the syslog message with our other routers without EEM

You mentioned that with my current configs I won't get the syslog message, this would suggest that you know what configs would generate the syslog. if youre suggesting that the only reason I'm not getting the syslog message is because of EEM, then I don't think your right.

I wonder if some other Experts can weigh in here?

Carlton
0
 
LVL 32

Expert Comment

by:harbor235
ID: 39884589
cpatte7372,

First do you see that your config specifies traponly at the end of each config statement?

I mentioned EEM scripting because you can generate custom syslog messages with the
following cmd "action 1.0 syslog msg "My_Message_Here" so you could generate a syslog message when a condition is met, not to hard but it would take time to develop like in this link, similar but not exact

http://www.cisco.com/web/about/security/intelligence/embedded-event-mgr.html

http://books.google.com/books?id=gAtUR93SO5gC&pg=PA89&lpg=PA89&dq=cisco+ip+sla+generate+local+syslog+message+-site:cisco.com&source=bl&ots=LgC23rdjC9&sig=NjiU5AQo3v08Mf8woPHamA9Mgms&hl=en&sa=X&ei=__8LU8zULoXq0AHU3IHwBw&ved=0CE0Q6AEwBA#v=onepage&q=cisco%20ip%20sla%20generate%20local%20syslog%20message%20-site%3Acisco.com&f=false


So, the ip sla commands you are using have an action-type at the end of the command, you have chosen to use "traponly" other possible types are as follows:

 Action type options for the ip sla monitor reaction-configuration command are as follows:

none—No action is taken.

trapOnly—Send an SNMP logging trap when the specified violation type occurs for the monitored element. IP SLAs logging traps are enabled using the ip sla monitor logging traps command. For SNMP logging traps to be sent, SNMP logging must be enabled using the appropriate SNMP commands, including the snmp-server enable traps syslog command.

triggerOnly—Have one or more target operation's operational state make the transition from "pending" to "active" when the violation conditions are met. The target operations to be triggered are specified using the ip sla monitor reaction-trigger command. A target operation will continue until its life expires, as specified by the target operation's configured lifetime value). A triggered target operation must finish its life before it can be triggered again.

trapAndTrigger—Trigger both an SNMP trap and start another IP SLAs operation when the violation conditions are met, as defined in the trapOnly and triggerOnly options above.

These are your choices, thats it.

here is a reference link for your version of code  for ip sla - http://www.cisco.com/c/en/us/td/docs/ios/12_4/ip_sla/configuration/guide/hsla_c/hsthresh.html#wp1082317

From cisco - link above

 IP SLAs Threshold Monitoring and Notifications

IP SLAs includes the capability for triggering SNMP notifications based on defined thresholds. This allows for proactive monitoring in an environment where IT departments can be alerted to potential network problems, rather than having to manually examine data.

IP SLAs supports threshold monitoring for performance parameters such as average jitter, unidirectional latency and bidirectional round trip time and connectivity. This proactive monitoring capability provides options for configuring reaction thresholds for important VoIP related parameters including unidirectional jitter, unidirectional packet loss, and unidirectional VoIP voice quality scoring (MOS scores).

IP SLAs can generate system logging (syslog) messages when the reaction threshold increases or decreases beyond the configured values for packet loss, average jitter, or MOS. These system logging messages can then be sent as SNMP notifications (traps) using the CISCO-SYSLOG-MIB.

For packet loss and jitter, notifications can be generated for violations in either direction (source to destination and destination to source) or for round trip values. Packet loss, jitter and MOS statistics are specific to IP SLAs Jitter operations. Notifications can also be triggered for other events, such as round-trip-time violations, for most IP SLAs monitoring operations.

So IP sla will not generate local syslog entries, however it will send notifications via snmp traps.

I hope this helps,

harbor235 ;}
0
 

Author Comment

by:cpatte7372
ID: 39885844
Hi Harbor,

Thanks for sticking with me on this:

I have removed the 'trapOnly' argument as follows for ip sla 155

ip sla 155
 icmp-echo 9.9.9.9 source-ip 172.17.143.1
 request-data-size 24
 tos 30  
 timeout 60000
 threshold 200
 owner NetFlow
ip sla schedule 155 life forever start-time now


AND THE TRACKER IS AS FOLLOWS:

MX_DMVPN_HUB6#show track 155
Track 155
  Response Time Reporter 155 reachability
  Reachability is Down
    12 changes, last change 00:04:35
  Delay up 10 secs, down 10 secs
  Latest operation return code: Timeout

However, I'm still unable to see the messages in syslog messages

This is a brain scratcher  .....
0
 
LVL 32

Expert Comment

by:harbor235
ID: 39885922
cpatte7372,

I am saying that ip sla does not generate local syslog messages. What is generated is a syslog message sent to the trap host, you see?

Do not look locally for the message, goto your trap host and look.


harbor235 ;}
0
 

Author Comment

by:cpatte7372
ID: 39885975
Hi Harbor,

Thanks again for not giving up on this.

you mentioned
So IP sla will not generate local syslog entries, however it will send notifications via snmp traps.

I don't want IP SLA to generate a syslog message. The syslog message that I want is from the tracker in the form of '%TRACKING-5-STATE'.

The tracker 155 should be triggered by ip sla, and it is the tracker that I want to be seen in the syslog .

Does that make sense?

Cheers
0
 
LVL 32

Expert Comment

by:harbor235
ID: 39886060
cpatte7372,

one depends on the other, the ip sla part performs the notification function so we are back to the intended feature.

Here is a link using custom syslog generation that you could customize for your environment:

http://nukethesitefromorbit.blogspot.com/2012/01/monitoring-connection-via-ip-sla.html


harbor235 ;}
0
 
LVL 28

Expert Comment

by:mikebernhardt
ID: 39886282
It's also possible that this is a bug and the trap is simply not being sent as it should. What version of IOS are you using exactly?
0
 

Author Comment

by:cpatte7372
ID: 39887396
Guys,

Just we're clear I can see the traps:

02-25-2014      23:12:12      Local7.Debug      MX_DMVPN_HUB6      community=netmgr, enterprise=1.3.6.1.4.1.9.9.42.2, enterprise_mib_name=rttMonNotificationsPrefix, uptime=1805834081, agent_ip=MX_DMVPN_HUB6, generic_num=6, specificTrap_num=5, specificTrap_name=rttMonNotificationsPrefix.5, version=Ver1, rttMonCtrlAdminTag.155=, rttMonHistoryCollectionAddress.155=            , rttMonReactVar.155=7, rttMonReactOccurred.155=1, rttMonReactValue.155=1, rttMonReactThresholdRising.155=0, rttMonReactThresholdFalling.155=0, rttMonEchoAdminLSPSelector.155=

The above is the trap sent to the NMS.

However, I can't see the syslog message that is generated from the track - which is shown with every other router......

The ios is:

Cisco IOS Software, C181X Software (C181X-ADVIPSERVICESK9-M), Version 12.4(6)T7, RELEASE SOFTWARE (fc5)
Technical Support: http://www.cisco.com/techsupport
Copyright (c) 1986-2007 by Cisco Systems, Inc.
Compiled Thu 29-Mar-07 02:54 by khuie

ROM: System Bootstrap, Version 12.3(8r)YH6, RELEASE SOFTWARE (fc1)

MX_DMVPN_HUB6 uptime is 29 weeks, 6 days, 14 minutes
System returned to ROM by error - a SIGTRAP exception, PC 0x800C694C at 00:00:51 GMT Wed Oct 29 2008
System restarted at 23:53:09 BST Wed Jul 31 2013
System image file is "flash:c181x-advipservicesk9-mz.124-6.T7.bin"



I don't understand
0
 

Author Comment

by:cpatte7372
ID: 39900225
Gents,

I'm going to change the IOS and see if that helps.

In the meantime, thanks for your support with this.

Cheers
0
 

Author Closing Comment

by:cpatte7372
ID: 39937970
Cheers
0

Featured Post

Highfive + Dolby Voice = No More Audio Complaints!

Poor audio quality is one of the top reasons people don’t use video conferencing. Get the crispest, clearest audio powered by Dolby Voice in every meeting. Highfive and Dolby Voice deliver the best video conferencing and audio experience for every meeting and every room.

Join & Write a Comment

Suggested Solutions

David Varnum recently wrote up his impressions of PRTG, based on a presentation by my colleague Christian at Tech Field Day at VMworld in Barcelona. Thanks David, for your detailed and honest evaluation!
Use of TCL script on Cisco devices:  - create file and merge it with running configuration to apply configuration changes
After creating this article (http://www.experts-exchange.com/articles/23699/Setup-Mikrotik-routers-with-OSPF.html), I decided to make a video (no audio) to show you how to configure the routers and run some trace routes and pings between the 7 sites…
Get a first impression of how PRTG looks and learn how it works.   This video is a short introduction to PRTG, as an initial overview or as a quick start for new PRTG users.

746 members asked questions and received personalized solutions in the past 7 days.

Join the community of 500,000 technology professionals and ask your questions.

Join & Ask a Question

Need Help in Real-Time?

Connect with top rated Experts

13 Experts available now in Live!

Get 1:1 Help Now