Cisco Network Monitoring Common Mistakes

I am going to touch on common mistakes in network monitoring in this post . You know for sure that you need a network-monitoring tool for managing your network. There are wide varieties of tools available that range from simple to complex and free to enterprise ones.
If you get one monitoring tool and install it, can you say that everything is under control? Are you going to be aware of what happened in your network? I will try to warn you about common mistakes in network monitoring. Actually, these mistakes are common for any kind of network however my experience on Cisco environment.

1. Monitoring without documentation
If you are monitoring your network while you don’t have the proper network documentation, then it will not be clear whether monitoring is complete or not. How can you be sure about reliability of your monitoring system without knowing exact number of devices, their models and their interconnections?

2. Only network specialists should watch over network.
Network specialists must setup network monitoring systems, but watching over them and taking first action should not be their task. If you have network monitoring screens, then such screens should be watched over by –
• A monitoring team – if the network is big enough (e.g. a NOC)
• Help desk – if you have
• End user support team
Any alert (alerts, events, mails, SMSs) should be directed to help desk or end user support team. The receiver must be able to handle it immediately. Alertness is the key here and therefore this task should not be assigned to staff who is involved in projects and moving often. Help desk staff should be intimated first and then information should move upwards based on the hierarchy, finally reaching the network admin to sort out the issue.

3. Unhandled alerts
All alerts should be checked and cleared. If there is expected maintenance on some devices, then they have to be excluded from monitoring system (This is a must have for a network monitoring tool). If some alerts stay on the monitoring system for a long time, then it will cause alert blindness on the team. False alerts may also drop your confidence in the monitoring system.

4. Correct probe points & traffic behavior
You have to understand your routing infrastructure very well, especially for flow monitoring. Sometimes, you can find undesirable traffic so easily, but it does not happen always. In case of a huge download, you just have to look at the right point in the backbone. In case of an antivirus update, traffic is one to many, you have to summarize collected data by source or target upon direction of traffic and in the case of many to many traffic like virus infections, you have to know or guess characteristics of undesired traffic (like tcp port). If you ignore these details, you can look at your netflow monitor and can swear that all seen traffic is necessary.

5. No history
If you have your monitoring system ready, but you monitor just some nodes and think that you can monitor any necessary point if something untoward incident happens (I mean SNMP monitoring),then you are playing with the fire! When something happens, to analyze it you will have to compare this condition with the normal conditions but you will be too late for that. It won’t be possible to acquire this information anymore. Therefore, you must monitor all ports and the interfaces that have to be monitored from the first day. Your monitoring technique is correct only when it is complete.

6. We have a huge tool – problem is over
This is about decision phase of network monitoring. You should define your needs well and choose fitting tool for your network. No more, no less. This decision is not just about cost. The concept will be clear with an example and a good example is Cisco Works. It is huge, capable and a brand that is trusted all over the world. However, if you don’t have a dedicated staff for this, then it is really hard to install and use it. I have come across many people who purchased Cisco in anticipation that it will be very beneficial to them, but did not make use of this powerful tool completely. It is like buying a truck and trying to park it in your car garage, which is a foolish decision!

7. Network monitoring is not a mission critical process
How much loss do you incur if your network monitoring system stops working? Is it going to stop production, sales or logistics? The answer is no. So, network monitoring system is not a mission critical system. This could be true. Network itself is mission critical. Everything stops when it stops. Network problems should be fixed immediately. You have to find the problem (here you need monitoring) in minutes. Nevertheless, your monitoring system can be down because it is not a mission critical system. If this is the case, you should connect each device separately and look for errors. It is similar to a situation in which you are driving on the highway with broken gauges (fuel, temperature, speed). Good luck!

These are the seven common mistakes in Cisco network monitoring. You are in charge of keep them away from your network.

Network Monitoring Tools

What are your daily duties as the network administrator? You have to keep your network up and running. You have to answer calls like “X location is down” or “Y location is so slow”. You should monitor your network as described below to fulfill your tasks.

  • You should monitor your network and react to device and line failures.
  • You should analyze line utilizations, errors on the line and be sure about network performance.
  • You should be aware of who talk with whom? How much bandwidth needed for every single application?
  • And sometimes you need to see exact data flow over the network.

If you have all these information people think twice before blame the network for every problem. How can you achieve this?
We need a layered approach to understand network monitoring. I am not talking about network layers but network monitoring layers? We have to involve closely to monitoring layers before decide about network monitoring software needs. A simple summary could be like below.

  • Preconditions of network monitoring.
  • Up/Down monitoring
  • Performance Monitoring / SNMP monitoring
  • Who talk with whom / Netflow monitoring
  • Data capture / Data sniffing

Preconditions of Network Monitoring
Network documentation is essential to monitor a network. Try set up network monitoring tools before documentation is just wasting time. You will see everything green on the screen but maybe while some of the redundant lines were down. And you will sit there, like a dumb without any alerts.
Documentation comes first.

Suggested tools: Powerpoint/Visio, NetViz

Up/Down monitoring
You have a map and some red and green lights on it. Green means up. Red means down. It is simple but powerful. You can immediately become aware of something goes down.
This is based on ping. Almost every IP devices support echo/echo reply. So you can monitor all IP devices in your network by using ping. To go one step further, you can monitor one application on a device instead of whole device. All network applications use TCP/UDP ports. You can monitor the applications by trying to telnet its TCP/UDP ports and if the port is open it means application is running.

Suggested tools: WhatsupGold, nmap

Performance monitoring / SNMP monitoring
The lines are up, the devices are up but life is not perfect. People complain about performance of data lines. Are they saturated? Do we have package losses on the lines? Are routers running out of memory? We need SNMP to monitor heart beat of the network.

Suggested tools: MRTG, Solarwinds Orion, PRTG

Who talk with whom / Netflow monitoring
You realized that the line is full. Someone / some applications make a high load of traffic. Who are they? Is it necessary traffic? In Cisco devices by using “ip accounting” command we can have an idea for current traffic sources and destinations. Nevertheless to analyze and to optimize the traffic we need flow monitoring. We need to know source and destination IP addresses and TCP/UDP ports and number of packages/bytes.
Everyone blame the network for slowness until you publish the network usage report which shows only 15% of the traffic is ERP traffic and rest is Internet access.
You should know that flow monitoring tools requires more server resources for the huge data they collect.

Suggested tools: Fluke Netflow monitor, Paasler

Data capture / RMON – Sniffer tools
Sometimes you need to see the exact data flow on the line instead of information about it. Just have a look to this sample scenario. After you find out the web service cause inappropriately high network traffic, the owner of the application just can say “No, we are not pushing this much of data to network. We just respond Yes or No in this web service and it is just 100 bytes” .Here you need to sniff the data flow on the line. Maybe you will find that web service respond yes or no (100 bytes) and the definition of web service (6 kilobytes).

Suggested tools: Wireshark,Gigamon

Standalone or stackable Cisco switches do not support NetFlow

NetFlow is a must have technology suitable for mid size to enterprise companies. Nowadays, it has become an IEEE standard as IPFIX (Internet Protocol Flow Information eXport). We will be able to find NetFlow technology support on any brand in the market soon. However, which devices of Cisco itself supports NetFlow technology?

All routers including the oldest (e.g. Cisco 2500 series) and smallest (e.g. Cisco 800 series) support NetFlow. Some functions does not exist in older IOS versions.
Catalyst 6500 series switches support NetFlow. Catalyst 4500 series switches support NetFlow with Supervisor IV/V + WS-F4531 Catalyst 4500 NetFlow Services Card.

Standalone or stackable switches do not support NetFlow. This means Catalyst 4948, Catalyst 3750 or Catalyst 3560 series switches do not support NetFlow. You can see the necessary commands on config mode, but they are not effective. It is not about IOS version or feature set. You need a modular switch for NetFlow.

Unfortunately, the answer of “What Cisco switches support netflow?” is only the modular switches.