1. Monitoring without documentation
If you are monitoring your network and don’t have the complete network documentation, then it will not be clear whether monitoring is beneficial or not. How can you be sure about reliability of your monitoring system without knowing exact number of devices, their models and their interconnections?

2. Only network specialists should watch over network.
Network specialists must setup network monitoring systems, but watching over them and taking first action should not be their task. If you have network monitoring screens, then such screens should be watched over by -
• A monitoring team – if the network is big enough (e.g. a NOC)
• Help desk – if you have
• End user support team
Any alert (alerts, events, mails, SMSs) should be directed to help desk or end user support team. The receiver must be able to handle it immediately. Alertness is the key here and therefore this task should not be assigned to staff who is involved in projects and moving often. Help desk staff should be intimated first and then information should move upwards based on the hierarchy, finally reaching the network admin to sort out the issue.

3. Unhandled alerts
All alerts should be checked and cleared. If there is expected maintenance on some devices, then they have to be excluded from monitoring system (This is a must have for a network monitoring tool). If some alerts stay on the monitoring system for a long time, then it will cause alert blindness on the team. False alerts may also drop your confidence in the monitoring system.

4. Correct probe points & traffic behavior
You have to understand your routing infrastructure very well, especially for flow monitoring. Sometimes, you can find undesirable traffic so easily, but it does not happen always. In case of a huge download, you just have to look at the right point in the backbone. In case of an antivirus update, traffic is one to many, you have to summarize collected data by source or target upon direction of traffic and in the case of many to many traffic like virus infections, you have to know or guess characteristics of undesired traffic (like tcp port). If you ignore these details, you can look at your netflow monitor and can swear that all seen traffic is necessary.

5. No history
If you have your monitoring system ready, but you monitor just some nodes and think that you can monitor any necessary point if something untoward incident happens (I mean SNMP monitoring),then you are playing with the fire! When something happens, to analyze it you will have to compare this condition with the normal conditions but you will be too late for that. It won’t be possible to acquire this information anymore. Therefore, you must monitor all ports and the interfaces that have to be monitored from the first day. Your monitoring technique is correct only when it is complete.

6. We have a huge tool – problem is over
This is about decision phase of network monitoring. You should define your needs well and choose fitting tool for your network. No more, no less. This decision is not just about cost. The concept will be clear with an example and a good example is Cisco Works. It is huge, capable and a brand that is trusted all over the world. However, if you don’t have a dedicated staff for this, then it is really hard to install and use it. I have come across many people who purchased Cisco in anticipation that it will be very beneficial to them, but did not make use of this powerful tool completely. It is like buying a truck and trying to park it in your car garage, which is a foolish decision!

7. Network monitoring is not a mission critical process
How much loss do you incur if your network monitoring system stops working? Is it going to stop production, sales or logistics? The answer is no. So, network monitoring system is not a mission critical system. This could be true. Network itself is mission critical. Everything stops when it stops. Network problems should be fixed immediately. You have to find the problem (here you need monitoring) in minutes. Nevertheless, your monitoring system can be down because it is not a mission critical system. If this is the case, you should connect each device separately and look for errors. It is similar to a situation in which you are driving on the highway with broken gauges (fuel, temperature, speed). Good luck!

These are the seven common mistakes in Cisco network monitoring. You are in charge of keep them away from your network.

Recently, I made an upgrade on some of 6509 switches. I bought totally new 6509-E system including chassis, fan, power, and some new line cards. I also insert my fiber line cards that I used in old 6509 with supervisor 2.
Everything was fine until I checked modules status with “show module” command. Switch was up, new line cards were functioning, but my old WS-X6516-GBIC’s were in PwrDown state. When I checked the switch logs I found

00:02:31: %C6KPWR-SP-4-UNSUPPORTED: unsupported module in slot 2, power not allowed: The image for the card is not bundled in image.

What does it mean? I had a recent IOS and it does not support this card. I was not surprised because it happened to me before. I just checked the software advisor tool from Cisco and found another version.I tried it, but it did not work out.

A wise friend of mine told me that I should give a try to a safe harbor image. I was not aware of safe harbor program until that time. Safe harbor images are tested images. They are stable, interoperable and solid. It was my last chance before I should have to escalate this problem.

I found out that there are some problems between Supervisor Engine 720 and WS-X6516-GBIC, but the problem were only valid for 5.0 to 5.3 Hw versions and I had 5.7. It should not be my problem but text mentioned about DFC Sub-modules. I focused on DFC daughter card then. Everything become clear when I found “Catalyst 6500 Series DFC, DFC3A, DFC3B, and DFC3BXL Installation Note” document. I saw a note mentioned below

Note You cannot have a DFC in a system with a Supervisor Engine 720

I removed one DFC (WS-F6K-DFC) from one of the WS-X6516-GBIC and finally it started to work. I also changed the IOS images to safe harbor ones. I spent days to solve this problem and this turned out to be a simple issue for which I got the answer instantaneously from the web!

• You should monitor your network and take actions with respect to situations like device and line failures.
• You should analyze line utilizations, errors on the line and be sure about network performance.
• You should be aware of who talks with whom? How much bandwidth is needed for every single application?
• And sometimes, you need to see exact data flow over the network.

If you have all these information ready, then people will think twice before they point finger at you. How can you achieve this?
We need a layered approach to understand network monitoring. I am not talking about network layers, but network monitoring layers. We have to involve deeply to monitoring layers before decide about network monitoring software needs. A simple summary could be like below.

• Preconditions of network monitoring.
• Up/Down monitoring
• Performance Monitoring / SNMP monitoring
• Who talks with whom? / Netflow monitoring
• Data capture / Data sniffing

Preconditions of Network Monitoring
Network documentation is essential to monitor a network. Trying to set up network monitoring tools before going through the documentation is complete waste of time. You will see everything green on the screen, but this maybe due to one of the redundant lines that are down. You will sit staring without knowing what is happening. Always remember, documentation comes first and everything follows.
Suggested monitoring tools: Powerpoint/Visio, NetViz

Up/Down monitoring
You have a map in which you can see some red and green lights glowing. Green means up and red means down. It is simple yet powerful. You will immediately come to know that there is some problem if the red light glows.
This is based on ping. Almost every IP devices support echo/echo reply. So, you can monitor all IP devices in your network by using ping. You go one step further by monitoring one application at a time present on a device instead of whole device. All network applications utilize TCP/UDP ports. You can monitor the applications by trying to access with telnet to its TCP/UDP ports. The port being open suggests that the application is running

Suggested monitoring tools: WhatsupGold, nmap

Performance monitoring / SNMP monitoring
The lines are up, the devices are up, but life is not perfect. People complain about performance of data lines. Are they saturated? Do we have package losses on the lines? Are routers running out of memory? We need SNMP to monitor heart beat of the network.

Suggested monitoring tools: MRTG, Solarwinds Orion, PRTG

Who talks with whom? / Netflow monitoring
You realized that the line is full. Someone / some applications make increase traffic load enormously. Who are they? Is it necessary traffic? In Cisco devices, by using “ip accounting” command we can get an idea of current traffic sources and destinations. Nevertheless, to analyze and to optimize the traffic we need flow monitoring. We need to know source and destination IP addresses and TCP/UDP ports and number of packages/bytes.
Everyone blames the network speed until you publish the network usage report that clearly shows only 15% of the traffic is ERP traffic and rest comes Internet access.
You should know that flow monitoring tools requires more server resources, since they collect enormous amount of data.

Suggested monitoring tools: Fluke Netflow monitor, Paasler

Data capture / RMON – Sniffer tools
Sometimes you need to observe the exact data flow on the line and not just information about it. Just have a look at this sample scenario. After you find out that the web service causes inappropriately high network traffic, the owner of the application just can say “No, we are not pushing this much of data to network. We just respond Yes or No in this web service and it is just 100 bytes”. Therefore, you should sniff the data flow on the line. Maybe, you will find that web service responds yes or no (100 bytes) and with the definition of web service (6 kilobytes).

Suggested monitoring tools: Wireshark

You can have a look at Network Monitoring Tools in Stanford University web site for a great list of network monitoring tools. You can find another tidy list at Network Traffic Monitoring in Alan Kennington’s topology.org.

You can use an access control list (ACL) for the filtering one way traffic, but what about the responding packages. You have to add an incoming ACL and it should include only sessions started from internal. Reflexive ACLs helps us in this point.

Requirements
To use reflexive ACLs
1. You MUST use named access lists
2. You MUST add “reflect samplename” to the end of permit line.
3. You MUST create a second named access list and add “evaluate samplename” line for responding traffic.

Sample Scenario
In our example, we have a proxy server (e.g. Microsoft ISA Server) with 122.22.22.1 IP address. This server needs access to internet via http (tcp 80) for web browsing and via dns (udp 53) for name resolution.

interface Serial0/0/0
description Internet connection
ip access-group INBOUND in
ip access-group OUTBOUND out
!
ip access-list extended OUTBOUND
permit tcp host 122.22.22.1 any reflect PROXYTCP
permit udp host 122.22.22.1 any eq domain reflect PROXYUDP
!
ip access-list extended INBOUND
evaluate PROXYTCP
evaluate PROXYUDP
!

We used reflect command to create reverse ACL and we added it to inbound ACL with evaluate command.

Last Words
This feature is really a powerful tool to increase network security, but you should not use it instead of a real firewall for Internet access. It should be used as another security layer.

Q1. What happens if I changed hostname or ip domain name after SSH settings has been done?
A1. Nothing. You need them to create rsa keys but, but afterwards, if you change them, only the key name changes and key data remain same.

ciscolab#sh crypto key mypubkey rsa
% Key pair was generated at: 13:08:15 UTC Aug 28 2007
Key name: ciscolab.mydomain.com
Storage Device: private-config
Usage: General Purpose Key
Key is not exportable.
Key Data:
30819F30 0D06092A 864886F7 0D010101 05000381 8D003081 89028181 00B67A9F
EED05E82 FFE41EB0 0CE9BC9E 40D1DD1D CF7AA44F CB5C1029 9502C379 469BF37D
099082BD 9618CC4E 8314866E 3B26F01B BE3AC27E 33EC7A2D 7FE5B503 3C24500B
733B391A D2DC4AAF C322C549 8A4638F1 9EAA0FF1 0ABCACD3 B1DF9753 02790FD7
E6A29602 39EFBAB4 2D4D7119 5C95D403 E1E9EB40 E01A1679 231C2F93 53020301 0001
.
.
ciscolab#conf t
Enter configuration commands, one per line. End with CNTL/Z.
ciscolab(config)#hostname sshrouter
sshrouter(config)#end
sshrouter#sh crypto key mypubkey rsa
% Key pair was generated at: 13:08:15 UTC Aug 28 2007
Key name: sshrouter.mydomain.com
Storage Device: private-config
Usage: General Purpose Key
Key is not exportable.
Key Data:
30819F30 0D06092A 864886F7 0D010101 05000381 8D003081 89028181 00B67A9F
EED05E82 FFE41EB0 0CE9BC9E 40D1DD1D CF7AA44F CB5C1029 9502C379 469BF37D
099082BD 9618CC4E 8314866E 3B26F01B BE3AC27E 33EC7A2D 7FE5B503 3C24500B
733B391A D2DC4AAF C322C549 8A4638F1 9EAA0FF1 0ABCACD3 B1DF9753 02790FD7
E6A29602 39EFBAB4 2D4D7119 5C95D403 E1E9EB40 E01A1679 231C2F93 53020301 0001
.
.
sshrouter#

Q2. Is there any other way to create rsa keys?
A1. Yes, There is. You can create rsa keys which are labeled by you. In this case, you don’t need a hostname(Always, you will have one) and an ip domain name.

ciscolab(config)#crypto key generate rsa general-keys label TEST
The name for the keys will be: TEST
Choose the size of the key modulus in the range of 360 to 2048 for your
General Purpose Keys. Choosing a key modulus greater than 512 may take
a few minutes.

How many bits in the modulus [512]: 1024
% Generating 1024 bit RSA keys, keys will be non-exportable…[OK]

ciscolab(config)#

Nov 24 20:08:59: %SSH-5-ENABLED: SSH 1.5 has been enabled

Q3. May I create more than one key?
A3. You can create several keys and chose one of them to use with SSH.You do not need to define which key to be used, but if you want to define, then you have to issue “ip ssh rsa keypair-name” command in the configuration mode.

ciscolab#sh crypto key mypubkey rsa
% Key pair was generated at: 20:08:59 UTC Nov 24 2007
Key name: TEST
Storage Device: not specified
Usage: General Purpose Key
Key is not exportable.
Key Data:
30819F30 0D06092A 864886F7 0D010101 05000381 8D003081 89028181 00BDFABF
948EF1FC 1CFC6C5C F5863980 D7D7B9E6 B256D84F 8F279E2E 63303403 A26E6160
A2928C87 4F0A846E F8A9FB0A 7D92108F ABD5734C AE7555BC 94CB13D9 41E8E04C
1514A499 68CC9925 A3DB2CFA 3176A65E 2DC504EE EF5C209E 4D348B20 9C324CBC
230451DD 96EC090C 99C5FB58 E06876D3 161E758E 486987B7 CD147AB0 0F020301 0001
% Key pair was generated at: 20:08:59 UTC Nov 24 2007
Key name: TEST.server
Temporary key
Usage: Encryption Key
Key is not exportable.
Key Data:
307C300D 06092A86 4886F70D 01720103 00036B00 30680261 00AEC9A3 4078450F
B1714135 F66FC617 3083F337 1309D493 654BC77D 4D08DE27 5A54FF44 C4CE0174
507385A9 99B93D70 4E980CE1 89465B14 00E2C26D A633F1FB C4D08A90 3A8EF761
EBB41B0D C3EB2190 E4FD1E4B E519A06E 4B6BAE46 4E1FA9D8 C1020301 0001
% Key pair was generated at: 20:11:56 UTC Nov 24 2007
Key name: CHECK
Storage Device: not specified
Usage: General Purpose Key
Key is not exportable.
Key Data:
305C300D 06092A86 4886F70D 01720103 00034B00 30480241 00CCB917 D58E9D45
BC5EFF15 E2945343 18E5338B 26E1ED9F 869C2B6F 77C27595 8AC0D7B7 9D503F31
192D08EF C5DE87B5 911779BD 464913CD BB93F883 6F23AE0A 91020301 0001
ciscolab#

ciscolab(config)#ip ssh rsa keypair-name CHECK

Q4. May I create more than one session from same computer like telnet?
A4. Yes.

Q5. Is it possible to use SSH1 and SSH2 at the same time?
A5. Yes. If you don’t fix the version with “ip ssh version” configuration command. You can use both protocol simultaneously as shown below.

ciscolab >sh ssh
Connection Version Encryption State Username
0 1.5 3DES Session started sshtest
Connection Version Mode Encryption Hmac State Username
1 2.0 IN aes256-cbc hmac-sha1 Session started sshtest
1 2.0 OUT aes256-cbc hmac-sha1 Session started sshtest
ciscolab>

As you can see here SSH1 uses 3DES and SSH2 uses AES.

Q6. Does SSH cause a slowdown on my device?
A6. No. I made some tests with both SSH1 and SSH2. Tests have been done on Cisco 7206 VXR, Cisco 3845 and Cisco 1841 with size of key modulus 2048. In the 1841 router, key generation took some time (30-40 seconds). There was a small delay (1-2 second) when I first connected the device, but rest of the interaction was same like the telnet. CPU utilization was at %1 and memory was OK.

Q7. I have copy my whole router / switch configuration but SSH does not work. Why?
A7. Did you create rsa certificate? Crypto key generate command is a configuration mode command, but it is not a part of the configuration. It will be used for creating your rsa certificate then it is gone. So, just copying configuration is not enough.

Please leave a comment if you have any other questions.

To show some different conditions, our connected Level3 router is also the BGP router, but Sprint BGP router has been located 3 hops away.

! We are starting with router command with our ASN like EIGRP/OSPF configuration
router bgp 6767
! We have one router in our case and we don’t need IGP / BGP synchronization it is default in newest IOSes
no synchronization
! To have some idea about our peering history we are recording events
bgp log-neighbor-changes
! We are announcing our network
network 193.93.93.0
! Our first neighbor is Sprint, we have to define neighbor ASN
neighbor 122.22.33.1 remote-as 1239
! Descriptions are always helpful
neighbor 122.22.33.1 description Sprint
! Our bgp neighbor in Sprint 3 hops away
neighbor 122.22.33.1 ebgp-multihop 3
! Sprint will use our Loopback IP (193.93.94.1) for us as the neighbor
neighbor 122.22.33.1 update-source Loopback 0
! If we have enough memory, we can prevent BGP session resets on inbound updates
neighbor 122.22.33.1 soft-reconfiguration inbound
! We won’t announce any network other than ourselves
neighbor 122.22.33.1 filter-list 1 out
! Now the Level 3
neighbor 111.11.11.1 remote-as 3356
neighbor 111.11.11.1 description LevelThree
neighbor 111.11.11.1 soft-reconfiguration inbound
neighbor 111.11.11.1 filter-list 1 out
no auto-summary
!
! Sprint BGP bridgehead is not directly connected to our router
! We must add necessary routing
ip route 122.22.33.1 255.255.255.255 122.22.22.1
!
! This filter means we are not announcing Sprint networks to Level3 or vice versa.
ip as-path access-list 1 permit ^$
!

In the end, if you are not on the middle of the Internet, BGP configuration is not a big issue.

In my example, I will show how I bundle two 2 Mbps frame relay line to act as one 4 Mbps line.

We have to do below configuration on both sides.

1)We will make an ordinary frame relay configuration on serial interfaces except “frame-relay interface-dlci 16 ppp Virtual-Template1” line. Here we are adding Virtual-Template1. frame-relay traffic-shaping command is a MUST.
2)Under “interface Virtual-Template1“, we describe that it is a part of multilink interface
3)Under “interface Multilink1” we will configure IP settings.

interface Serial0/0
description Physical Interface 1
bandwidth 2000
no ip address
encapsulation frame-relay
frame-relay fragmentation voice-adaptive deactivation 15
frame-relay traffic-shaping
frame-relay interface-dlci 16 ppp Virtual-Template1
frame-relay lmi-type ansi

interface Serial0/1
description Physical Interface 2
bandwidth 2000
no ip address
encapsulation frame-relay
frame-relay fragmentation voice-adaptive deactivation 15
frame-relay traffic-shaping
frame-relay interface-dlci 16 ppp Virtual-Template1
frame-relay lmi-type ansi

interface Virtual-Template1
no ip address
ppp multilink
ppp multilink group 1

interface Multilink1
description Bundled Interface
bandwidth 4000
ip address 10.87.1.1 255.255.255.248
ppp multilink
ppp multilink group 1

All routers including the oldest (e.g. Cisco 2500 series) and smallest (e.g. Cisco 800 series) support NetFlow. Some functions does not exist in older IOS versions.
Catalyst 6500 series switches support NetFlow. Catalyst 4500 series switches support NetFlow with Supervisor IV/V + WS-F4531 Catalyst 4500 NetFlow Services Card.

Standalone or stackable switches do not support NetFlow. This means Catalyst 4948, Catalyst 3750 or Catalyst 3560 series switches do not support NetFlow. You can see the necessary commands on config mode, but they are not effective. It is not about IOS version or feature set. You need a modular switch for NetFlow.

Unfortunately, the answer of “What Cisco switches support netflow?” is only the modular switches.

I had a chance to install a new Cisco 3845 with 1GB memory. Everything was fine except SSH access that I needed because of security policy. I searched the web and found “Configuring Secure Shell on Routers and Switches Running Cisco IOS” document on Cisco web site. It was a little bit different than current one. I made a configuration as explained, but it was not good enough and access to the router via SSH was not possible.

About one week later, I realized that

• SSH only supports authentication with username/password, but it does not support just access password like telnet
• So, I had to create a user and set a password with username command
• I had to enable aaa new-model OR issue login local command under line vty for username/password authentication.
• Also a hostname and a domain name were required to generate the keys, since router uses its FQDN as the label of the key pair.
• SSH is enabled by default and I do not need to enable it myself.

Necessary steps to enable SSH are mentioned below.

PRECONDITION: You need a K9 IOS (newer than 12.1) to enable SSH. Catalyst 2900 Series switches do not support SSH.

1) You MUST set a host name
hostname ciscolab

2) You MUST set a ip domain name
ip domain-name mydomain.com

3) You MUST enable aaa new-model OR set “login local” under vty configuration but not just “login”
aaa new-model

4) You MUST create a user
username sshtest password 0 sshpass

5) You MUST generate RSA keys
crypto key generate rsa

if you have RSA keys before you will receive a message, type yes
% You already have RSA keys defined named ciscolab.mydomain.com.
% Do you really want to replace them? [yes/no]: yes

it will ask for modulus size, 1024 is fine (it depends your security needs)
How many bits in the modulus [512]: 1024

6) You MUST set vty access method to all OR ssh (if you chose ssh telnet will be disabled)
line vty 0 4
  transport input ssh

7) By using SecureCRT (licensed) or Putty (free), chose SSH1 (SSH in Putty) for the protocol enter hostname or IP address and click connect (Open in Putty). It will ask for username and password. Do not touch the other settings, you do not need them.

If you are able to access your device with SSH and still have some other questions please have a look at Advanced SSH settings for Cisco IOS .