The mystery of Cisco 2960s and strange ARP cache

Posted on by
Reply

This all started when I introduced a new network monitoring tool on to the network, the tool was “cisco prime”, but before I say any more lets be clear the issue here has nothing to do with prime which is a great tool for managing cisco devices. I noticed that when pushing new IOS files and backing up switch configuration that some time they would seem to lose network connectivity. I was able to ping them and ssh to them from my desktop, but they would simple not speak to the prime server.

So lets start with the set up (ip address modified of course)

Prime server – a vmware guest , in vlan 1, ip address 10.10.224.98/21

my desktop – physical machine, in vlan 1 also, ip address 10.10.226.46/21

Switch – management interface in vlan 666, ip address 20.20.255.6/24

interface Vlan666
ip address 20.20.255.6 255.255.255.0
no ip proxy-arp
end
switch#sh ru int vlan 1
Building configuration…
Current configuration : 65 bytes
!
interface Vlan1
no ip address
no ip proxy-arp
shutdown
end

Router – 6506 with a live interface in both vlan 1 and 666 set as DFGW on clients.

So to start with I can ping every thing and every thing can ping every thing else, and on the switch with a show arp I see

switch#sh arp
Protocol Address Age (min) Hardware Addr Type Interface
Internet 20.20.255.1 0 4403.a754.8300 ARPA Vlan666
Internet 20.20.255.6 – 7010.5c99.f241 ARPA Vlan666
switch#

So all looking good, I can see the switch IP address and that of the DFGW

switch#ping 10.10.224.98
Type escape sequence to abort.
Sending 5, 100-byte ICMP Echos to 10.10.224.98, timeout is 2 seconds:
!!!!!
Success rate is 100 percent (5/5), round-trip min/avg/max = 1/2/6 ms

But then the ping stops working? and the switch can no longer contact the prime server, however I can still see it from my desktop? Logging back on to the switch I again look at the ARP cache..

Protocol Address Age (min) Hardware Addr Type Interface
Internet 10.10.224.99 2 0050.5683.61ce ARPA Vlan1
Internet 20.20.255.1 0 4403.a754.8300 ARPA Vlan666
Internet 20.20.255.6 – 7010.5c99.f241 ARPA Vlan666

What?? why has the switch learnt a mac address on vlan 1? I have two issues with it doing this, first Prime is still trying to contact it via the DFGW ( I can see this in a packet trace) so the switch is not seeing the packets coming in on vlan 1, second the interface for vlan 1 is shut down so there should be no ARP entries on it! to get it working I clear the ARP cache of this entry and all is good… well for a few days / hours and then it happens again, but the time between issues seems very random. Keeping an eye on it I see it happen with anther monitoring server, and then another? The one thing I notice is that all the times it happens it is always a server on vmware. Physical servers/appliance and desktops never seem to have this issue. This is the first piece of the puzzle, what does Vmware do different to other servers? They Migrate! And when they migrate between the physical hosts the vmware system sends a gratuitous arp on to the network to alert switches what port in the network to now find the server on? And some switches that have “ip arp gleaning” switched on which is the default hear this and place an entry in to there ARP table. Even though the switch had the interface vlan 1 shut down, it still passed vlan1 traffic through the switch at layer 2 and this seemed to be enough that it saw the ARP packet, and added the entry to its ARP table. Then of course the it try’s to use this entry for communication but as the interface is indeed shut it will not work!

A little bit of time with CISCO TAC and the solution was to disable IP arp gleaning on all the access switches, it might be useful for provisioning switches but as I found it can cause issues.

no ip arp gleaning tftp
no ip arp gleaning udp

The fact it was learning on a disabled interface is a bug and something CISCO are looking in to.

However that’s not quite the end of the story, disabling “IP ARP Gleaning” did not work! and this had us scratching out heads for a while, until one of the cisco engineers I was talking to noticed this..

Cisco Bug CSCun38166

“no ip arp gleaning tftp and udp” doesn’t work

CSCun38166

 Description
Symptom:
On 2960,when we configure “no ip arp gleaning tftp” and “no ip arp gleaning udp” then do
tftp use command”copy flash:teset tftp:”
It is expected following behavior:1.do not learn a IP address from other segment.
2.add a arp entry corresponding with HSRP virtual mac addressHowever in our case,it turns out like below:
1.generate a arp entry of IP address from other segment
2.add a arp entry corresponding with HSRP PHY mac address.

Conditions:
1.no ip arp gleaning tftp
OR
no ip arp gleaning udp
2.do tftp use following command
copy flash:teset tftp:

And looking at the version code, oh yes I would be running one of the affected versions. 15.0(2)SE6, a quick upgrade to version SE7 and all is good.

I am impressed though, hitting 2 cisco bugs in one issue.

And in the end it was prime that sorted it all out, a single click and it pushed the IOS update and the configuration for “no ip arp gleaning” to all 2960s affected devices, not going to ake this a post for plugging prime, but it does have its good points.

Download article as PDF

Notepad++ Editing Linux with SFTP.

Posted on by
3

Now if you haven’t tried Notepad++ get over to http://notepad-plus-plus.org/ and give it a go. It is once of the best text editors I have come across, not only is is light weight and gives you tabs for you files, it also has syntax highlighting for a good number of programming and scripting languages, can be run from a USB drive with out installing and On top of all that its a free download, so perfect in every respect.

Nopepad++ screen shot

Then there is the standard “compare” functions, ability to write macros, and a whole host tool for editing text.

But just a few days ago I found another function that I had not noticed before. Under the plugins one of the defaults is “Nppftp” clicking on this brings up a side panel that allows you to set up profiles for FTP and SFTP connections to remote servers.

Once you have set one up (SFTP for Linux ) and connected the remote file system will appear in a tree structure in this pane, allow you top open the files in Notepad++ edit and re-save them. And you get all the standard benefits of the application such as highlighting of code and copy paste between multiply files.

One thing to be careful of is the EOL settings, when creating new files, these defaults to windows format that will be miss read by the Linux system (look up EOL in google for more ifo on this). This can be changed either for an individual file under “Edit >> ELO Conversion >> Unix Format” on the menu bar. Or under “Settings >> preferences >> New Documents/….. ” to change the default.

So that’s one bonus point about Notepad++, but honestly even if this is not something you would use, give it a try, it has so many features over the basic notepad that comes with windows that there is something there for every one.

Download article as PDF

More Generating Test Data.

Posted on by
Reply

So Last time I mentioned you can generate traffice between two routers using TCP small Servers. This does work fine but there are some limitations, it can’t genetrate large amounts of traffic, it puts a high load on the CPU and it does not tell you much once it has completed.

A second method I came across is “TTCP” (Test TCP) which is avalible on many of the more recent IOS versions (11.2 upwards). This method not only gives you more control over the data that is sent, but also will provided you with infomation on how the trasfer of data went once it is complete. TTCP is also avalible for Windows and Linux/Unix, which means you can test between various end stations on the network.

It is all very simple to set up, simply at the “>” or “#” prompt type “TTCP”, and then follow the prompts. You will need to set up one router as the sender first, and then the second as the reciver. You can leave all the settings as default (you may want to reduce the “nbuf” setting as the default of 2048 can take some time to complete, espicaly on a slow link).

Once complete you will get an out put of time taken and bandwith achived among other stats. Again this is a very simple tool to give you a indication as to the state of a link, you can find more details in the below link to the CISCO site.

Cisco TTCP Document

Download article as PDF

Generating traffic from a router

Posted on by
Reply

I was looking how to do this to test a monitoring system using GNS 3, and came across this little gem

http://etherealmind.com/the-poor-mans-ios-traffic-generator/

nothing fancy just enabling the small TCP servers (#service tcp-small-servers), but a nice simple way to push some data across a link between two routers. The article also mentions some other methods so will be looking in to them as well.

Always nice to be able to test some things with out having to fire up servers or other hard/soft ware 🙂

Download article as PDF

New Job,

Posted on by
Reply

A few weeks ago I accepted a new job role, and I can’t wait to get started. I think I have mentioned before on this blog that I origianl trained as a scientist and for a time worked in a science lab before moving on to networks.

Well I am now going back in to the lab, but this time to help grow and develop there network through a large restructuring project, that will see the building of several world leading laboratory buildings and a through overhaul of the entire site. And the best part about it is I will have a major role in all of it. 🙂 With lots of hands on across lots of different network technologies, and while some are things I have experience with there are a few new ones as well to get my teeth in to 🙂

So after a fairly mundane year as far as networking has gone, I think getting back in to it will really give me the motivation to get back  studying and learning it all. Last year has not been standing still by any means, but has been a little slow in terms of networking.

To head up the network on one of the world leading laboratories for study of animal diseases is really a dream job for me, combining both of my hobbies in to one. All the hard work of the last 5 or so years really playing off..

Once I have started the role and settled in I will be posting a few more details about the position and where I can what things I am getting up to.

Download article as PDF

SecureCRT sending commands to multiple sessions.

Posted on by
5

I came across this in secureCRT and thought I would share it.

When labing things up (and indeed on real networks), there are times when you need to send the same command to multiple devices. you can of course copy and paste between the sessions but what about if you want to past the exact same block of configuration to 20 devices, or just want to do something simple like save the running configuration on your devices in you lab before you close down?

Well SecureCRT has a nice little feature to do this, so before enabled secure CRT looks much like below, as you can see I have several tabs open.

Default SecureCRT Window

However by going in to the view menu up the top there is a option to enable the “chat window”, this will bring up an extra panel at the bottom of the screen. Then by right clicking in this new panel you can enable the “send chat to all tabs” option as shown below.

Chat window enabled

Now any command typed in the chat window will be sent to all devices. Commands typed in the main terminal are still only sent to a single device.

What would be even nicer is if you could highlight multiple tabs and have the commands only sent to those terminal sessions. At the moment it is an all or nothing solution, maybe I will go suggest it to them as an improvement for future versions 🙂

The more I use CRT the more I like it, written quite a few scripts for it now, if you know any VB script or Java you can pretty much do what ever you like as SecureCRT has a nice simple API in to it.

I am finally moving house this week, so after that should have more time to post on here, and will take some of the script I have and tidy them up and post them for people.

Take care

DevilWAH

Download article as PDF

Raspberry Pi

Posted on by
2

I am sure a lot of you have hear about these, I have been trying to get one since the day it was released but still in the queue. http://www.raspberrypi.org/

Response was overwhelming and took down the sites of distributes, and this is what it is all about!

While it may not look much this is a 700mhz ARM processor, can play back Video at HD, with USB, Ethernet, HDMI output + more all on a board the size of a credit card, and for the sum of £22 ($30ish).  Perfect for a home brew media server or web server. or indeed many other hobby projects.

But it gets better the whole purpose behind this is to encourage kids to get in to programming. There are a lot of kids around that can use computers but as computers get easier to use, less of them a delving behind the screens to see how they work and more importantly how to make them work. So later in the year they will be releasing them in to the schools and collages. pre installed with tools and languages for the kids to have a play on.

One thing that hasn’t been widely covered is that when you purchase one you will be able to chose to purchase an additional unit that will be donated on your behalf to a school or other good cause. I think its a great idea, and while I will be getting one or two to play with over the next few weeks I hope for my own enjoyment and leaning experience. I hope the developers do see there dream bear fruit and get taken up by the schools and colleges to encourage the next round of IT stars.

I would encourage any one who is thinking of purchasing one to remember that they are a charity, and while the have produced something that many of us will get enjoyment from. That where possible we should try to feed back and encourage there idea of getting them in every school and collage in the country.

As soon as I get my hands on one I will be updating this blog, who knows in a month or two it might well be running from one 🙂

DevilWAH

Download article as PDF

Switch Vs Route

Posted on by
1

I see a lot of people as Questions such as what is harder the Switch or the Route exam, Or Why is the Route coarse materials so much larger than the Switch, does this mean there is less to it? 642-902 ROUTE  642-813 SWITCH

So having now completed both foundation and cert guides here are my views.

First the two have very different goals that they are trying to teach, and approach things in the same way as you would likely see in the Real world.  

ROUTE

In the real world generally Routing protocols stand apart, while you may run EIGRP and OSPF with in he same organisation, most people will keep them separate and they will only interact at the borders. And there are only 3/4 major routing protocals that you woudl expect to see.

RIP,

OSPF,

EIGRP and

BGP.

While there are others these are the common ones that most people will using there jobs. So the ROUTE exam deals with these along with redistributing the routes between them.

This give the following Topics to study

EIGRP
OSPF
BRP
Redistribution and Patch control
IPv6

And each is covered in some detail.

SWITCH

On the other hand has many more topics, and in the case of switch’s many of these will be run on the same devices across the entire network, (eg. VLANS, Spanning Tree, ACL’s Switch Security) so the number of topics in the SWITCH exam is much higher. They are covered in less depth individual than the topics in ROUTE, however you are expected to understand how they all work together and how issues configuring one can cause issues in others.

A partial list of topics covered in switch are.

VLANS
Switch Operation (CAM TCAM and other switch tables)
CEF
VLANS
STP (all modes)
STP enhancements like BPDU guard and ULD detection.
Ether channels and port channels
Multilayer switches
High availabilities (redundet router and redundant supervisors)
IP telephony
Wireless
Securing switch devices
Port security
ACL’s
Vlan ACL’s
Private VLANS
QOS
and the list goes on….

So the question of what one is hard and what one is easy will very much depend on the person taking them, and the current experience they have. Many people do seem to find the Routing exam nicer and I think this is because you can take each topic seperatly and concentrate with out worrying about the rest. While I enjoyed Switch as it was lots of bite size chunks to get stuck in to.

People also ask what one to take first, honestly I don’t think knowing either one will help learning the other one, as long as you have  a decent understanding of networks. Personal I would first go for the one you have most experience with, and get it under your belt first.

The only one I would suggest leaving till last is the Trouble Shoot as this assumes you have knowlage of both Switch and Route.

Download article as PDF