Category Archives: CISCO

For All CISCO blogs

The mystery of Cisco 2960s and strange ARP cache

Posted on by
Reply

This all started when I introduced a new network monitoring tool on to the network, the tool was “cisco prime”, but before I say any more lets be clear the issue here has nothing to do with prime which is a great tool for managing cisco devices. I noticed that when pushing new IOS files and backing up switch configuration that some time they would seem to lose network connectivity. I was able to ping them and ssh to them from my desktop, but they would simple not speak to the prime server.

So lets start with the set up (ip address modified of course)

Prime server – a vmware guest , in vlan 1, ip address 10.10.224.98/21

my desktop – physical machine, in vlan 1 also, ip address 10.10.226.46/21

Switch – management interface in vlan 666, ip address 20.20.255.6/24

interface Vlan666
ip address 20.20.255.6 255.255.255.0
no ip proxy-arp
end
switch#sh ru int vlan 1
Building configuration…
Current configuration : 65 bytes
!
interface Vlan1
no ip address
no ip proxy-arp
shutdown
end

Router – 6506 with a live interface in both vlan 1 and 666 set as DFGW on clients.

So to start with I can ping every thing and every thing can ping every thing else, and on the switch with a show arp I see

switch#sh arp
Protocol Address Age (min) Hardware Addr Type Interface
Internet 20.20.255.1 0 4403.a754.8300 ARPA Vlan666
Internet 20.20.255.6 – 7010.5c99.f241 ARPA Vlan666
switch#

So all looking good, I can see the switch IP address and that of the DFGW

switch#ping 10.10.224.98
Type escape sequence to abort.
Sending 5, 100-byte ICMP Echos to 10.10.224.98, timeout is 2 seconds:
!!!!!
Success rate is 100 percent (5/5), round-trip min/avg/max = 1/2/6 ms

But then the ping stops working? and the switch can no longer contact the prime server, however I can still see it from my desktop? Logging back on to the switch I again look at the ARP cache..

Protocol Address Age (min) Hardware Addr Type Interface
Internet 10.10.224.99 2 0050.5683.61ce ARPA Vlan1
Internet 20.20.255.1 0 4403.a754.8300 ARPA Vlan666
Internet 20.20.255.6 – 7010.5c99.f241 ARPA Vlan666

What?? why has the switch learnt a mac address on vlan 1? I have two issues with it doing this, first Prime is still trying to contact it via the DFGW ( I can see this in a packet trace) so the switch is not seeing the packets coming in on vlan 1, second the interface for vlan 1 is shut down so there should be no ARP entries on it! to get it working I clear the ARP cache of this entry and all is good… well for a few days / hours and then it happens again, but the time between issues seems very random. Keeping an eye on it I see it happen with anther monitoring server, and then another? The one thing I notice is that all the times it happens it is always a server on vmware. Physical servers/appliance and desktops never seem to have this issue. This is the first piece of the puzzle, what does Vmware do different to other servers? They Migrate! And when they migrate between the physical hosts the vmware system sends a gratuitous arp on to the network to alert switches what port in the network to now find the server on? And some switches that have “ip arp gleaning” switched on which is the default hear this and place an entry in to there ARP table. Even though the switch had the interface vlan 1 shut down, it still passed vlan1 traffic through the switch at layer 2 and this seemed to be enough that it saw the ARP packet, and added the entry to its ARP table. Then of course the it try’s to use this entry for communication but as the interface is indeed shut it will not work!

A little bit of time with CISCO TAC and the solution was to disable IP arp gleaning on all the access switches, it might be useful for provisioning switches but as I found it can cause issues.

no ip arp gleaning tftp
no ip arp gleaning udp

The fact it was learning on a disabled interface is a bug and something CISCO are looking in to.

However that’s not quite the end of the story, disabling “IP ARP Gleaning” did not work! and this had us scratching out heads for a while, until one of the cisco engineers I was talking to noticed this..

Cisco Bug CSCun38166

“no ip arp gleaning tftp and udp” doesn’t work

CSCun38166

 Description
Symptom:
On 2960,when we configure “no ip arp gleaning tftp” and “no ip arp gleaning udp” then do
tftp use command”copy flash:teset tftp:”
It is expected following behavior:1.do not learn a IP address from other segment.
2.add a arp entry corresponding with HSRP virtual mac addressHowever in our case,it turns out like below:
1.generate a arp entry of IP address from other segment
2.add a arp entry corresponding with HSRP PHY mac address.

Conditions:
1.no ip arp gleaning tftp
OR
no ip arp gleaning udp
2.do tftp use following command
copy flash:teset tftp:

And looking at the version code, oh yes I would be running one of the affected versions. 15.0(2)SE6, a quick upgrade to version SE7 and all is good.

I am impressed though, hitting 2 cisco bugs in one issue.

And in the end it was prime that sorted it all out, a single click and it pushed the IOS update and the configuration for “no ip arp gleaning” to all 2960s affected devices, not going to ake this a post for plugging prime, but it does have its good points.

Download article as PDF

Notepad++ Editing Linux with SFTP.

Posted on by
3

Now if you haven’t tried Notepad++ get over to http://notepad-plus-plus.org/ and give it a go. It is once of the best text editors I have come across, not only is is light weight and gives you tabs for you files, it also has syntax highlighting for a good number of programming and scripting languages, can be run from a USB drive with out installing and On top of all that its a free download, so perfect in every respect.

Nopepad++ screen shot

Then there is the standard “compare” functions, ability to write macros, and a whole host tool for editing text.

But just a few days ago I found another function that I had not noticed before. Under the plugins one of the defaults is “Nppftp” clicking on this brings up a side panel that allows you to set up profiles for FTP and SFTP connections to remote servers.

Once you have set one up (SFTP for Linux ) and connected the remote file system will appear in a tree structure in this pane, allow you top open the files in Notepad++ edit and re-save them. And you get all the standard benefits of the application such as highlighting of code and copy paste between multiply files.

One thing to be careful of is the EOL settings, when creating new files, these defaults to windows format that will be miss read by the Linux system (look up EOL in google for more ifo on this). This can be changed either for an individual file under “Edit >> ELO Conversion >> Unix Format” on the menu bar. Or under “Settings >> preferences >> New Documents/….. ” to change the default.

So that’s one bonus point about Notepad++, but honestly even if this is not something you would use, give it a try, it has so many features over the basic notepad that comes with windows that there is something there for every one.

Download article as PDF

More Generating Test Data.

Posted on by
Reply

So Last time I mentioned you can generate traffice between two routers using TCP small Servers. This does work fine but there are some limitations, it can’t genetrate large amounts of traffic, it puts a high load on the CPU and it does not tell you much once it has completed.

A second method I came across is “TTCP” (Test TCP) which is avalible on many of the more recent IOS versions (11.2 upwards). This method not only gives you more control over the data that is sent, but also will provided you with infomation on how the trasfer of data went once it is complete. TTCP is also avalible for Windows and Linux/Unix, which means you can test between various end stations on the network.

It is all very simple to set up, simply at the “>” or “#” prompt type “TTCP”, and then follow the prompts. You will need to set up one router as the sender first, and then the second as the reciver. You can leave all the settings as default (you may want to reduce the “nbuf” setting as the default of 2048 can take some time to complete, espicaly on a slow link).

Once complete you will get an out put of time taken and bandwith achived among other stats. Again this is a very simple tool to give you a indication as to the state of a link, you can find more details in the below link to the CISCO site.

Cisco TTCP Document

Download article as PDF

Generating traffic from a router

Posted on by
Reply

I was looking how to do this to test a monitoring system using GNS 3, and came across this little gem

http://etherealmind.com/the-poor-mans-ios-traffic-generator/

nothing fancy just enabling the small TCP servers (#service tcp-small-servers), but a nice simple way to push some data across a link between two routers. The article also mentions some other methods so will be looking in to them as well.

Always nice to be able to test some things with out having to fire up servers or other hard/soft ware 🙂

Download article as PDF

New Job,

Posted on by
Reply

A few weeks ago I accepted a new job role, and I can’t wait to get started. I think I have mentioned before on this blog that I origianl trained as a scientist and for a time worked in a science lab before moving on to networks.

Well I am now going back in to the lab, but this time to help grow and develop there network through a large restructuring project, that will see the building of several world leading laboratory buildings and a through overhaul of the entire site. And the best part about it is I will have a major role in all of it. 🙂 With lots of hands on across lots of different network technologies, and while some are things I have experience with there are a few new ones as well to get my teeth in to 🙂

So after a fairly mundane year as far as networking has gone, I think getting back in to it will really give me the motivation to get back  studying and learning it all. Last year has not been standing still by any means, but has been a little slow in terms of networking.

To head up the network on one of the world leading laboratories for study of animal diseases is really a dream job for me, combining both of my hobbies in to one. All the hard work of the last 5 or so years really playing off..

Once I have started the role and settled in I will be posting a few more details about the position and where I can what things I am getting up to.

Download article as PDF

SecureCRT sending commands to multiple sessions.

Posted on by
5

I came across this in secureCRT and thought I would share it.

When labing things up (and indeed on real networks), there are times when you need to send the same command to multiple devices. you can of course copy and paste between the sessions but what about if you want to past the exact same block of configuration to 20 devices, or just want to do something simple like save the running configuration on your devices in you lab before you close down?

Well SecureCRT has a nice little feature to do this, so before enabled secure CRT looks much like below, as you can see I have several tabs open.

Default SecureCRT Window

However by going in to the view menu up the top there is a option to enable the “chat window”, this will bring up an extra panel at the bottom of the screen. Then by right clicking in this new panel you can enable the “send chat to all tabs” option as shown below.

Chat window enabled

Now any command typed in the chat window will be sent to all devices. Commands typed in the main terminal are still only sent to a single device.

What would be even nicer is if you could highlight multiple tabs and have the commands only sent to those terminal sessions. At the moment it is an all or nothing solution, maybe I will go suggest it to them as an improvement for future versions 🙂

The more I use CRT the more I like it, written quite a few scripts for it now, if you know any VB script or Java you can pretty much do what ever you like as SecureCRT has a nice simple API in to it.

I am finally moving house this week, so after that should have more time to post on here, and will take some of the script I have and tidy them up and post them for people.

Take care

DevilWAH

Download article as PDF

Switch Vs Route

Posted on by
1

I see a lot of people as Questions such as what is harder the Switch or the Route exam, Or Why is the Route coarse materials so much larger than the Switch, does this mean there is less to it? 642-902 ROUTE  642-813 SWITCH

So having now completed both foundation and cert guides here are my views.

First the two have very different goals that they are trying to teach, and approach things in the same way as you would likely see in the Real world.  

ROUTE

In the real world generally Routing protocols stand apart, while you may run EIGRP and OSPF with in he same organisation, most people will keep them separate and they will only interact at the borders. And there are only 3/4 major routing protocals that you woudl expect to see.

RIP,

OSPF,

EIGRP and

BGP.

While there are others these are the common ones that most people will using there jobs. So the ROUTE exam deals with these along with redistributing the routes between them.

This give the following Topics to study

EIGRP
OSPF
BRP
Redistribution and Patch control
IPv6

And each is covered in some detail.

SWITCH

On the other hand has many more topics, and in the case of switch’s many of these will be run on the same devices across the entire network, (eg. VLANS, Spanning Tree, ACL’s Switch Security) so the number of topics in the SWITCH exam is much higher. They are covered in less depth individual than the topics in ROUTE, however you are expected to understand how they all work together and how issues configuring one can cause issues in others.

A partial list of topics covered in switch are.

VLANS
Switch Operation (CAM TCAM and other switch tables)
CEF
VLANS
STP (all modes)
STP enhancements like BPDU guard and ULD detection.
Ether channels and port channels
Multilayer switches
High availabilities (redundet router and redundant supervisors)
IP telephony
Wireless
Securing switch devices
Port security
ACL’s
Vlan ACL’s
Private VLANS
QOS
and the list goes on….

So the question of what one is hard and what one is easy will very much depend on the person taking them, and the current experience they have. Many people do seem to find the Routing exam nicer and I think this is because you can take each topic seperatly and concentrate with out worrying about the rest. While I enjoyed Switch as it was lots of bite size chunks to get stuck in to.

People also ask what one to take first, honestly I don’t think knowing either one will help learning the other one, as long as you have  a decent understanding of networks. Personal I would first go for the one you have most experience with, and get it under your belt first.

The only one I would suggest leaving till last is the Trouble Shoot as this assumes you have knowlage of both Switch and Route.

Download article as PDF

Using Syslog while Studying in GNS3 (or indeed and cisco Lab)

Posted on by
Reply

I have been getting back in to my studying a lot lately and one thing I have found is the need to use a lot of debug commands so I can watch what is happening during things like routing updates and neighbour formation. One thing I do find though is that I am forever having to turn debug on and off, forgetting to do one or the other, and when it is on it clutters up the screen a breaks up the config I am entering making it difficult to read back.

Which got me thinking, I have used syslog servers a lot in the past, so why not send all the debugging out put to a syslog server and turn of logging to the console? This way I can have all the debugs in one place, and keep the console of the devices tidy so I can see what I am doing.

Now if you are doing this through GNS3 you will need a cloud connection so your PC can talk to your GNS3 network. If you are not sure how to do this there are lots of videos and walk though on the net, however the one below is one of the best I have found, very clear and complete.

How-To: Using the Cloud in GNS3 to Provide Internet Access from Matthew on Vimeo.

So once you have your cloud set up you then need to set up a simple GNS3 topology, Here I have set up 4 routers running OSPF connected through a switch as I am looking at the DR and BDR election process.

I have given R1 and R2 F0/1 address 192.168.10.10 and 192.168.10.20, and the loopback adapter used by the cloud is 192.168.10.254. Once the routers are booted and connected to the cloud, check they can ping the loop-back address (you may need to disable your fire wall on the loop-back connection.)

then of course you will need a SFTP server, in windows there are two good free choises, for a realy simple server that can run with out install try, http://tftpd32.jounin.net/tftpd32.html simple but does all you need, just make sure you disable dhcp and other none necessaries services in the settings. For a more complete tool try http://kiwisyslog.com/, they have a free syslog server offering that allows filtering and more.

In either case set it up and insure it is listing on the loopback interface, in the case of TFTP32d this is simple a case of choosing the interface from the drop down list.

Finale we need to change the logging setting of R1 and R2 to direct debugging message to the syslog server and not to the console. Remember debug messages are level 7 so we need to set console logging to level 6 or lower and trap logging to level 7. the following code will do just this from global config mode.

#logging 192.168.10.254
#logging console 6
#logging trap 7

So now we can enable the debugging and reset the neighbour relation ships to see what it looks like.

From the console

So not much there apart from we see the neighbours bounce as I clear the OSPF process.

So how about on the syslog server?? 

So here are all our debug messages, for us to scroll through and review at our leisure, If you have something like Kiwicat syslog server you could filer them in to views, based on device that sent it, or text with in message, ect.

You need to make sure of course that you either have the device connected directly to the syslog server network, or it has a route to get there. Directly connected is always best of course as you will insure that as long as that interface on the device is up you will catch all messages. On real hardware simply use a spare switch or create a separate VLAN and do exactly the same thing.

I have found for large labs this works great, indeed for testing setups for clients its great as well. once you have insured the correct debugging is enabled you can walk though test scripts and plans, safe in the knowledge that you have a full detailed log of every thing that has happened.

Simple to set up and hopefully some of you will find it useful.

DevilWAH

Download article as PDF

I need to learn more C#

Posted on by
Reply

Been playing around with it a lot, but come to the conclusion that I need to learn a bit more out of a book, than just using google and Microsoft sites. To be fair not done badly just a few areas I have been working on lately the code is a bit messy and I know there must be a better way to do it. So to help me on my way I have done two things.

Number one. Brought a Kindle, or to be more accurate my wife has got me one, just awaiting its arrival.

Number two. Getting hold of “Headfirst in to C#” and a few other C# books for it.

Hopefully by the end of this week I will have achieved the following in my application.

  1. Added controls to the output form for “Next config” and “Cancel”, Plus a button to copy to clipboard or download to file.
  2. Added the ability to create a single output from multiply lines in the entry form, and ability to chose a single line and out put only that config.
  3. After that I will be adding a header and footer section to the templates, that will all user to enter varibles aht are to be standard to all configs. Things like company information, user name, date, etc.

All simple stuff but just making sure it do it right now so when it comes to updating later it is all nice and clear. All about adding controls at run time at the moment and making sure its done in the most efficient way.

Download article as PDF

Continuing in to C# and Automation.

Posted on by
Reply

So still not had much time to be studying CCNP, but have been getting my hands dirty on C# around the automation of generating configuration for cisco devices.

As mentioned before part of my job is change management which means a lot of repetitive configurations. In the past the tools I have written in VB and Excel have been limited to a single master config, what I decided was needed was a tool that could take a generic config written in notepad or other simple text editor and present the variables to the end user in a friendly form, for them to complete and generate the config.

Confgen

Screen shot of early version of Confgen

This is currently has a simple template loaded for changing the vlan and description of a port and enabling / disabling it. But the application is flexible to be able to take any length or complexly of config you can enter.

There are still lots of things to tidy up,

currently only the first row of variables entered can be parsed in to a complete script, I also want to eventually use a word template to give a professional output for the final script. And have a menu driven system that will look for files in a specified folder at start up and load them ready to be picked by the end user.

However its a start and the core of the application is working. I would say its still very much a Beta version, but am hoping over the coming weeks and months to develop it in to a much more compete tool. A few bugs to resolve and functions to add but I hope it gives people an idea of what i am looking achieveive.

I also have a stage 2 idea to add to this that will make it even mousefulull for new deployments to build up complex configurations, as well as these simple snipits.

If any one wants to try it out the zip file is below, simple extract to a folder and run, file >> open. to open the included example. (you will need .net version 3.5 framwork installed to run it)

Confgen

Please play around and let me know what you think.

Edit: added page to site to keep track of this tool and its development, see above.

DevilWAH

Download article as PDF