Filtering you Logs

I came across this feature today and thought I would share it with you.

The standard way we are taught to set up logging for Cisco IOS devices is something along the lines of,

(config)#logging buffer 5
(config)#logging trap error

where you state where you want to log to and then what level of messages you wish to log in the range 0 to 7, with 0 the most critical and 7 debug messages.

This is all fine until you find you want to log a specific alert that is informational (level 6), but you don’t want to log every level 6 event that happens. My example of this was I wanted to log a specific ACL match but not link sate or other notification level alerts (ACL’s log at level 6 / informational by default).

Cisco in there wisdom have though of this for us and given us the “discriminator” command to allow us to manage logging in more detail. The outline is that we can create a named “discriminator” that will filter out / capture events based on things like the facility, the mnemonics, a string with in the msg-body, or the severity level. This can then be used in further logging statements to determine what happens to these messages. So lets have a quick example to see how it works. As I said in my case it was an ACL I wanted to log from so lets go for that.

First we can set up an access list with a logging statement in it and assign it to an interface

(config)#ip access-list extended ACS_Critical
(config-ext-nacl)#permit ip any any eq bootps log
(config-ext-nacl)#permit ip any any
(config-ext-nacl)#exit
(config)#int vlan 888
(config-if)#ip address 192.168.5.254 255.255.255.0
(config-if)#ip access-group ACS_Critical in
(config-if)#exit

So we now have an access list that will log any DHCP traffic received coming in on the VLAN 888 interface, In my cace VLAN 888 is a critical VLAN that client pc’s will end up in if there are problems with network authorisation. So testing for DHCP traffic will let me know if the VLAN is being used and therefore if there is a authorisation issue. (last line of checking)

As I said before though ACL’s log at level 6, but if you watch a switch you will see a lot of lthese messages appearing in the day to day running of the switch and most of them you don’t need to worry about. Generally I would only save to buffer and forward to a syslogging server any level 5 (notification) or higher messages. But I do want to see the ones from this ACL…. We can set it up to do this as below

(config)#logging trap 5
(config)#logging discriminator ACS msg-body includes ACS_Critical
(config)#logging host 192.168.5.50 discriminator ACS

The Trap statement sets the default level for logging to the syslog server, the logging discriminator creates the discriminator called ACS and will filter incoming events for a message body containing ACS_Critical (name of the access list), and we then apply this to the syslog host when we set it up. Now all the default level 5 – 0 events will be logged plus the specific ACL’s generated ones.

A single host can have one discriminator assigned to it, but this can have multiply statements, and much like an access list these can allow or deny event to be passed, the commands are “includes” as used above, or “drop” to prevent a message getting logged. I find it especially useful for filtering logging to the syslog servers, as these log files can get huge if you are not careful.

Attached here is a CISCO document with more examples of setting this up.

Hope you found that intresting, and now I’m of to the seaside with wife and baby for a few days to fly my new kite 🙂 have a great weekend all.

DevilWAH

PS. I promise once I get back I will be getting on with my CCNP ROUTE posts as I said I would. Just been a busy week that I will tell you all about soon. I have also been asked to set up a web server and a few other projects so time has been tight. But I have had time to do some reading. I think I will start of with a review of sub-netting before I hit ROUTE core topics.

Visiting the Outside from the Inside (or DNS re-writing)

For a while now I have had an idea in the back of my head to sort out a long standing issue. However due to the fact this has been and issue on a secondary network, and only affected myself (to which I had a easy work around as you will see), I have not pressed to hard to find the solution. However I had promised my self that after my exam last week I would sit down and sort it out once and for all.

So here’s the situation, on a small network you have a client in one network, a web server in a second subnet and a single router acting as both the Internet gateway router, the firewall and handling all internal routing. Now this web server is of course accessible to the outside world as many web servers are. However the Domain name of the web server is not a domain I have control over. It is in fact a mirror of a 3rd party website. So they deal with all the DNS settings.

Fig.1

So of course the first thing to do is insure that outside users can connect in to the Webserver, and the internal clients can connect to the internet. So we require a basic NAT/PAT set up. Some thing along the lines of.

!
! First we set up the interfaces with there IP addresses and determin if they are
! inside or outside for NAT
!
(config)#int Dialer 1
(config-if)#description ## Internet link #
(config-if)#ip address 82.62.42.22 255.255.255.252
(config-if)#ip nat outside
!
!
(config)#int F0/0
(config-if)#description ## Client network #
(config-if)#ip address 192.168.10.254 255.255.255.0
(config-if)#ip nat inside
!
!
(config)#int F0/1
(config-if)#description ## Web Servers #
(config-if)#ip address 172.16.10.254 255.255.255.0
(config-if)#ip nat inside
!
! next we set up a list of IP address to be NATed from inside to outside
!
(config)#ip access-list standard NAT_IP’s
(config-std-nacl)#permit 192.168.10.0 0.0.0.255
(config-std-nacl)#exit
!
!
! Now set up the client NAT’ing to allow internet access for the clients and a
! static port forward, so all Packets comming in on the external IP address
! to port 80 get directed to the internal server.
!
!
(config)#ip nat inside source list NAT_IP’s interface Dialer 1 overload
(config)#ip nat inside source static tcp 172.16.10.30 80 82.62.42.22 80 extendable
!
!


So far all straight forward. Outside users can get to the web server and inside clients can use the internet (assuming firewall rules and so forth allow it). But what happens if the inside client tries to go to the web address Mirror3.companyX.com? Well if we assume you have not set up an internal DNS zone for companyX.com. Then the internal client will use CompanyX’s DNS server, to resolve the name to the external IP address of the network 82.62.42.22, just as an outside user would. This create a problem, due to how the router processes the steps on NAT and Routing, it will not correctly forward the packets between the client and the server. Instead you will end up with a host unreachable error (if you want more detail what is happening please comment and I will add it in). So how can you allow the internal hosts to browse the web server then?

Well you have 4 (OK I am sure there are more 😉  but these are the main ones) solutions.

1. Simply point the client web browser to the internal IP address of the server. As I was almost the only person who ever needed to get to the server from the inside this was the method I have been using up until now. However this does not scale well, and neither is it pretty, so I knew there had to be a better way.

2. You could set up a DNS zone for companyX.com with a record for mirror3.companyx.com using it’s internal IP address . By pointing your internal clients to this DNS server they would  pick up a rout able IP and browse the site. This is a standard method of DNS, and in many cases would be the preferred solution. However in this case it means setting up a Zone for a name you do not own. You also have to be careful to insure the inside clients can still resolve the mirror1 and mirror2 IP addresses. With out carefully planning this can get messy, especially if you have multiply mirrored servers from multiply domains.

3. Carry out NAT hair-pinning. This was original my first choice, and I do intend to come back to this as there are benefits to this method. But as I will cover also some issues with it. As I mentioned above trying to use the outside IP address of the server from the inside client results in issues to do with routing and NAT translations. Because the packets aren’t passing between the inside and outside the router can’t carry out correct NAT translations. However with a bit of cleaver configuration, and by using a loop back interface assigned to the NAT outside. You can with the use of multiply NAT statements and a static route get the router to pass the packet twice through the NAT algorithm and have it routed correctly. While this is a nice solution, its has the issue of huge CPU overheads for the router. Routing and the NAT has to be carried out in software with this method meaning that apart from in cases where there is very low traffic you need to be very careful if you implement this. (as I mentioned I will be covering how to set this up in a later post)

4. DNS re-writes! There is a not very well documented but very useful feature of NAT, called DNS re-writing, which does exactly what it says and by default is enababled. DNS response packets are checked by the router and any matching the NAT statements for the payload IP address will get re-written.  However this will not work on PAT statements, only on pure one to one NAT. So the configuration above will NOT invoke this feature as it is. To get it working we need to add one more command.

!
!
(config)#ip nat inside source static 172.16.10.30 82.62.42.22 extendable
!
!


Now with this command added the router will replace any IP address inDNS requests coming from outside the network that contain 82.62.42.22 with the internal IP address of 172.16.10.30. Now both the external users and the internal clients can use the same DNS server to resolve the address, and each will end up with the correct (but Different) IP address to be able to contact the web server.

To me solution 4 seems to be the neatest. Both companies keep full control of there networks and DNS settings, and there is much less over heads for the router to worry about. However note two issues.

First although we can still use PAT mappings on the IP external and internal address. So for instance where above we have a mapping set up “(config)#ip nat inside source static tcp 172.16.10.30 80 82.62.42.22 80 extendable” We could also have another mapping such as “(config)#ip nat inside source static tcp 172.16.10.40 22 82.62.42.22 22 extendable” Where SSH traffic goes to 172.16.10.40 and HTTP goes to 172.16.10.30. However any DNS request will use the plain NAT mapping so will always get replaced in the response with the 172.16.10.30 address in our set up. So you need to be aware of this and plan for it.

Second by using a NAT statement, all incoming traffic to any port will be translated through NAT. So you should insure that either an ACL or Firewall is blocking all traffic apart from what you want to come through.

So none are perfect but like with every thing in networking, you chose the one that fits best. And if you can’t find one, then your not looking hard enough. Or maybe you just need to invent a new method 😉

DevilWAH

CCNP Route

Well I Official started my ROUTE studies last night. 🙂 Back when I started this blog I was already well in to my study of the SWITCH material, and like wise creating Flashcard pack I made up with Anki was a rushed job as I went over the material. So I know neither the blog or the cards are really suitable for any one but me to use for studying.However for the ROUTE I want to do things correctly, even if this means it take me a little longer to achieve. I hope (and please let me know if I don’t) to post around two or three Lab based posts for each of the major area’s of the ROUTE material. Including EIGRP, OSPF, IPv4, BGP, Redistribution and IPv6. These will be tagged in category with a quick link on the menu bar for easy access. I also will be attempting to make a much better set of Flash cards this time round, that are more suitable for using as revision / memory aid. I hope combined these will make useful addition materials to people studying for there ROUTE exam.

Of course along side the ROUTE posts, I will also be trying to keep up with other interesting things I come across both inside and outside of IT. We shall have to see how things go. But I hope over the next few months I can really start to fill out this blog and produce some thing that other people will find useful. After all half the fun in learning things is to be able to pass it on to others.

On a side note I came across this today, EIGRP disababled by default. This is interesting for two reasons. The first of course being that it is nice to see CISCO have listened and have set the default to what 95% of us have to manualy set it to any way :). And secondly this comes from www.tekcert.com. Which is a new blog that Jeremy Cioara of CBT nuggets Fame is now posting on along with a fellow IT tec guru Adam. Another site to add to the Favourite bar, don’t worry if you forget it will be in the links above. 😉

DevilWAH

SDM in LINUX (Wine)

OK leaving CCNP SWITCH aside for a bit, I finely got around to setting up Linux to allow me to run SDM.

I should point out that I am not a great fan of SDM, but I do run the IOS based firewall on one of my small networks. And while I am happy to change the config of policy’s from the command line, it can be hard to visualise what is going on in 600+ lines of code. So I do fall back to it every now and then.

On the above network I have 100% Linux machines, which included those sitting in the management subnet. So up utill now if I wanted to run SDM I had to get out a windows laptop and plug it in, so for a while I have been looking how I could get this up and running.

There is no Linux SDM version but you can install SDM either on the PC or on the router.According to CISCO as long as the web browser has jarva script enabled, then if you use the version installed on the Router it will work. However despite 2 different versions of Firefox, Google Crome, and numerous attempts with Java versions. Trying this way would always hang at the same point on all three of the PC’s I tried it from. I also don’t really like the idea of running SDM from the router, it takes up space and resources and is another thing to go wrong.

So the alternative was to attempt to run SDM from with in Linux. You will read on the web that CISCO SDM is a Java based html applications, and so in theory you can simple copy the install file across from windows to Linux, move a few files around, and then open up your web browser and point it to the “launcher.html” file you will find in the install directory. However my attempt at this again proved unsuccessful. (I am not sure if this was due to the incorrect Java version I did try with a few but SDM is very fussy with Java and Linux is not so happy with multiply Java versions. (see here for instruction s for this method)

So I decided to go the whole hog and experiment with WINE. Wine for those of you who don’t know is a platform that allows you to run native windows application with in LINUX, I like to think of it as a windows emulator, however some purists will tell you this is not quite correct. But what ever it will allow you to run many windows application on LINUX, and while some people may rebel at the idea of that, I am more of the opinion if it works and gets the job done, then I don’t really have a problem.

So setting it all up.

The first thing to do is add the wine repository ( ppa:ubuntu-wine/ppa) to you distribution. In Unbuntu this can either be done using the option settings in the graphical package manager software, or by running the following command.

sudo add-apt-repository ppa:ubuntu-wine/ppa

Then update the repository cache, (“sudo apt-get update” from the command line).

If you are running the GUI package manager, search for wine and tick the wine1.2 install (at time of writing this is the current stable version, you should pick the latest stable). or from the cli type

sudo apt-get install wine1.2

Wine will now be installed.

You now need to get hold of CISCO SDM, Firefox 3.5 (must be 3.5 this will not work with version 3.6 due to java issues), and a copy of JAVA 6 update 11 (make sure it is this exact version SDM is very very picky).

Once you have downloaded them all, you can simple open them in the GUI, you may get an error saying that they are not executable files. Linux by default will not allow a file to be executed unless it has been set to be allowed. If you get this message simple right click, go to the properties and tick the execute box under the permissions tab. You can also run “sudo chmod +x <filename>” to achieve the same.

You should not be able to run the setup and follow the install exactly as you would under windows. Once you have installed all three, check you can open Firefox. You can find this either up in the application menu under

Wine >> wine applications >> firefox

Or you should have an short cut on the desk top (you may need to make this short cut executable like above).

You will also have a SDM shortcut on the desktop, however this will bring up the WINE IE browser which does not work, so you can’t use this direct.

Instead open up the Firefox you have just installed, and in the address bar type “c:” and hit “return” / click go. This will bring up a folder list for the Wine created windows file system. Open  “programs files” >> cisco systems >> SDM >> common files >> common files. Here you will find a file called Launcher.html which you want to open (I would also suggest add this as a short cut)

And there you are, CISCO SDM will now function as in windows, pop up boxes and all. You can even create a desktop icon that will pass the file above to Firefox if you wish.

Hope that’s of some help to people. If I get it running completely native with out the need for WINE I will be sure to let you know.

DevilWAH

CCNP SWITCH update

Well no luck I’m afraid 🙁

I agree with many of the other complaints about this exam, there seems to be a large number of questions that are not covered in the course material. I say that having read the foundation guide, cert guide, flash cards, and quick reference sheets.

CISCO have now made a statement that due to the high levels of complaints they will be reviewing the exam. So rather than wast time trying to pass it again. I will carry on my studies with the ROUTE course, which has had much better reviews, and come back to the SWITCH. Hopfuly by then CISCO will have sorted out the issues.

DevilWAH

Time up!

Well a night of study and now its to bed for some sleep before my CCNP SWITCH exam.

Hopefully by the time of my next post I will be one third of the way towards achieving my CCNP.

If I’m honest I have not been impressed with the Cisco Press books, or the BOSON test exam, both I have found many errors in. (Hopefully the fact I spot the errors means I understand the topics)

But all going well I will be back with a little something on getting the CISCO SDM to work in Linux soon.

Trouble Shooting with ACL’s (part 2, naughty CISCO and there firewall)

OK so following on from here  Trouble shooting with ACL’s (part 1).

To recap for un-know reasons packets had begun to get lost on one of my firewalls, and by using a combination of ACL’s applied to interfaces, logging commands and debug commands, I had established that while icmp packets sent from the router to the inside network where coming back in the interface. they where then some how getting lost with out any notifications.

Fig 1

So the last think I had done was enable the “#debug ip packets 150” on the router where 150 was an access list to capture any traffic to or from the 192.168.10.254 address. From this I was receiving (after a display of the packet going and coming) the following last line from the debug.

000801: Sep 13 12:40:35.452 UTC: pak 64A7D05C consumed in enqueue feature , packet consumed, CCE Firewall(2), rtype 0, forus FALSE, sendself FALSE, mtu 0, fwdchk FALSE.

This didn’t really help to much to start with, as any google searches on various parts of that got me no where. I then spent several moments looking at the firewall policy’s. I knew that the router set up as a fire wall, places any connected interfaces in to the “self” zone.  So although the interfaces 192.168.10.254 and 192.168.20.254 are servicing two manually configured zones (“inside2 and “management”), the ports them selves are actual part of the “self” zone. So I was looking for any policies between the “inside” and “self” and the “management” and “self” zones.

All I could find was a single policy that was assigned between the “inside” and “self” zone. However the direction for this was from the “inside” to “self“, that allowed ICMP and denied every thing else (so inside network can’t manage the router). So this still did not seem to explain the issue I was seeing, as the default policy unless configured is “self” is allowed to talk to any thing.

However after much searching on the internet I finally came across this.

"Although the router offers a default-allow policy between all zones and the self zone, if a policy is configured from any zone to the self zone, and no policy is configured from self to the router’s user-configurable interface-connected zones, all router-originated traffic encounters the connected-zone to self-zone policy on its return the router and is blocked. Thus, router-originated traffic must be inspected to allow its return to the self zone."

From Cisco’s documentation.

It goes on to describe how if a policy is applied in to “self“, then a policy must also be applied outgoing from self to the zone to allow return traffic to be inspected… So yes that little policy I had noticed above really was causing all the trouble. And guess how it got there?

Well it had originally been an ACL applied to the interface. But when I ran CISCO SDM to help configure Easy-VPN, it had asked to make changes to the fire wall to insure still worked. And created the policy for me and applied it.. Which is the reason for the title of the post. I don’t generally like to use the SDM, but for learning it is useful. However this just shows how important it is to check the configs first and insure you keep  record of exactly what it is doing in case problems arise.

Solution was simple, either remove the policy above and replace it as an ACL assigned to the interface, or others wise set up an out going policy from “self” to “inside“, to either allow all traffic and inspect (or just allow the traffic you want to go to self).

In my view you don’t want any traffic from “inside” to “self“, apart from ICMP. This allows you to check a user can see the DFGW, but prevents any management traffic, so stops any attacks on the router from users or compromised systems inside your network. (Oh if you use IP helper address for DHCP the router must also be able to see these through your policy).

But yes all working fine now and lots more learnt about fire wall policies. Been a slight distraction from my CCNP switch studies but these are still going well. Just 7 points to go over before the exam, all simple ones just want to go through configuring them once more. Wish me luck!!

DevilWAH

Trouble shooting with ACL’s

We all know of ACL’s for use in restricting traffic when applied to an interface, and also for classing traffic such as when used in NAT to chose the ranges to apply NATing to. But they can also be very useful in trouble shooting you network, and the last few days brought this back to me.

It all started with what seemed like a simple problem. On one of my networks the DHCP helper function had stopped working, and clients could no longer get an IP address. However a quick check of he DHCP server and a glance over the config on the network devices and it all seemed fine.

Now the set up is quite simple, your standard basic router on a stick set up. With a CISCO 1841 as the router, which as well as working as the router also is set up as one of the network firewalls. With one interface pointing to the internet (not shown) and the other to the internal network.


FIG 1

We can imagen that the DHCP server is sitting in VLAN 200 and the clients that have stopped working are in VLAN 100. So what’s going on?

Well first move was to look at the DHCP logs on the server to see any sign of requests eing received. Nothing there suggesting the packets whegetting stopped before they gotthere.

Check the router config for the “ip-helper” command. This all looked fine and a quick ping from the router to the DHCP server shows that there is not issue with the router forwarding packets to it. Net step ping the Client PC from the router….. OK here’s an issue router can’t ping the Client? But the client can reach the internet through the router? And stranger still the Client CAN ping the router interface of  192.168.10.254??

To bypass any other part of the network, I set up two SVI on vlan 100 and 200 on the switch directly connected to the router and checked the trunk was carrying both. Again the switch could ping both the interface on the router, but the router could only ping the IP address assigned to the SVI for vlan 200?

Well the first step was to work out if the router was indeed sending a packet out, as I mentioned the Router also acts as a fire wall so could a policy update be causing the issue?

Here is the first use for ACL’s in trouble shooting. Debug commands in cisco are very useful as we know, and one I have used often is the “debug ip packet detail”. But before you go typing it in to a router to test, be aware it will have a massive hit on the CPU and you will be over whelmed with information as the detail of every packet crossing your router is displayed to you.

Before you start debugging create an access list that will permit all the traffic you are interested in. In this case I only want to see traffic to and from 192.168.10.254, so logging on the the router create the access list.

ip acccess-list extended 150

permit ip any host 192.168.10.254

permit ip host 192.168.10.254 any

Then you can run the debug command and only view the details about packets covered by this access list.

debug ip packets 150 detail

Enabling this on the Router and again pinging the 192.168.10.250 address and the debug output show the packets sent out on vlan 100, and to be sure enabling the same debug on the switch and I could see the packets both received from the router and being sent back out the same vlan interface. Yet the router logs show no sign of packets getting dropped or even being received. Neither dose this debug show any sign of the packets this is not surprising as debugging IP packets shows packets that are crossing the control plane of the router and if an  ACL or the fire wall are blocking them they will not reach this.

So here is the next use for a ACL in trouble shooting. One of the first steps a packet takes when received on an interface is getting checked by any applied ACL. This is a reasonable step as for security reasons you want to drop any rogue packets ASAP.  So by creating adding the line “permit ip any any” to the end of the above ACL, and the command “log” to the first two line. I then applied this ACL to the interface in the inward direction.

Now repeating the ping to 192.168.10.250 from the router and I see in the logs packets being transmitted and getting received. Now I know that the issue is with in the firewall policy’s on the router.

So yes ACL’s are not only great for security and for managing live data flows across the network. But they are also useful in trouble shooting, especially when used to filter outputs of show and debug commands to  useful information. And using the log function you can capture sporadic issues with out having to be logged on the whole time watching for it.

DevilWAH

PS. There is also the “debug packet” command to capture traffic received on an interface, but I like the simplicity and logging ability of using an ACL.

Quotes

Man I seem to be contently adding my quote collection but still no where near the end! In fact not even 20% of the way through! Main trouble is that I have them all over the place with different formats so have to copy each one by hand! Once there all up I shall be tidying up the quote pages.

On a side note added a few more flash cards to Anki, some QoS stuff.

QoS and leaky Buckets

Just been going through QoS in the foundation guide, it has a small bit on the Leaky bucket algorithm, but I think the wikipedia article explains in much clearer.

I had always though of it as the packets where the water running in to the bucket and there was a small hole in the bottom from which it drained out. As long as the average in was less than the drainage hole, and bursting did not over flow the bucket the water flows out with out spilling.

However I see now that in the case of CISCO switches the leaky bucket is a metering method. The packet it self does not flow in to the bucket. Rather how fast the packets flow in to the switch determines how fast the tap above the bucket it flowing. While the bucket it not full the packets can pass through the switch. But if the bucket should over flow the packets are dropped until enough water has run out of the bucket that they can continue.

Like I said the article on wikipedia explains it all very nicely, so if like me it is taking a bit to get your head around have a look.

DevilWAH