Cisco Catalyst 2950 HTTP: out of lines

This week I have being working on a lab setup to test some new features on an ACS deployment for a customer. For the lab, I took a couple of Catalyst 2950s/3550s and did some AAA configuration on them, so that pretty much everything AAA related went to the ACS servers.

After everything was done, I was doing verification on different things and everything was working as it should and then it was HTTP turn on the 2950s, the configuration was there but nothing was happening, all I got was a blank page every time I tried to enter the Web GUI.

Doing a debug ip http transactions showed:

000605: Sep 30 09:58:32.736 CEST: HTTP: out of lines

000606: Sep 30 09:58:32.748 CEST: HTTP: out of lines

000607: Sep 30 09:58:32.768 CEST: HTTP: out of lines

000608: Sep 30 09:58:32.780 CEST: HTTP: out of lines

My first thought was check to see if indeed the switch was out of lines, but that wasn't the case:

SW1#sh users all
    Line       User       Host(s)              Idle       Location
   0 con 0                                     00:00:00
*  1 vty 0     lab        idle                 00:00:00 1.1.1.1

   2 vty 1                                     00:00:00
   3 vty 2                                     00:00:00
   4 vty 3                                     00:00:00
   5 vty 4                                     00:00:00
   6 vty 5                                     00:00:00
   7 vty 6                                     00:00:00
   8 vty 7                                     00:00:00
   9 vty 8                                     00:00:00
  10 vty 9                                     00:00:00
  11 vty 10                                    00:00:00
  12 vty 11                                    00:00:00
  13 vty 12                                    00:00:00
  14 vty 13                                    00:00:00
  15 vty 14                                    00:00:00
  16 vty 15                                    00:00:00

Some searching later I found that apparently on the 2950s (Version 12.1(22)EA13) if you have transport input ssh on the vty lines, HTTP doesn't work, the issue was solved changing the vty  lines configuration to transport input ssh telnet.

I wasn't able to reproduce the same behavior on the 3550s (Version 12.2(53)SE).

Cisco Secure ACS password recovery

So today I got an ACS backup from a client to test some features on a lab setup and since I had never done it, instead of asking for the GUI password, I decided to do password recovery.

After some googling, I found a couple of ways, both on the official Cisco FAQ:

1. Using allowAutoLocalLogin to enable that if you are opening the GUI from the server, it doesn't ask for a password.

2. Removing existing entries in:

HKEY_LOCAL_MACHINE\SOFTWARE\Cisco\CiscoAAA##\CSAdmin\Administrators

Unfortunately, neither of those methods seemed to work on my version of ACS (4.2(1) Build 15 Patch 2), the keys were not in the Windows registry, so after more searching I got the fabulous news that:

"On ACS 4.0 or later, reinstallation is the only way"

So, I'm supposed to reinstall? Come on, this makes no sense at all, specially considering that after reinstalling, the next logical step would be to load the last good backup, which will use the same administrator password that you don't know. Not only you are supposed to reinstall but you are also supposed to reconfigure the ACS again by hand?.

Bad design, just bad.

I guess I'm going to take extra extra extra care of my production ACS passwords now ;)

Uber CCIE R&S bootcamp Tour de force: Narbik, Scott and Eman

From Routing-Bits:

"Today was a real busy day for me (Eman).  I drove to visit with Narbik at one of his CCIE Boot Camps he was teaching in Herndon, VA.  Since I was driving so far to see him I asked our mutual friend Scott Morris to join us for lunch.  I am amazed at how the fifteen students in Narbik’s class responded to Scott dropping by to say hello.  He is still held in awe by many neophyte CCIEs and some well established CCIEs around the world.  Both Scott and Narbik have paid their dues as CCIE trainers and mentors.  Both have made their mark on the CCIE community by giving freely of their time to motivate, mentor and improve the arena as a whole.  For me sitting with the two of them was a treat because I have known them both for so long and only at the CCIE party this past Cisco Live have I had the pleasure of their company together.

Where?

Bangalore is our first stop for the dynamic duo.  January 2011 Narbik and Scott will hold the first of these historic twelve day sessions.  This will be followed by Sydney, Australia in April, Milton Keynes, UK in July and finally Wilmington, Delaware in October.  These twelve day comprehensive classes will equip aspiring CCIEs for success.

How Much?

You pay only, $4,500 for both Narbik and Scott, for twelve days of learning unavailable from any other source, anywhere in the world.  There may be other twelve day or longer classes offered but none have these two dynamos taking the lead.  You might pay more and you might now find lower priced venues, but the CCIE Flyer has both Narbik and Scott.

More!

You want more?  Well how about the chance to pay nothing for the second attempt at the lab if you fail the first attempt after taking this class?  Yup, both are Cisco 360 Learning trainers.   So you get the added insurance of knowing you are joined on your journey by Cisco Systems approved trainers.  The two weeks of training will also have a few surprises after class activities and career networking opportunities.

COME JOIN EMAN, NARBIK AND SCOTT FOR TWELVE DAYS OF LIFE ALTERING CAREER ENHANCING TRAINING!"

       For details email to eman@ccieflyer.com

More info

CCIE R&S::Review Sheets / Guides / Notes

There are some interesting review sheet / guides / notes out there:

CCIE R&S Short Notes | Demo Switching Chapter
CCIE Routing and Swithing Quick Review Kit
Packet Life Cheat Sheets

CCIE FREE vLecture sessions

From IPexpert:

"All our vLecture sessions are recorded and available for those who have missed our FREE vLecture and for participants who want to review the vLectures sessions again. We have saved the session recordings for you. Watch our world renowned CCIE instructors explaining specific technical topic in our technology-focused classes and capture the technical knowledge needed to increase your chances of passing CCIE exam."

More

FREE CCIE Lab training from IPexpert

From IPexpert:

"IPexpert’s vLectures are pre-scheduled (and free!) online technology-focused lectures that last between 2 to up to 4 hours in length. Join one of our industry-recognized instructors, in our online classroom, and listen to them discuss and configure various topics seen on the CCIE Lab exam. Our vLectures are FREE to existing clients and will also be recorded and available in our client’s Member’s Area so they can be watched at a later date"

Schedule R&S
Schedule Security
More info

WCCP

A summary on WCCP:

WCCPv1

• Only a single router services a cluster of systems
• Supports HTTP (TCP port 80) traffic flows only
• Provides generic routing encapsulation (GRE) to prevent packet modification
• Routers and cache engines communicate to each other via a control channel based on UDP port 2048

WCCPv2

• Allows for use across up to 32 routers (WCCP servers)
• Supports up to 32 engines/accelerators (WCCP clients)
• Supports any IP protocol including any TCP or UDP
• Supports up to 256 service groups (0-255)
• Adds MD5 shared secret security

Configuration:

Enable WCCP:

R1(config)#ip wccp web-cache

Optional things for the global command

Redirect-list: what traffic gets redirected to the web-cache (eg: send only requests from 192.168.0.1)
Group-list: which cache server gets used
Password: credentials for auth cache.

If the web-cache is in the same interface that requests are coming in:

R1(config-if)#ip route-cache same-interface

Outside int where requests go (Internet facing connection):

R1(config-if)#ip wccp web-cache redirect out

Inside int where requests come in (LAN connection):
R1(config-if)#ip wccp web-cache redirect in

Verification:

R1#show ip wccp web-cache detail

Wikipedia WCCP
Cisco Docs

TCAMINTERRUPT

The name of the blog comes from an odd error I got on a customer 4510R-E switch running 12.2(50)SG1, the log showed:

%C4K_SWITCHINGENGINEMAN-4-TCAMINTERRUPT: flCam0 aPErr interrupt. errAddr: 0x2464 dPErr: 1 mPErr: 0 valid: 1

%C4K_SWITCHINGENGINEMAN-4-TCAMINTERRUPT: flCam0 aPErr interrupt. errAddr: 0x2464 dPErr: 1 mPErr: 0 valid: 1

%C4K_SWITCHINGENGINEMAN-4-TCAMINTERRUPT: flCam0 aPErr interrupt. errAddr: 0x2464 dPErr: 1 mPErr: 0 valid: 1

%C4K_SWITCHINGENGINEMAN-4-TCAMINTERRUPT: flCam0 aPErr interrupt. errAddr: 0x2976 dPErr: 1 mPErr: 0 valid: 1

The device was not affected, even though according to the docs, this is supposed to mean "A parity error in a TCAM entry was detected. Contents of the log register are printed out Software will automatically perform error recovery on the defective TCAM entry"

It the end it was the result of the following IOS bug:

CSCsv17545 %C4K_SWITCHINGENGINEMAN-4-TCAMINTERRUPT: flCam0 aPErr interrupt
Externally found severe (Sev2) bug: R-Resolved

Symptoms
Under normal operation we see the following messages appearing frequently in the logs:

%C4K_SWITCHINGENGINEMAN-4-TCAMINTERRUPT: flCam0 aPErr interrupt. errAddr: 0x2947 dPErr: 1 mPErr: 0 valid: 1

%C4K_SWITCHINGENGINEMAN-4-TCAMINTERRUPT: flCam0 aPErr interrupt. errAddr: 0x2B59 dPErr: 1 mPErr: 0 valid: 1

Conditions
The issue appears to happen under normal operation, so far this has been observed after around two weeks of uptime, but needs to be confirmed.

Workaround
None At Present.

Solution
Upgrade software to IOS version 12.2(52)SG or later OR 12.2(50)SG4 or later.

The rules of multicast

Great post here

Frame Relay - Broadcast Queue

I stumbled with this today doing INE WB Volume I, so googling I found:

The Cisco IOS creates a broadcast queue for interfaces running frame-relay. This queue performs 2 functions: it ensures routing traffic is dealt with as a priority but it also limits the bandwidth that can be consumed by such traffic.

When an interface has many DLCIs the overhead of replicating routing traffic can be significant.

The interface has the following default settings

size: 64 packets
byte-rate: 256000 bytes per second
packet-rate: 36 packets per second

To change the settings the following interface command can be used

frame-relay broadcast-queue {x} {y} {z}

Source 
Cisco Docs

Things to remember after a 12 Day CCIE R&S bootcamp

Things to remember after 12 days of bootcamp:

• Don't assume anything.
• Read exactly what the question asks of you, even if it means doing it word by word.
• Read the entire lab, twice if possible, and try to setup dependencies (eg: trunks->vtp->vlans).
  
• The diagrams are always correct (eg: if it says Vlan20, it means that the IP address should be on a SVI and not the Physical interface)
• Cut & Paste when possible.
• Do wr often.
• Time management is critical, set a time-frame for troubleshooting and enforce it. Getting stuck 20/30 minutes on a single 2 points task is not the way to go.
• Make sure you are doing the configuration on the correct device, no points for applying perfect QoS on the wrong router.
• Interface level dot1x commands don't appear if the interfaces are in dynamic mode.
• If you are told "there is a syslog server at 10.10.10.10", be sure to check you have a route to get to 10.10.10.10
• Clock rate is your friend.
  
• Check for 0.0.0.0 mappings in frame-relay. I lost points on PIM related tasks for leaving them on the router (great explanations here).
• Check default commands disabled such as: no ip classless, no ip subnet-zero, no ip cef, no service prompt config (more). Easy way of checking most of them: sh run | i no
  
• Look out for kron tasks and EEM doing nasty things like changing the enable password or killing a routing process.
• "When doing redistribution from BGP prefixes into OSPF, you should make sure that OSPF ASBR Router-id matches originating BGP router ID."
• Use TCL/Macros on SWs for reachability tests (more info). Example: 
• macro name PING
  do ping 1.1.1.1
  do ping 2.2.2.2
  @
  
  You can then use it on the switch:
  
  Switch(config}#macro global apply PING
• Verify, verify, verify and after that verify again.
  

Guide to CCIE Lab Locations: Brussels

Great video from IPExpert Short walk around CCIE Lab location in Brussels, Belgium.

CCIE R&S Doc CD URL

http://www.cisco.com/cisco/web/psa/default.html