Scenario: 2 Cisco 3750X’s in a stack, connected to a 6509 VSS core via a port-channel with one port from each 3750 (1/0/24 and 2/0/24) in the stack being in the channel (channel-group 13 mode on). I see the 6509 in the cdp neighbor table (on both interfaces), show etherchannel summary shows the port-channel to be up and active…no issues there, native vlans match, all appears well with the world, right? I have vlan 254 on the 6509 with an ip of x.x.x.1/24, and vlan 254 on the 3750 stack is x.x.x.174/24.

I issue a ping from the stack to the core, expecting to see …!! and go about my merry way. Instead, of course I got ….. Hmm. Odd. Double check configs, they look fine. Shut, no shut the port-channel, everything comes up as normal (again). Ping x.x.x.1 — still get no response. Now, I’ll admit to missing the small details sometimes, but after about 30 minutes or so, I was officially stumped.

More info and diagnostics: Switch 2 in the stack used to be a member of another stack. It was removed from the previous stack, erase startup-config was performed, and verified that I started with a blank slate. Connect it to the master switch, it came up fine with no errors. After further troubleshooting, I could narrow it down to a possible issue with switch 2. Basically, if I shut down 2/0/24, the ping succeeded. If 2/0/24 was up, ping failed.

Again, thinking I was missing something, I asked another engineer to take a look and make sure I wasn’t missing anything silly. He doubled checked and all configurations were correct. I tried the same config of 2/0/24 on 1/0/23 and shut down 2/0/24, and the ping succeeded. So we at least know creating a port-channel works, generally speaking.

After an hour or so, I finally opened a TAC case. TAC double checked everything again and could find nothing wrong with the configuration. He thought it may have been a limitation on the number of port-channels you could have on a single line card on the 6509, but there was only one other port-channel connected to it. TAC engineer said he was going to research it and get back to me. Luckily, this was a new stack and wasn’t in production. I told TAC that I was going to erase switch 2 and start over, but we both agreed that probably won’t change things.

Well, we were wrong. As soon as switch 2 rejoined the stack and I added the config back to 2/0/24, the port-channel came up, and the pings finally succeeded. TAC never came back with any updates/bugs and we closed the case, but how annoying is that?

Cisco, if you’re listening, I’d like those 2 hours of my life back, please. I’ve included the relevant snippets of code below in case you’re interested.

Configuration on the 3750 stack:

3750Stack#sh etherc sum
Omitted....
Group Port-channel Protocol Ports
------+-------------+-----------+-----------------------------------------------
13 Po13(SU) - Gi1/0/24(P) Gi2/0/24(P)

3750Stack#show switch
Switch/Stack Mac Address : 503d.e5fa.0f00
H/W Current
Switch# Role Mac Address Priority Version State
----------------------------------------------------------
*1 Master 503d.e5fa.0f00 15 1 Ready
2 Member 503d.e5f9.e900 14 1 Ready


So, this really didn’t have much technical value (I think), but maybe will save someone 2 hours somewhere down the road. Happy packet pushing!

EDIT June 30th, 2011

Customer called saying stack was down again, but since it wasn’t really in production (yet) they hadn’t been using it, so I have no idea how long it had been down. Another 2 hour TAC case, and again, they found nothing wrong with the configuration of course, but couldn’t diagnose the issue.

Replacement switch arriving tomorrow…go figure. I may be jumping on the “I hate 3750 stacks” train soon…

So when I was given a PCF file yesterday, I imported it like I normally do, changed the NAT traversal setting, and I was ready to go. Much to my surprise, I kept getting a “Connection to XXXX vpn failed” message. Weird. I rechecked the VPN settings on the ASA (one that someone else had configured): tunnel group name and password matched, IP address was fine, username and password were correct. Hmph. I tried the connection on an XP laptop, and it connected fine. Double hmph.

While perhaps there may be easier ways to do this, here’s what I did to troubleshoot. If you open a terminal and run sudo vpnc, Ubuntu prompts you for all of the endpoint information, just as if you created it via the GUI.

Enter IPSec gateway address: x.x.x.x
Enter IPSec ID for x.x.x.x:
Enter IPSec secret for @x.x.x.x:
Enter username for x.x.x.x:
Enter password for username@x.x.x.x:

IPSec ID in this case is looking for the tunnel group password. So after entering all of the required info, I see this:

vpnc: peer selected (single) DES as "encryption" method.
This algorithm is considered too weak today
If your vpn concentrator admin still insists on using DES
use the "--enable-1des" option.

Strange, but ok. Apparently the connection was set to use DES instead of AES (which is what vpnc defaults to). To remedy this, I had to modify the encryption method parameter in the vpnc GUI from secure to weak (I’ll spare you the discussion on DES vs. AES). Once I did that, I connected without a problem.

1. Don’t try to FTP an image using anything but the management interface. CoPP takes over and it will take forever to get the image to the device. I tried to upload about a 180Mb image, and it took roughly 40 minutes (using 2 different FTP programs and servers to ensure it wasn’t one or the other)

2. You can’t use promiscuous ports for private vlans on a 2000 series fabric extender. Nope, can’t do it…see here.

3.  To upgrade the system images: (I’m assuming you can get them to bootflash on your own)

install all kickstart bootflash:///KickStartFileName.bin system bootflash:///SystemImageFileName.bin

If you you want to see the whole Cisco doc on this procedure, go here.

Hopefully this may help someone in the future, or at least helpful for me next time I do this!

Cutover morning, we get to the DC, plug in the pair, all interfaces come up, failover tests out ok, the interwebs are reachable, etc. etc. So we exit the DC and go into the customer area for additional testing. So there were a few NAT statements that I had transposed the IPs on, or accidentally pasted the previous IP I had on the clipboard. No problem…easy fix. Site to site tunnels came up mostly, except for a few mismatched parameters. Another easy fix. All in all, with the exception of some little stuff, things were going ok.

Here’s where the fun starts.

The network was one flat layer 2 domain…sort of. Gateway (ASA) was x.y.5.1/19, which would include x.y.0.1 through x.y.31.254. The internal scheme was such that domain Windows servers were x.y.10.z/20, and their virtual servers were all x.y.20.x/24. ALL hosts on the network had 5.1/19 as their gateway. On their previous firewall, the hosts at x.y.10.z were able to communicate with hosts on x.y.20.z without a problem. Can you spot the issue here before I tell you what it is?

If we look at x.y.10.z/20, this includes x.y.0.1 through x.y.15.254. The virtual server “subnet” was x.y.20.z which includes x.y.20.1 trough x.y.20.254. The problem showed up in that hosts on the x.y.10.z subnet could not communicate with the x.y.20.z hosts. Since 10.z/20 only includes up to 15.254, the hosts would send their packets to the default gateway, destined for the off-net 20.0 subnet. The ASA for whatever reason (and this is the part I haven’t had a chance to really research — I welcome all input as to why though), would drop the traffic going to 20.z, even though the same-security-traffic permit intra-interface was enabled. Nothing was working, and I was getting frustrated trying to figure out the answer.

I opened a TAC case and the engineer, within a few minutes, said that we need to enable TCP state bypass. Not something I’ve ever run into before, but ok. I won’t go into the configuration details, because they’re in the link, but for some reason, this worked fine. Based on the Cisco document behind the link, this occurs more frequently with asymmetric routing issues. I have had a chance to really go back and figure out why this was occurring, but the tcp state bypass resolved the issue.

Can anyone out there elaborate on why the ASA would drop that traffic?

EDIT: The subnets listed above are incorrect: The gateway is correct, the 10/.x is /21, not /20, and the 20.x is /21 as well.

That’s what makes this hard…great boss, good company, good coworkers, but it’s just not what I want to do anymore. There’s little true “network” work to be done, outside of setting up new offices one or two times per year. I did however learn a ton while working there since we were always a one or two man shop. Exchange, AD, Storage, VMWare, LAN/WAN design, security, etc. You name it, I’ve most likely done in it in some fashion. This job has been great for my career while I was there, but it’s time to move on.

I will be starting on Monday at a Cisco Gold Partner with a fairly large footprint. I hope to get to work with mostly voice and R&S, but I’m not really sure what to expect at the moment. I know that it will be a great change for my career, but it’s always a little unnerving leaving a safe and stable job for something new. Nothing ventured, nothing gained I suppose.

I’ve merged my two blogs here, as I’ve realized I don’t really enjoy writing out technical posts like others do. There are a ton of great sites out there for purely technical content, so I’ve given up on that. Packet Racket will of course still contain some technical content, but nothing like “Everything you wanted to know about protocol X, but were afraid to ask.” I’ll be keeping track of my new adventures here, as well as writing about some of the technical challenges and projects I am working on.

So here’s to a new chapter for me in life. It’s always tough saying good bye to an old friend, but sometimes you just have to change it up a bit.

One problem I ran into is that the 7942 phones we had were displaying the wrong time. After verifying that all of the time zone and clock settings on the router and the ISM were correct, I was lost. I then ran across this link on Cisco’s website. Simple enough…seems like the firmware could affect the time being displayed on the phone. Weird, but ok.

So I begin on the mission learning how to upgrade the firmware. I like to think that I am relatively competent. I will however, sometimes glance over small details that can make a lot of difference. After finally ending up here, (from here — probably the more important link), I found the firmware files, but couldn’t figure out which load file to use. Therein lies the details…

If you look at the section labeled “Important Information about Configuring Cisco Unified IP Phone Support” – it tells you to “configure only the filenames that are marked with an asterisk (*) in the table below.” HA! Well, that was an important fact…the one that I glanced over probably 5 times and didn’t register for some reason.

Anyhow, the simplified version (which is presumably what you ended up here for) is this:

1. Find your CUCME version at this link
2. Find your phone(s) model (and protocol) in the first table shown (Cisco Unified IP Phone Support) – take note of which file has the asterisk next to it
3. Download the firmware files from this link (CCO Login Required – Click Voice and Unified Communications, then IP Telephony, then IP Phones, and then select your model,  protocol, and which firmware version you want to use.
4. Download the zip file to your computer – in my case it was the cmterm-7942_7962-sccp.8-5-4.zip file
5. Extract the files, and upload them all using copy tftp flash
6. Once you have all files loaded in flash, do this:
   1. (config)#telephony-service
   2. (config-telephony)#load PhoneModelNumber FilenameWithAsterisk Example: (config-telephony)#load 7942 SCCP42.8-5-3S.loads (The documentation says to use the file extension for CUCME version >= 7.0.1, but I also saw something that said don’t. I tried both ways and it seemed to work. YMMV)
7. Once all that is done, reload your IP phone (either power cycle it or reset it through the IOS or CUCME GUI)
8. Ideally, the phone will see the new firmware configuration and proceed with the upgrade

Please note that there may be some caveats/prereq’s that I didn’t include here (I don’t think there are, but I didn’t triple check everything). As always, this worked for me, it may or may not work for you. Hopefully, it will.

I hope this serves as an easy to follow guide to upgrading the firmware on your Cisco IP phone. I’m sure for those of you that have done it many times, its rather simple. But for voice noobs such as myself, it was quite a PITA.

We recently brought up a satellite office on a call manager express set up, which will eventually tie into our main office when we roll it out later this year. We’re also migrating all of our (albeit only 3) remote offices to voice over IP, so I think that will help to keep me in good shape for the CCVP battery of tests.

While the CCIE R&S is generally just an expansion of the CCNP topics, I know very little about voice and related concepts. It will be nice learning completely new technologies, and not just revisiting stuff I already know somewhat, but need to learn in more depth. I’ve already ordered the CVOICE study guide and am working my way through that for now. So far, so good! It’s nice to be genuinely interested in the material again. Hopefully, I can pull this off by the end of the year…a little ambitious, but manageable I think. QoS may have its way with me though…we’ll see.

Mode On = No negotiation takes place

PAgP (Cisco proprietary)
Desirable: send PAgP packets
Auto: listen for PAgP packets
D+D = EtherChannel
D+A = EtherChannel
A+A = No EtherChannel

LACP (Industry standard)
Active: send LACP packets
Passive: listen for LACP packets
A+A = EtherChannel
A+P = EtherChannel
P+P = No EtherChannel

Carry on…nothing else much to see here.

I’ve given myself until my birthday (in a few weeks) to not think about it…and then, it is time! Time to get back in the packet groove. I’ve got my tentative study plan laid out here for others to use if they’d like. Its not final yet, and there are some holes to be filled, but at least I have a defined set of tasks to accomplish.

So, in the words of Eminem: “Let’s get down to business, I don’t got no time to play around, what is this, must be a circus
in town, let’s shut the shit down on these clowns, can I get a witness, {hell yeah}”

Ah hem…sorry ‘ bout that. My social filter malfunctioned.

NOTE: After going back into the INE online classroom, I’ve found that the ATC COD has changed for the v4 exam. The spreadsheet referenced above may contain some of the older ATC COD videos, but I am updating them as I go through them.

The phone rings and its my boss who says there are a lot of people who can’t connect via VPN and RDP. The odd thing was that if you would try again in 20-30 minutes, chances are it would work perfectly. Weird? Yeah, but I didn’t think too much about it at first. I began some basic research into the issue, but couldn’t really come up with anything at the moment.

I got into the office on Thursday, and began to dig a little deeper. I checked out some of the graphs on our NMS, and found that about every hour, for about 25 minutes, something on one of our subnets was maxing out our internet bandwidth (1OMb circuit). It was very weird…high peak, valley, peak, valley, peak, etc. every since Monday night at around 10. If you look at the chart below, you can see that all day on Thursday this was occuring.

At this point, I was a bit stumped. Up until this point, the only thing I could determine was that there were a ton of connections to a bunch of IP addresses in Kanas in the 12.200.0.0/16 range. If you browsed out to one of the IPs you would get what seemed like a 404 error, but it didn’t seem to be your typical 404. An IP whois lookup only revealed that the IPs came back to AT&T Worldnet Services. Hmph…

I decided to do a packet capture on the firewall to try and narrow it down a bit. First thing I notice is a ton of http connections and get requests to the IP range I mentioned before. Weird. At one point, I ran one of the analysis tools in Wireshark (I don’t remember which one unfortunately). Whatever it was, turned up the fact that all of these connections were to nai.com. Interesting!

If you go to nai.com, it redirects to McAffee’s website. I get the proverbial “ah-ha!” moment. I ask my coworker if/when he last rebooted the McAffee policy server. He stated that it was Monday night around 10 or so. Funny…that’s when these funky spikes started happening. I ask if all the clients get their updates from the internal server, and they are supposed to. So here’s what happened…

The McAffee policy server apparently ran out of disk space, and was unable to download the latest updates. It apparently sent a message to the clients to get your updates from the web since the server’s copy is outdated. Turns out that the spikes we were seeing were all of the clients trying to grab their updates at roughly the same time. The 30 minute spike was actually 24 minutes if you looked more closely, which correlated with the policy that said “Try for 24 minutes, every hour.”

If you look at the graph from above,  you can see that sometime around 3PM, we corrected the problem, and all was well. This was one of the challenges that because you don’t see things like it everyday, it takes a bit of time to track down the issue, but it was definitely an interesting and dare I say fun task!