The Ghost in the Stack

Hot off the “WTF” presses…for your reading pleasure.

Scenario: 2 Cisco 3750X’s in a stack, connected to a 6509 VSS core via a port-channel with one port from each 3750 (1/0/24 and 2/0/24) in the stack being in the channel (channel-group 13 mode on). I see the 6509 in the cdp neighbor table (on both interfaces), show etherchannel summary shows the port-channel to be up and active…no issues there, native vlans match, all appears well with the world, right? I have vlan 254 on the 6509 with an ip of x.x.x.1/24, and vlan 254 on the 3750 stack is x.x.x.174/24.

I issue a ping from the stack to the core, expecting to see …!! and go about my merry way. Instead, of course I got ….. Hmm. Odd. Double check configs, they look fine. Shut, no shut the port-channel, everything comes up as normal (again). Ping x.x.x.1 — still get no response. Now, I’ll admit to missing the small details sometimes, but after about 30 minutes or so, I was officially stumped.

More info and diagnostics: Switch 2 in the stack used to be a member of another stack. It was removed from the previous stack, erase startup-config was performed, and verified that I started with a blank slate. Connect it to the master switch, it came up fine with no errors. After further troubleshooting, I could narrow it down to a possible issue with switch 2. Basically, if I shut down 2/0/24, the ping succeeded. If 2/0/24 was up, ping failed.

Again, thinking I was missing something, I asked another engineer to take a look and make sure I wasn’t missing anything silly. He doubled checked and all configurations were correct. I tried the same config of 2/0/24 on 1/0/23 and shut down 2/0/24, and the ping succeeded. So we at least know creating a port-channel works, generally speaking.

After an hour or so, I finally opened a TAC case. TAC double checked everything again and could find nothing wrong with the configuration. He thought it may have been a limitation on the number of port-channels you could have on a single line card on the 6509, but there was only one other port-channel connected to it. TAC engineer said he was going to research it and get back to me. Luckily, this was a new stack and wasn’t in production. I told TAC that I was going to erase switch 2 and start over, but we both agreed that probably won’t change things.

Well, we were wrong. As soon as switch 2 rejoined the stack and I added the config back to 2/0/24, the port-channel came up, and the pings finally succeeded. TAC never came back with any updates/bugs and we closed the case, but how annoying is that?

Cisco, if you’re listening, I’d like those 2 hours of my life back, please. I’ve included the relevant snippets of code below in case you’re interested.

Configuration on the 3750 stack:

3750Stack#sh etherc sum
Omitted....
Group Port-channel Protocol Ports
------+-------------+-----------+-----------------------------------------------
13 Po13(SU) - Gi1/0/24(P) Gi2/0/24(P)

3750Stack#show int p13
interface Port-channel13
switchport trunk encapsulation dot1q
switchport trunk native vlan 254
switchport trunk allowed vlan 104,254
switchport mode trunk
end

3750Stack#sh run int g1/0/24
interface GigabitEthernet1/0/24
switchport trunk encapsulation dot1q
switchport trunk native vlan 254
switchport trunk allowed vlan 104,254
switchport mode trunk
channel-group 13 mode on
end

3750Stack#sh run int g2/0/24
interface GigabitEthernet2/0/24
switchport trunk encapsulation dot1q
switchport trunk native vlan 254
switchport trunk allowed vlan 104,254
switchport mode trunk
channel-group 13 mode on
end

3750Stack#show cdp nei
6509-VSS
Gig 2/0/24 127 R S I WS-C6509- Gig 2/3/24
6509-VSS
Gig 1/0/24 149 R S I WS-C6509- Gig 1/3/24

(I didn't include the 6509 config as it's basically identical of course)

3750Stack#show switch
Switch/Stack Mac Address : 503d.e5fa.0f00
H/W Current
Switch# Role Mac Address Priority Version State
----------------------------------------------------------
*1 Master 503d.e5fa.0f00 15 1 Ready
2 Member 503d.e5f9.e900 14 1 Ready


So, this really didn’t have much technical value (I think), but maybe will save someone 2 hours somewhere down the road. Happy packet pushing!

EDIT June 30th, 2011

Customer called saying stack was down again, but since it wasn’t really in production (yet) they hadn’t been using it, so I have no idea how long it had been down. Another 2 hour TAC case, and again, they found nothing wrong with the configuration of course, but couldn’t diagnose the issue.

Replacement switch arriving tomorrow…go figure. I may be jumping on the “I hate 3750 stacks” train soon…

Ubuntu VPNC – Cannot Connect to Cisco ASA

I’m running Ubuntu 10.04 on my laptop, and use Ubuntu’s vpn client frequently. I usually have no problems importing PCF files and connecting successfully, with the exception of having to change the NAT Traversal parameter from Cisco UDP to NAT-T. It seems that for some reason, it defaults to UDP, even though it shows NAT-T as being default. Eh…if that’s the only thing I have to change, I can live with that. I’ve been using Ubuntu and vpnc for about 6 months now, and have been really happy with it.

So when I was given a PCF file yesterday, I imported it like I normally do, changed the NAT traversal setting, and I was ready to go. Much to my surprise, I kept getting a “Connection to XXXX vpn failed” message. Weird. I rechecked the VPN settings on the ASA (one that someone else had configured): tunnel group name and password matched, IP address was fine, username and password were correct. Hmph. I tried the connection on an XP laptop, and it connected fine. Double hmph.

While perhaps there may be easier ways to do this, here’s what I did to troubleshoot. If you open a terminal and run sudo vpnc, Ubuntu prompts you for all of the endpoint information, just as if you created it via the GUI.

Enter IPSec gateway address: x.x.x.x
Enter IPSec ID for x.x.x.x:
Enter IPSec secret for @x.x.x.x:
Enter username for x.x.x.x:
Enter password for username@x.x.x.x:

IPSec ID in this case is looking for the tunnel group password. So after entering all of the required info, I see this:

vpnc: peer selected (single) DES as "encryption" method.
This algorithm is considered too weak today
If your vpn concentrator admin still insists on using DES
use the "--enable-1des" option.

Strange, but ok. Apparently the connection was set to use DES instead of AES (which is what vpnc defaults to). To remedy this, I had to modify the encryption method parameter in the vpnc GUI from secure to weak (I’ll spare you the discussion on DES vs. AES). Once I did that, I connected without a problem.