ASA 1: Engineer 0
A recent project found me migrating from a pair of Juniper firewalls to an HA pair of ASA 5540s. I have never really used Junipers before, but the configuration seemed relatively straightforward. I could tell how they were defining their NAT statements, VPN tunnel configs, etc. No major issues that I saw, just a lot of cut/paste operations by hand. There were roughly 130 or so NAT statements to be migrated, which was tedious, but not difficult. There were some questions I had on the site-to-site configs, but nothing major. I had everything configured and ready to go for an early Saturday morning cutover. Wake up, get to the DC by 0500, out by 0900 or so, right?
Cutover morning, we get to the DC, plug in the pair, all interfaces come up, failover tests out ok, the interwebs are reachable, etc. etc. So we exit the DC and go into the customer area for additional testing. So there were a few NAT statements that I had transposed the IPs on, or accidentally pasted the previous IP I had on the clipboard. No problem…easy fix. Site to site tunnels came up mostly, except for a few mismatched parameters. Another easy fix. All in all, with the exception of some little stuff, things were going ok.
Here’s where the fun starts.
The network was one flat layer 2 domain…sort of. Gateway (ASA) was x.y.5.1/19, which would include x.y.0.1 through x.y.31.254. The internal scheme was such that domain Windows servers were x.y.10.z/20, and their virtual servers were all x.y.20.x/24. ALL hosts on the network had 5.1/19 as their gateway. On their previous firewall, the hosts at x.y.10.z were able to communicate with hosts on x.y.20.z without a problem. Can you spot the issue here before I tell you what it is?
If we look at x.y.10.z/20, this includes x.y.0.1 through x.y.15.254. The virtual server “subnet” was x.y.20.z which includes x.y.20.1 trough x.y.20.254. The problem showed up in that hosts on the x.y.10.z subnet could not communicate with the x.y.20.z hosts. Since 10.z/20 only includes up to 15.254, the hosts would send their packets to the default gateway, destined for the off-net 20.0 subnet. The ASA for whatever reason (and this is the part I haven’t had a chance to really research — I welcome all input as to why though), would drop the traffic going to 20.z, even though the same-security-traffic permit intra-interface was enabled. Nothing was working, and I was getting frustrated trying to figure out the answer.
I opened a TAC case and the engineer, within a few minutes, said that we need to enable TCP state bypass. Not something I’ve ever run into before, but ok. I won’t go into the configuration details, because they’re in the link, but for some reason, this worked fine. Based on the Cisco document behind the link, this occurs more frequently with asymmetric routing issues. I have had a chance to really go back and figure out why this was occurring, but the tcp state bypass resolved the issue.
Can anyone out there elaborate on why the ASA would drop that traffic?
EDIT: The subnets listed above are incorrect: The gateway is correct, the 10/.x is /21, not /20, and the 20.x is /21 as well.
Yeah, I’ve hit that before. The issue is that even with same-security on, the TCP state machine is still inspecting the traffic. When you have a case where the session bounces off the ASA interface, but the response goes directly back to the host (and does not symmetrically flow back through the ASA) it breaks the ASA TCP inspection. The ASA sees the SYN, but not the SYN/ACK, so when the original host sends the final ACK for the handshake, the ASA blocks it because it thinks it’s premature. If you did ‘sh conn’ you’d have seen sessions with like the SAab flags indicating the sessions were stuck in handshake limbo.
Same thing happens if you have the ASA and a WAN router on the same segment as the users, and you try to set the ASA as the default gateway. Won’t work, gotta use the router as the gateway. That recently-added TCP state bypass does fix it, but now the ASA isn’t even doing basic TCP protocol inspection on that traffic. I’d recommend using a tight class map to only apply the bypass to intra-site traffic.
The bigger issue, of course, is inconsistent masking on the network where some hosts think they’re L2 adjacent to others, but those others think they’re on a different subnet! Gotta fix that!
Nice writeup on the issue, though!
Just to follow up on this Bob…I was talking this over with another engineer today, and I’m thinking that (along the same lines as what you said), the hosts in question were dual-homed, so I’m thinking that the 10.x host sent the syn packet, the received 20.x (with a 10.x NIC as well) replied w/SYN ACK back to the send, but sent it out the 10.x interface, so the ASA never sees the syn/ack, but sees the final ACK and denies the connection.
Thanks again for your input!