Verizon Fios with Static IPs and Cisco ASA Firewall

Hello everyone. I’m writing this post not because I am looking for help but rather to share the solution to a problem that I have had. I spent 6-7 days trying to make a Cisco ASA Firewall work with an upgraded FIOS connection and didn’t really find the solution anywhere online. So here is my story and how to make a Cisco ASA Firewall work with FIOS. (The TL:DR version is that proxy ARP has to be enabled on the outside interface and on NAT entries). First off some background: In 2010 FIOS was introduced to my neighborhood. We replaced 2 bonded T1’s (almost $1200/month) with a 35/35 FIOS package with 13 static IP addresses for something like $189 month. The ONT we have is the Alcatel O-821-M-E with Ethernet. In 2010 our setup was as follows: Alcatel ONT > Cisco 871w > Private Network. As soon as the FIOS was installed we noticed that All the static IP’s, except for the primary IP on the outside interface of the Cisco 871w ceased to function to inbound traffic after 6 hours. (See my thread from 2010 here: http://www.dslreports.com/forum/remark,24843732). What appeared to be happening is that my firewall would update the Verizon ARP table but after 6 hours the Verizon hardware’s ARP table would clear its arp cache and send the packets to my firewall destined 0.0.0.0 which the Cisco 871w listed ‘invalid’ and dropped. This was apparently a known issue with Verizon Fios and Static IP’s. (See this thread https://supportforums.cisco.com/discussion/10814731/dissapearing-nat-3725-fios-loss-ftp-service-help) The Verizon tech (who was awesome) got in touch with an NT (Network Technician) who made some changes including updating the number of MAC addresses and I believe “rebuilding the ARP table”. Unfortunately I don’t have a copy of the trouble ticket from back then so I can’t find out what the actual solution was. I *believe* what they did was hardcode my routers MAC address into the Verizon ARP table meaning that packets always showed up to the right address. This solution has been working for almost 4 years. Fast forward to last month, where our Cisco 871w started to crap out… The 871w was really only capable for about 18/18 speed wise anyways so we figured that if we needed to buy a new firewall/router that we might as well get a great firewall and upgrade our connection. So we purchased a new Cisco ASA 5512-X firewall and upgraded our connection to 300/65 with 13 static IP’s. The connection went live with the new router 6 days ago. For reference here is the basic layout again Alcatel ONT > Cisco ASA5512-X > Private Network. The problem that occurred back in 2010 immediately started to occur again. The primary IP stays up and running 24/7, but the NAT’d static ip addresses don’t reach the Cisco ASA5512-X. By temporarily changing the primary address on the outside interface of the router to one of the static nat addresses I can force the ASA5512-X to perform a gratuitous ARP update for that static address. This only lasts 6 hours until the entry expires from the Verizon cache, then the packets stop coming in. I opened a trouble ticket with Verizon and got a good tech who called an NT again. The NT told him that they have taken the ability to rebuild the ARP table away from the NT’s for network safety reasons. The NT was able to rebuild the ARP table by what he called “kill and build the cross connects”. He said that would likely fix the problem. I had also read that Verizon had updated their hardware with a fix so that the ARP table updates were no longer required and that they now by default allowed the same of mac addresses as the number of your static IP’s (i.e. if you have 13 static IP’s they allow you 13 MAC addresses). Anyway Verizon’s fix didn’t hold and six hours later I was ready to escalate it further up the chain at Verizon. However before I escalated the issue with Verizon I decided to try a few other items on my own. Here are the items I tried #1 - First I altered the altered the default MAC address on the new Cisco ASA 5512X to spoof that of my old Cisco 871w router. This was done because my thought was that when we got this working in 2010 with the 871w that Verizon had created a static ARP entry on their end (This is speculation, not verified). When I replaced the 871w with the ASA 5512-X the ASA the static route was no longer valid. My theory was that the ASA 5512-X would GARP the new info when forced, but after 6 hours it reverted to the original static route. #2 - Second I enabled Proxy ARP on my firewall. I found one online reference to the Proxy ARP settings on the Cisco Firewall. Apparently when proxy ARP is on, the firewall will gratuitously ARP for secondary addresses (Proxy ARP basically uses the same one MAC address for the primary and NAT’d IP’s). This may be important because unlike traditional cisco routers, you cannot enter ‘secondary’ IP addresses on the outside interface. Instead you have a primary outside interface address, and the secondary addresses are listed in NAT. If proxy ARP is disabled then the firewall is probably not responding to the Verizon ARP packets properly. There are two places on the ASA that Proxy ARP has to be enabled: First on the interface via a weird double negative command “no sysopt noproxyarp ” and secondly in the nat entries by clearing the checkbox “Disable Proxy ARP on egress interface”. Well Success! I now have a Cisco ASA 5512-X that fully works with FIOS! The router/firewall properly GARP’s and connection stays up 24/7. The bandwidth through the firewall is flawless. Speed tests indicate 300/70. So now I’ve gone back to determine which of the two items actually fixed the problem. Yesterday I cleared the spoofed MAC address field but left the Proxy ARP on. After forcing a GARP update with Verizon (by quickly and briefly changing the outside interface address to each of the NAT’d static IP’s then back to its default) everything works and has continued to do so for 15+ hourss. By process of elimination this means the issue was #2 above. Ultimately what this translates to is that in order for a Cisco ASA firewall to work with FIOS, Proxy ARP must be enabled on both the Outside interface of the router and on the advanced tab of the NAT entries. Svirfnebli