Setup a pfSense 2.0 firewall when default gateway is on a different subnet

I have written a better article, using the firewall in transparent mode here .

This article has been updated for pfSense 2.0. The original article about pfSense 1.2.X has moved here.

News: It looks like the OVH gateway works like an universal ARP proxy or Captive portal. I means my OVH gateway replies to any ARP request. This means that for any given a.b.c.d/32 failover IP, I can setup a host or virtual host using a mask /24 (instead of /32) and my gateway a.b.c.X where X can be anything not in ( 0, 255 or d ) and it will works. And It works, at least on the Kimsufi I have tested it. And it works even for IPs in a.b.c.0/24. I think this is how OVH setup some (maybe all) of their routers to be able to support migration of failover IP or block without too much headache. What is fun is to traceroute some IPs in a.b.c.* and see how they are not directly attached to the WAN but are behind some routers.
OVH don't say anything about this setup and then this feature is funny but cannot be used on a production server.
Be careful if you use a wrong setup and generate a lot of unexpected ARP requests, OVH can warn you to quickly fix the problem or even disable your network link.

If you buy a VMware server and an IP block from OVH you will be surprised because the default gateway don't match the IP block. Even if this setup is unusual, it is valid and give full satisfaction if you know how to configure your firewall and hosts.

There are some advantages to use this technique for the provider/WEB hoster: this make the router configuration a lot simpler (no need to setup an IP address for each underlying IP block, they can merge routes for adjacent IP blocks together) and the most important, this save one IP address in the block.

Windows host accepts this unusual configuration and just work, thanks Bill for this great job .

pf-shellcmd

Linux host requires a little trick.

[root@fc6-pmx ~]# route add default gw 192.168.23.254
SIOCADDRT: Network is unreachable

Linux refuses to add the route because it don't know how to reach the gateway itself. Add the appropriate route for the gateway, before the default route, solves the problem.

[root@fc6-pmx ~]# route add -host 192.168.23.254 dev eth0
[root@fc6-pmx ~]# route add default gw 192.168.23.254

This works !

[root@fc6-pmx ~]# route -n
Kernel IP routing table
Destination     Gateway         Genmask         Flags Metric Ref    Use Iface
192.168.23.254  0.0.0.0         255.255.255.255 UH    0      0        0 eth0
10.0.0.0        0.0.0.0         255.255.255.0   U     0      0        0 eth0
0.0.0.0         192.168.23.254  0.0.0.0         UG    0      0        0 eth0

To configure a firewall, depending of the firewall, you will have to be imaginative !

Differences with the 1.2.X config

The main ideas are the same as in the previous article , but the trick used to connect the gateway is different and finally a lot simpler. I recommend this setup !

Schema

Here is the schema I used to test this configuration.

Network schema

All IPs are from the Address Allocation for Private Internets but it is for testing ! Use the addresses you get from your provider or WEB hoster. For real, only the 172.22.22.1 and 19.168.1.0/24 can be in a private network.

  • 192.168.23.254 is the default gateway given by my provider.
  • 10.0.0.0/24 is the IP block given by my provider. I assign it to my DMZ.
  • 192.168.1.0/24 is a for a virtual LAN, to put machines that help to configure and manage the FW and servers in the DMZ.

The gateway trick

Instead of creating an ARP entry using a command line at startup, I force a route to the gateway by using the route command twice. The trick is the identical to the one used for the Linux in the previous article. It is not possible to create such routes using the Web interface then once more the shellcmd module come to rescue to setup the route at startup.

To create a route up to 192.168.23.254, on an interface having no IP in this range, I use the commands:

route add -net 192.168.23.254/32 -iface em0
route add default 192.168.23.254  

The first line tell the firewall that IP address 192.168.23.254 is on the side of the em0 interface (em0 is my WAN interface), the second one use this address as the default gateway.

This time, their is no need to found the MAC address of the gateway like in the first article. But some operations like: disable the em0 interface or setup a default gateway; can break the trick and would require to reload the route manually or reboot the firewall.

To remove the route you can use:

route del default 192.168.23.254  
route del -net 192.168.23.254/32 -iface em0

You can create the default route as soon has you have access to the firewall, using ssh, the console or by using the Command prompt in the Diagnostics menu of the Web interface. To be sure the routes are there, click the Routes option in the Diagnostic menu. An look for the two routes.

Routes

Be careful You have to remove any default route before to run these two commands !

This will not give you access to the Internet forthwith, you need some more settings.

The WAN interface

I don't want to waste an IP address, I choose a completely unrelated address 172.22.22.1, and don't setup any gateway because the job is already done by the 2 commands above.

WAN config

The DMZ interface (OPT1)

This is where the servers having a public address live. I give the 10.0.0.1 address to the firewall, this will be the default gateway for servers in the DMZ but also the public IP of the firewall on the WAN side.

DMZ config

The LAN interface

The LAN can be used if you need additional hosts that don't need to be reachable from the Internet but are required to manage the DMZ or for any other purpose. These hosts can access the DMZ (and vice versa when required). This is where I put a virtual machine to configure this firewall. Machines in this zone can be accessed from the Internet too, see later.

LAN config

I keep the default settings of the firewall for this interface.

Setup the Proxy ARP

The 10.0.0.0/24 subnet is on the DMZ side. To allow the firewall to reply to ARP requests for these addresses on the WAN interfaces, we have to add a proxy ARP entry.

Proxy ARP

I do it for the full subnet at once, in previous article I did it address by address. This is faster but also bypass a bug or a feature in 2.0 that forbid the use of an address already used by an interface. I'm thinking here about DMZ address 10.0.0.1. It is possible to go around this by creating the Proxy ARP before to assign the address to the DMZ interface. But using a subnet here bypass the problem !

Masquerade the source address

For now, packets leaving the firewall have address 172.22.22.1, replies will never come back ! We need to rewrite the source address for packet leaving the firewall. I use hide NAT to give them the 10.0.0.1 address. I assign this address for packet coming from the firewall, but also to masquerade the LAN zone.

Outbound NAT 1

Here are the detail for the LAN, the config for the firewall is similar.

Outbound NAT 2

Now any packets from the firewall or hosts from the LAN will leave the firewall with address 10.0.0.1

Double check the rules for the LAN, and be sure the "Default allow LAN to any rule" permit outgoing connections :

Rules LAN

Don't hesitate to be more strict, for example my second rule block port 25 to the Internet, but not to the DMZ. Here I allow all protocols except some, but the good way when configuring a firewall is to block all traffic by default and permit only some protocols.

The gateway: trick part 2

Now the firewall and the LAN have Internet access, at least after you have setup your DNS. You can now hardcode the gateway trick. You need to install the shellcmd package. The version 0.5 is for pfSense 1.0 but works well with 2.0 too. Install it from the package manager in the System menu!

Package Manager

And in the Service menu, select the Shellcmd option and setup the two commands :

Shellcmd

The DMZ zone (OPT1)

To use your DMZ you have to add filter rules to allow packets to leave the DMZ to the WAN side. Here for outgoing packets...

DMZ outgoing rule

Here I block packets to the LAN, because the DMZ is no more than a part of the Internet itself, any access to the LAN from the DMZ or the Internet must be carefully thought through.

.. and here incoming packets to my public WEB server 10.0.0.2 (the first rule)!

WAN Incoming rule

Create other rules for your other servers and services inside your DMZ !

Because we are using routing we don't need any NAT rules between WAN and DMZ !

LAN has already a full access to the DMZ because of the rule "Default allow LAN to any rule" seen previously. !

The LAN zone

If you need to access some resources inside your LAN from Internet, you can NAT some ports from address 10.0.0.1. Here I forward RDP to my 192.168.1.100 Windows host :

LAN NAT

Double check, pfSense has created the appropriate filter rules.

WAN incoming rules

That'it !

The final touch

Their is lot of other thing to say and to do, but this is not a tutorial about firewall. Anyway I was very impatient to try the new Floating tab in the Rules screen ! I have added a rule to let DMZ hosts reply to ping request. Here it is:

Allows ICMP echo request

Before the Floating tab, you add to duplicate some rules in each interface tab. This was making pfSense 1.2.X a bit unsuitable for configuration with lot of interfaces and rules !

Add IP fail-over

If you need to manage IP fail-over inside this configuration, take a look at this post

Advantages of this configuration

The biggest advantage of this configuration is the use of routing instead of NAT to forward packets. The other are:

  • this config provide a zone for your hosts in your DMZ and your LAN with usual network settings (a gateway in the same LAN subnet).
  • this config is based on routing instead of NAT, this avoid problems with NAT sensible protocol like: ftp, pptp, ...
  • NAT drops connection if no packets are going through for too long. Routing don't and don't require any keep alive plaster!
  • the hosts in your DMZ use the public IP addresses, this make things simple and avoid confusion.
  • LAN access your DMZ using public IP addresses.
  • no need to define NAT rules, only the filter rules are required.
  • reduce the MEMORY and CPU usage of the firewall.

Hope this help !

Comments

Thanks a lot for this great article
Corrine Dipas
corrine.dipas@mail.com

Sorry the document is good and it makes sense in some areas, however i dont own an IP block instead i have a Kimsufi server and only allowed to have 4 Public IPS. One is for the ESXI box. I would want to do it, is have 3 WANS with the MACS assigned to me and a LAN for the servers to get out on. Your method for 1.2.3 about doing an arp- s for the real gateway of the server worked, but now in this version it doesnt seem to work. Can you help me out? Not sure what i am doing wrong.

aspineux's picture

The purpose of this article is to make the network setup of the VM hosts in the DMZ easy, with a gateway on the same network like usual. This is at the cost of a very difficult setup of the firewall.

You are speaking about 4 public IPs for your OVH Kimsufi. These are probably failover IPs, and these have a different purpose ! IP block allows to have multiple host on the same sub-network. failover IPs allows to assign one IP to one of your existing host and later to assign it to another (for example if the first become unavailable). These IPs are not in any subnet and then, not on the same subnet, then you cannot put these in the same DMZ. You even don't have such DMZ !

Anyway I understand your urge to assign these IPs to your VM.
What you could try:
- follows this article, but
- you don't have any subnet then don't create a DMZ and don't setup your OPT1 interface.
- in Setup the Proxy ARP use 10.0.0.1/32 instead (my 10.0.0.1 is one of your public IP)
- put all your VMs in your LAN and assign them addresses in 192.168.1.0/24
- use article about fail-over IPs (link at the end of the article) to map your other public IPs to VMs in your LAN (use 1on1).

I have not tested this setup ...
If it works, please report to help others.

Thank you for your reply. However i dont think this method is right. Correct me if im wrong, but i believe the ip addresses they assign are per server, you can only have a maxium of 4 and they are all authenicate on the OVH network if the system sees the correct MAC address coming from the virtual NIC of the VM ie pfsense WAN interface. Your menthod last time arp - s 172.22.22.254 00:00:00 (real MAC of the phsical gateway) worked really well with version pfsense 1.2.3 now it doesnt seem to work at all with 2.0. And if i wanted multi WANs or IPs to work, i had to create another Opt1 network called "wan2" and give it the MAC OVH give me to do the same method, and then use a NAT functionality to port forward traffic from different IPs. Any further advice? Sorry to be a pain, just i feel at a stopping point, cant quite work this one out.

aspineux's picture

OVH provides new services and new features all the time. I have written an article that matched my needs to use OVH features and the lack of OVH documentation at one moment.

If you are speaking about MAC addresses provided by OVH then you have bridge in mind and this is not the subject of this article, even if some idea can be useful for other use.

It look like you have to take a leaf on another article/sample for your need.
Regards.

aspineux's picture

Lot of thing have changed, I'm writing a new document right now. It is still under construction at
http://magiksys.blogspot.be/2012/12/pfsense-bridge-gateway-vmware-ovh-ip...
Try it, and comment it.
Regards

I just wanted to say i was wrong in what i put, this document is very, very useful! The problem i had was with Kimsufi/OVH. They hadnt activated my Failover IPs so the method i followed in this document wasnt working. But it is now when they fixed the problem. I didnt purchase a block of IPs because i didnt need them, i just added 3 more failover IPs to my server which is ample and is the max your allowed. I followed this document and changed it slightly to accommodate the IPs i appied for. I am happy to write a document for people to follow on this forum if need be if people are confused. But let me know.

aspineux's picture

Thanks for the info

Hello, thank you for this article..

I have a problem, when restart pfsense the routes tables resets and I have to re-insert the commands

route add -net 192.168.23.254/32 -iface em0
route add default 192.168.23.254

how to solve?

aspineux's picture

Use the shellcmd package.
You can found all the details in the article.

Thank you for this guide, I took some part of it.

I used "route add default -iface em0" to get the same result...

Regards....

I have 3 virtual switches configured in ESXi WAN, DMZ, LAN. Pfsense has interfaces connecting to each one accordingly.
For the WAN interface I assigned a random IP like you did and added the gateways with shellcmd, I also added a virtual mac address which is setup on my ovh panel for .225(the IP of my DMZ interface)
For the DMZ interface I assigned 10.0.0.225/28 and setup the proxyarp on 10.0.0.224/28
LAN is default and using 192.168.1.1
I configured the NAT outbound rules as instructed above however I do not have any connection from the firewall or lan clients, any ideas?

aspineux's picture

I would check that the route added with shellcmd is still there. Reconfiguring the WAN interface kill the route and you have to restart the router or add it by hand. Did you make the right conversion when converting my 10.0.0.* addresses to you IP range ? Reread all the article one more time.

I cannot help you more.

As you could guess reading my article I made the setup using try and error. Of course I have a good knowledge in this stuff, and always find one solution for any problem. But I used the right tools for the diagnostic :
I would create a linux VM and connect 3 interfaces to the 3 virtual switches with unrelated addresses (ore carefully choose one, at least for LAN you can always assign a "working" address) and start a tcpdump on each interface to see packets going through the interfaces. Last I would try to follow request and reply packed between the interfaces, check that IPs are always the one expected when reaching/leaving an interface and adapt the firewall setup to reach the expected result.

Good luck

Thanks, this is very hepful. Previously I received a complaint from OVH that I was sending ARP requests over the WAN as I had my WAN interface configured as x.x.x.x/24 so I could reach the gateway.

Using the guide I configured the WAn interface to be a /32 and used your routing and shellcmd trick to tell pfsense where the default gateway actually was, worked a charm...

I now have a slightly different issue, I've assigned a second failover IP to the mac address of the WAN interface which I did through the OVH portal. I then added this second IP to the pfsense WAN interface as a virtual IP (type: IF Alias), again as a /32. Adding it combined with a NAT rule and it works, I'm now accepting connections and making connections on the secondary IP.

A couple of hours later, another email from OVH complaining that they are seeing ARP requests every 20 minutes hitting the default gateway.

Any ideas on how to prevent this as I believe they will block all outbound connections on the IP until its resolved.

Thanks in advance,

Charlie

aspineux's picture

Look at all routes related to your wan interface, you must have only routes attached to IP having a mask equal to /32 or the default route.
But try to use tcpdump to know witch ARP is sent by your FW, using something like this :

tcpdump -n -i em0 -p arp

Also look at all r
It must be relate to one IP attached (one that is not /32) at your WAN interface or to a route

Add new comment