Bradley M. Kuhn

• Contact
• Blog  
• Interviews / Articles  
• Software
• Résumé

More Xen Tricks

Friday 24 August 2007 by Bradley M. Kuhn

In my previous post about Xen, I talked about how easy Xen is to configure and set up, particularly on Ubuntu and Debian. I'm still grateful that Xen remains easy; however, I've lately had a few Xen-related challenges that needed attention. In particular, I've needed to create some surprisingly messy solutions when using vif-route to route multiple IP numbers on the same network through the dom0 to a domU.

I tend to use vif-route rather than vif-bridge, as I like the control it gives me in the dom0. The dom0 becomes a very traditional packet-forwarding firewall that can decide whether or not to forward packets to each domU host. However, I recently found some deep weirdness in IP routing when I use this approach while needing multiple Ethernet interfaces on the domU. Here's an example:

Multiple IP numbers for Apache

Suppose the domU host, called webserv, hosts a number of websites, each with a different IP number, so that I have Apache doing something like¹:

            Listen 192.168.0.200:80
            Listen 192.168.0.201:80
            Listen 192.168.0.202:80
            ...
            NameVirtualHost 192.168.0.200:80
            <VirtualHost 192.168.0.200:80>
            ...
            NameVirtualHost 192.168.0.201:80
            <VirtualHost 192.168.0.201:80>
            ...
            NameVirtualHost 192.168.0.202:80
            <VirtualHost 192.168.0.202:80>
            ...
            

The Xen Configuration for the Interfaces

Since I'm serving all three of those sites from webserv, I need all those IP numbers to be real, live IP numbers on the local machine as far as the webserv is concerned. So, in dom0:/etc/xen/webserv.cfg I list something like:

            vif  = [ 'mac=de:ad:be:ef:00:00, ip=192.168.0.200',
                     'mac=de:ad:be:ef:00:01, ip=192.168.0.201',
                     'mac=de:ad:be:ef:00:02, ip=192.168.0.202' ]
            

… And then make webserv:/etc/iftab look like:

            eth0 mac de:ad:be:ef:00:00 arp 1
            eth1 mac de:ad:be:ef:00:01 arp 1
            eth2 mac de:ad:be:ef:00:02 arp 1
            

… And make webserv:/etc/network/interfaces (this is probably Ubuntu/Debian-specific, BTW) look like:

            auto lo
            iface lo inet loopback
            auto eth0
            iface eth0 inet static
             address 192.168.0.200
             netmask 255.255.255.0
            auto eth1
            iface eth1 inet static
             address 192.168.0.201
             netmask 255.255.255.0
            auto eth2
            iface eth2 inet static
             address 192.168.0.202
             netmask 255.255.255.0
            

Packet Forwarding from the Dom0

But, this doesn't get me the whole way there. My next step is to make sure that the dom0 is routing the packets properly to webserv. Since my dom0 is heavily locked down, all packets are dropped by default, so I have to let through explicitly anything I'd like webserv to be able to process. So, I add some code to my firewall script on the dom0 that looks like:²

            webIpAddresses="192.168.0.200 192.168.0.201 192.168.0.202"
            UNPRIVPORTS="1024:65535"
            
            for dport in 80 443;
            do
              for sport in $UNPRIVPORTS 80 443 8080;
              do
                for ip in $webIpAddresses;
                do
                  /sbin/iptables -A FORWARD -i eth0 -p tcp -d $ip \
                    --syn -m state --state NEW \
                    --sport $sport --dport $dport -j ACCEPT
            
                  /sbin/iptables -A FORWARD -i eth0 -p tcp -d $ip \
                    --sport $sport --dport $dport \
                    -m state --state ESTABLISHED,RELATED -j ACCEPT
            
                  /sbin/iptables -A FORWARD -o eth0 -s $ip \
                    -p tcp --dport $sport --sport $dport \
                    -m state --state NEW,ESTABLISHED,RELATED -j ACCEPT
                done  
              done
            done
            

Phew! So at this point, I thought I was done. The packets should find their way forwarded through the dom0 to the Apache instance running on the domU, webserv. While that much was true, I now have the additional problem that packets got lost in a bit of a black hole on webserv. When I discovered the black hole, I quickly realized why. It was somewhat atypical, from webserv's point of view, to have three “real” and different Ethernet devices with three different IP numbers, which all talk to the exact same network. There was more intelligent routing needed.³

Routing in the domU

While most non-sysadmins still use the route command to set up local IP routes on a GNU/Linux host, iproute2 (available via the ip command) has been a standard part of GNU/Linux distributions and supported by Linux for nearly ten years. To properly support the situation of multiple (from webserv's point of view, at least) physical interfaces on the same network, some special iproute2 code is needed. Specifically, I set up separate route tables for each device. I first encoded their names in /etc/iproute2/rt_tables (the numbers 16-18 are arbitrary, BTW):

            16      eth0-200
            17      eth1-201
            18      eth2-202
            

And here are the ip commands that I thought would work (but didn't, as you'll see next):

            /sbin/ip route del default via 192.168.0.1
            
            for table in eth0-200 eth1-201 eth2-202;
            do
               iface=`echo $table | perl -pe 's/^(\S+)\-.*$/$1/;'`
               ipEnding=`echo $table | perl -pe 's/^.*\-(\S+)$/$1/;'`
               ip=192.168.0.$ipEnding
               /sbin/ip route add 192.168.0.0/24 dev $iface table $table
            
               /sbin/ip route add default via 192.168.0.1 table $table
               /sbin/ip rule add from $ip table $table
               /sbin/ip rule add to 0.0.0.0 dev $iface table $table
            done
            
            /sbin/ip route add default via 192.168.0.1 
            

The idea is that each table will use rules to force all traffic coming in on the given IP number and/or interface to always go back out on the same, and vice versa. The key is these two lines:

               /sbin/ip rule add from $ip table $table
               /sbin/ip rule add to 0.0.0.0 dev $iface table $table
            

The first rule says that when traffic is coming from the given IP number, $ip, the routing rules in table, $table should be used. The second says that traffic to anywhere when bound for interface, $iface should use table, $table.

The tables themselves are set up to always make sure the local network traffic goes through the proper associated interface, and that the network router (in this case, 192.168.0.1) is always used for foreign networks, but that it is reached via the correct interface.

This is all well and good, but it doesn't work. Certain instructions fail with the message, RTNETLINK answers: Network is unreachable, because the 192.168.0.0 network cannot be found while the instructions are running. Perhaps there is an elegant solution; I couldn't find one. Instead, I temporarily set up “dummy” global routes in the main route table and deleted them once the table-specific ones were created. Here's the new bash script that does that (lines that are added are emphasized and in bold):

            /sbin/ip route del default via 192.168.0.1
            for table in eth0-200 eth1-201 eth2-202;
            do
               iface=`echo $table | perl -pe 's/^(\S+)\-.*$/$1/;'`
               ipEnding=`echo $table | perl -pe 's/^.*\-(\S+)$/$1/;'`
               ip=192.168.0.$ipEnding
               /sbin/ip route add 192.168.0.0/24 dev $iface table $table
            
               /sbin/ip route add 192.168.0.0/24 dev $iface src $ip
            
               /sbin/ip route add default via 192.168.0.1 table $table
               /sbin/ip rule add from $ip table $table
            
               /sbin/ip rule add to 0.0.0.0 dev $iface table $table
            
               /sbin/ip route del 192.168.0.0/24 dev $iface src $ip
            done
            /sbin/ip route add 192.168.0.0/24 dev eth0 src 192.168.0.200
            /sbin/ip route add default via 192.168.0.1 
            /sbin/ip route del 192.168.0.0/24 dev eth0 src 192.168.0.200
            

I am pretty sure I'm missing something here — there must be a better way to do this, but the above actually works, even if it's ugly.

Alas, Only Three

There was one additional confusion I put myself through while implementing the solution. I was actually trying to route four separate IP addresses into webserv, but discovered that I got found this error message (found via dmesg on the domU): netfront can't alloc rx grant refs. A quick google around showed me that the XenFaq, which says that Xen 3 cannot handled more than three network interfaces per domU. Seems strangely arbitrary to me; I'd love to hear why cuts it off at three. I can imagine limits at one and two, but it seems that once you can do three, n should be possible (perhaps still with linear slowdown or some such). I'll have to ask the Xen developers (or UTSL) some day to find out what makes it possible to have three work but not four.

---

¹Yes, I know I could rely on client-provided Host: headers and do this with full name-based virtual hosting, but I don't like to do that for good reason (as outlined in the Apache docs).

²Note that the above firewall code must run on dom0, which has one real Ethernet device (its eth0) that is connected properly to the wide 192.168.0.0/24 network, and should have some IP number of its own there — say 192.168.0.100. And, don't forget that dom0 is configured for vif-route, not vif-bridge. Finally, for brevity, I've left out some of the firewall code that FORWARDs through key stuff like DNS. If you are interested in it, email me or look it up in a firewall book.

³I was actually a bit surprised at this, because I often have multiple IP numbers serviced from the same computer and physical Ethernet interface. However, in those cases, I use virtual interfaces (eth0:0, eth0:1, etc.). On a normal system, Linux does the work of properly routing the IP numbers when you attach multiple IP numbers virtually to the same physical interface. However, in Xen domUs, the physical interfaces are locked by Xen to only permit specific IP numbers to come through, and while you can set up all the virtual interfaces you want in the domU, it will only get packets destine for the IP number specified in the vif section of the configuration file. That's why I added my three different “actual” interfaces in the domU.

Posted on Friday 24 August 2007 at 15:10 by Bradley M. Kuhn.

Submit comments on this post to <bkuhn@ebb.org>.