CCIE R&S lab - Quad-NIC PCI network cards option

The idea here is to build a dedicated server using Quad-NIC PCI network cards and then bind virtual router interfaces with real Ethernet interfaces that, transparently thru the host, connect to real switches. If you are studying for your CCIE Routing & Switching you will need at least 3 Quad-NIC network cards (3 * 4 ports = 12 Ethernet interfaces).

This dedicated server is going to be a terminal server too, you will need to configure Ser2net as explained in how to console into your devices.

Requirements

• Quad-NIC network cards.
• A PC or server with free PCI slots.
• One straight through Ethernet cable per connection to your switches.

Choosing and installing Quad-NIC network cards

The best thing to do is probably to run Linux on the server and use D-Link network cards as many people reported they have worked great for them. Depending of your budget and what you can find, on eBay for instance, you can look at the following list of Quad-NIC cards reported to work:

• D-Link DFE-570TX
• D-Link DFE-580TX
• Adaptec Quartet64 ANA-62044
• HP A5506-60102
• Sun Quad 501-4366

An important point to consider is to check the card compatibility with your motherboard. Note that you can use a PCI-X card into a PCI slot as this is backward compatible and be warned that some of these card were made for servers and therefore are quite long (e.g. Sun Quad), meaning they may not fit in a desktop computer case.

You can search on Google for how some CCIE candidates set up their lab and what hardware they used, for instance this page made by our friend of GNS3 Vault.

Once installed, check that the network cards are detected on Linux using lspci.

Example for D-Link DFE-580TX:

user@ubuntu:~$ lspci | grep Ethernet
 04:04.0 Ethernet controller: D-Link System Inc DL10050 Sundance Ethernet (rev 15)
 04:05.0 Ethernet controller: D-Link System Inc DL10050 Sundance Ethernet (rev 15)
 04:06.0 Ethernet controller: D-Link System Inc DL10050 Sundance Ethernet (rev 15)
 04:07.0 Ethernet controller: D-Link System Inc DL10050 Sundance Ethernet (rev 15)

Example for Sun Quad 501-4366:

user@ubuntu:~$ lspci | grep Ethernet
 05:00.1 Ethernet controller: Sun Microsystems Computer Corp. Happy Meal 10/100 Ethernet [hme] (rev 01)
 05:01.1 Ethernet controller: Sun Microsystems Computer Corp. Happy Meal 10/100 Ethernet [hme] (rev 01)
 05:02.1 Ethernet controller: Sun Microsystems Computer Corp. Happy Meal 10/100 Ethernet [hme] (rev 01)
 05:03.1 Ethernet controller: Sun Microsystems Computer Corp. Happy Meal 10/100 Ethernet [hme] (rev 01)

Topology

Virtual topology

Using physical network interfaces is quite straightforward in GNS3, just remember to start it with root permissions. Then link every of your virtual router interface to the Ethernet interface corresponding to the switch you want to get network connectivity with. Note that at the opposite of the following diagram, you can use multiple “clouds”, one for each network interface in order to make your diagram prettier (especially if using 12 connections).

GNS3 topology file

autostart = False
[127.0.0.1:7200]
    udp = 10000
    [[3640]]
        image = /path/to/ios/c3640.bin
        idlepc = 0x60578c20
        sparsemem = True
        ghostios = True
        chassis = 3640
    [[ROUTER R1]]
        model = 3640
        console = 2000
        slot0 = NM-1FE-TX
        f0/0 = nio_gen_eth:eth1
    [[ROUTER R2]]
        model = 3640
        console = 2001
        slot0 = NM-1FE-TX
        f0/0 = nio_gen_eth:eth2
    [[ROUTER R3]]
        model = 3640
        console = 2000
        slot0 = NM-1FE-TX
        f0/0 = nio_gen_eth:eth3
    [[ROUTER R4]]
        model = 3640
        console = 2001
        slot0 = NM-1FE-TX
        f0/0 = nio_gen_eth:eth4

Physical topology

Now that the network cards are operational, you can physically connect each interface with Cisco switches. This is that simple!

Host configuration

If needed, you can easily change the MTU on Linux, for instance for a MTU of 9000:

ifconfig ethx mtu 9000

The Quad-NIC option should also work with OSX and Windows without too much difficulty; however I haven’t heard about someone who did it and I cannot personally test it.

Pros and cons

The pros:

• Simple set up, one interface = one connection.
• No need for a breakout switch.
• You can connect to your lab from a distance, thanks to the terminal server.

The cons:

• Requires extra hardware (server and cards).
• Need to find the correct network card that is compatible with the server and that physically fit in it.
• Extra cabling.

Conclusion

This option is quite simple, providing you can get a computer with enough PCI slots, the right cards and you are not allergic to penguins (Linux). The option has proven to work very well and this is also one of the best option for a remote lab.