This method, originally posted on Hacki’s forum, uses the GNS3/Dynamips virtual switch, one physical network interface and a breakout switch to create a connection between virtual devices and Cisco Catalyst switches. The virtual switch is used because you can’t bind all the routers to the same physical interface and the breakout switch is a physical switch that splits VLANs to switches.
Table of Contents
Requirements
- One straight through Ethernet cable for the connection from the physical interface to the breakout switch.
- One crossover Ethernet cable per connection from the breakout switch to a real switch (can be straight through if one end supports MDIX).
- The physical network interface and the breakout switch must support vlan tagging (802.1Q/dot1q).
- The physical network interface should have a configurable MTU over 1500 (usually depends on the driver) as well as the breakout switch to prevent having to worry about MTU issues.
- Optional but recommended, the breakout switch should support dot1q tunnels (QinQ) and l2protocol tunneling features.
Topology
To demonstrate the QinQ with virtual switch option, I will make a simple lab consisting of 2 virtual routers and 2 real switches that can easily be extended to meet your needs. The following topology has been made on Mac OS X but excepting the real network interface name, the lab is gonna be the same on Windows and Linux.
Note it’s recommended to set the MTU on the physical network interface and breakout switch to a higher than default value to prevent headaches.
Virtual topology
Let’s build the virtual topology first. Once made, our lab will look like the following diagram. Remember that GNS3 must be started with Administrator or root rights in order to bind a virtual interface with physical interface. R1 is connected to the virtual switch, port 1 and R2 connected to port 2.
The virtual switch is configured to aggregate access VLANs to a trunk link bidden with the host physical network interface (en0 in this example).
The “cloud” represents the “real world”, in our case the cloud is gonna be the host real network interface (named en0 in this example). Always use the “Generic Ethernet” NIO, even on Linux has it works best.
R1 and R2 FastEthernet interfaces are configured. Note that R2 interface has 2 sub-interfaces, in vlan 10 and vlan 20 respectively.
R1 configuration
interface FastEthernet0/0 ip address 10.1.1.1 255.255.255.0 duplex full speed 100
R2 configuration
interface FastEthernet0/0 no ip address speed 100 full-duplex ! interface FastEthernet0/0.10 encapsulation dot1Q 10 ip address 10.2.10.1 255.255.255.0 ! interface FastEthernet0/0.20 encapsulation dot1Q 20 ip address 10.2.20.1 255.255.255.0
GNS3 topology file
autostart = False
[127.0.0.1:7200]
udp = 10000
[[3640]]
image = /path/to/ios/c3640.bin
idlepc = 0x60578c20
sparsemem = True
ghostios = True
chassis = 3640
[[ETHSW SW1]]
1 = access 101 R1 f0/0
2 = access 102 R2 f0/0
24 = dot1q 1 nio_gen_eth:en0
[[ROUTER R1]]
model = 3640
console = 2003
slot0 = NM-1FE-TX
f0/0 = SW1 1
[[ROUTER R2]]
model = 3640
console = 2005
slot0 = NM-1FE-TX
f0/0 = SW1 2
Physical topology
The physical is quite simple, the physical trunk link leaves the PC to connect to the breakout switch which then connect with the 2 other switches, SW1 and SW2. The breakout switch dispatches traffic to the switches based on the VLAN ID (802.1Q tag) in the frames.
How it works, example:
- On router R1, traffic leaves f0/0
- Virtual switch SW1 receives the traffic and inserts vlan tag 101. You can see this tag as the service provider tag.
- Virtual switch SW1 sends the tagged traffic out of the physical network interface (en0 in the diagram, naming depending of the OS).
- Breakout switch receives the traffic on the its trunk interface (GigabitEthernet 1/0/24 in this example).
- Breakout switch removes vlan tag 101 and sends whatever is left out to the interface bidden with vlan 101. (GigabitEthernet 1/0/1 in this example).
- Real switch receives the traffic (without vlan tag 101) on its interface. (f0/1 in this example).
R2 f0/0 uses sub interfaces with different vlans. When traffic leaves f0/0.10, it is tagged with vlan tag 10, virtual switch adds the service provider tag 102 on top of tag 10. This is called QinQ.
Next, the 2 real switches are configured.
SW1 configuration
interface FastEthernet0/1 switchport mode access spanning-tree portfast ! interface Vlan1 ip address 10.1.1.254 255.255.255.0
SW2 configuration
Note that 2 SVI interfaces are configured, one for vlan 10 and the other for vlan 20.
interface FastEthernet0/1 switchport trunk encapsulation dot1q switchport mode trunk spanning-tree portfast ! interface Vlan10 ip address 10.2.10.254 255.255.255.0 ! interface Vlan20 ip address 10.2.20.254 255.255.255.0
The breakout switch
The breakout switch model is crucial if you want to emulate as much of the R&S lab as possible. QinQ support for layer 2 protocols such as CDP, VTP, STP work bidirectionally only with 3750 and 4948 switches (and surely with more expensive models); however you can use pretty much use any old switch as a breakout switch and all your layer 3 traffic should be fine, for instance a Cisco 2950 or 3550 as a breakout switch is still fine and more importantly is less costly. You could even use a non-Cisco switch as long as it supports 802.1Q trunking.
Here is what to expect for CDP using the following models as a breakout switch:
- Cisco 2950: CDP will not work in both direction as layer 2 tunneling is not supported on this switch.
- Cisco 3550: CDP work only in one direction, neighbors cannot be seen on the switches. Regardless of the IOS version.
- Cisco 3560: CDP work only in one direction, neighbors cannot be seen on the switches. Regardless of the IOS version.
- Cisco 3750: Bidirectional CDP and fully functioning lab (layer 2 and layer 3) using IP Services image. IP Base image does not support tunneling.
- Cisco 4948: Bidirectional CDP and fully functioning lab (layer 2 and layer 3) using minimum of IP Services image.
Cisco 3750 as a breakout switch
The switch used here is a WS-C3750G-24 with 24 GigabitEthernet ports and with the c3750-ipservicesk9-mz.122-55.SE4 IOS image. 1998 is the maximum MTU value. Also note that turning off CDP on all the interfaces assures us there will be no duplication of CDP entries and confusion on the lab equipment.
hostname Breakout ! system mtu 1998 ! vtp mode transparent ! vlan 101-102 ! interface GigabitEthernet1/0/1 description Down to SW1 f0/1 switchport access vlan 101 switchport mode dot1q-tunnel l2protocol-tunnel cdp l2protocol-tunnel stp l2protocol-tunnel vtp no cdp enable spanning-tree portfast ! interface GigabitEthernet1/0/2 description Down to SW2 f0/1 switchport access vlan 102 switchport mode dot1q-tunnel l2protocol-tunnel cdp l2protocol-tunnel stp l2protocol-tunnel vtp no cdp enable spanning-tree portfast ! interface GigabitEthernet1/0/24 description Up to PC interface switchport trunk encapsulation dot1q switchport trunk allowed vlan 101-104 switchport mode trunk l2protocol-tunnel cdp l2protocol-tunnel stp l2protocol-tunnel vtp no cdp enable spanning-tree portfast !
Cisco 3550 or 3560 as a breakout switch
The switch used here is a WS-C3550-48 with 48 FastEthernet interfaces and the c3550-ipservicesk9-mz.122-44.SE6 IOS image. The maximum MTU value is 1546. Remember layer 2 tunneling is not supported on a regular trunk interface, only access and dot1q-tunnel modes are supported.
hostname Breakout ! system mtu 1546 ! vtp mode transparent ! vlan dot1q tag native ! vlan 101-102 ! interface FastEthernet0/24 description Up to PC interface switchport trunk encapsulation dot1q switchport trunk allowed vlan 101-102 switchport mode trunk l2protocol-tunnel cdp spanning-tree portfast trunk ! interface FastEthernet0/1 description Down to SW1 f0/1 switchport access vlan 101 switchport mode dot1q-tunnel l2protocol-tunnel cdp no cdp enable spanning-tree portfast ! interface FastEthernet0/2 description Down to SW2 f0/1 switchport access vlan 102 switchport mode dot1q-tunnel l2protocol-tunnel cdp no cdp enable spanning-tree portfast
Cisco 2950 as a breakout switch
Note that using a 2950 will not work for QinQ as it doesn’t support do1q-tunnel mode. In our topology, it means that R2 tagged traffic (vlan 10 and 20) cannot pass through the switch. This eliminates the option to make router-on-a-stick scenarios.
hostname Breakout ! interface FastEthernet0/1 description Down to SW1 f0/1 switchport access vlan 101 switchport mode access no cdp enable spanning-tree portfast ! interface FastEthernet0/2 description Down to SW2 f0/1 switchport access vlan 102 switchport mode access no cdp enable spanning-tree portfast ! interface FastEthernet0/24 description Up to PC interface switchport trunk allowed vlan 101,102 switchport mode trunk spanning-tree portfast trunk
Verification
If using a 3750 as the breakout switch, you should see a CDP neighbor on both sides, otherwise only on the virtual routers. Other checks are done using ping and arp.
R1#show cdp neighbors
Capability Codes: R - Router, T - Trans Bridge, B - Source Route Bridge
S - Switch, H - Host, I - IGMP, r - Repeater
Device ID Local Intrfce Holdtme Capability Platform Port ID
SW1 Fas 0/0 122 S I WS-C3550-4Fas 0/1
SW1#show cdp neighbors
Capability Codes: R - Router, T - Trans Bridge, B - Source Route Bridge
S - Switch, H - Host, I - IGMP, r - Repeater, P - Phone
Device ID Local Intrfce Holdtme Capability Platform Port ID
R1 Fas 0/1 165 R S I 3640 Fas 0/0
R1#ping 10.1.1.254 Type escape sequence to abort. Sending 5, 100-byte ICMP Echos to 10.1.1.254, timeout is 2 seconds: !!!!! Success rate is 100 percent (5/5), round-trip min/avg/max = 20/22/24 ms R1#show arp Protocol Address Age (min) Hardware Addr Type Interface Internet 10.1.1.1 - cc03.0db4.0000 ARPA FastEthernet0/0 Internet 10.1.1.254 0 000a.41ad.ca00 ARPA FastEthernet0/0
SW1#show arp Protocol Address Age (min) Hardware Addr Type Interface Internet 10.1.1.1 0 cc03.0db4.0000 ARPA Vlan1 Internet 10.1.1.254 - 000a.41ad.ca00 ARPA Vlan1 SW1#debug ip icmp ICMP packet debugging is on *Mar 1 01:09:04.500: ICMP: echo reply sent, src 10.1.1.254, dst 10.1.1.1 *Mar 1 01:09:04.515: ICMP: echo reply sent, src 10.1.1.254, dst 10.1.1.1 *Mar 1 01:09:04.547: ICMP: echo reply sent, src 10.1.1.254, dst 10.1.1.1 *Mar 1 01:09:04.555: ICMP: echo reply sent, src 10.1.1.254, dst 10.1.1.1
R2#show cdp neighbors
Capability Codes: R - Router, T - Trans Bridge, B - Source Route Bridge
S - Switch, H - Host, I - IGMP, r - Repeater
Device ID Local Intrfce Holdtme Capability Platform Port ID
SW2 Fas 0/0 149 S I WS-C3550-4Fas 0/1
R2#ping 10.2.10.254
Type escape sequence to abort.
Sending 5, 100-byte ICMP Echos to 10.2.10.254, timeout is 2 seconds:
!!!!!
Success rate is 100 percent (5/5), round-trip min/avg/max = 20/24/32 ms
R2#ping 10.2.20.254
Type escape sequence to abort.
Sending 5, 100-byte ICMP Echos to 10.2.20.254, timeout is 2 seconds:
!!!!!
Success rate is 100 percent (5/5), round-trip min/avg/max = 20/24/36 ms
R2#show arp
Protocol Address Age (min) Hardware Addr Type Interface
Internet 10.2.10.1 - cc05.0db4.0000 ARPA FastEthernet0/0.10
Internet 10.2.20.1 - cc05.0db4.0000 ARPA FastEthernet0/0.20
Internet 10.2.20.254 0 0012.8034.ae80 ARPA FastEthernet0/0.20
Internet 10.2.10.254 0 0012.8034.ae80 ARPA FastEthernet0/0.10
SW2#sh arp Protocol Address Age (min) Hardware Addr Type Interface Internet 10.2.10.1 1 cc05.0db4.0000 ARPA Vlan10 Internet 10.2.20.1 1 cc05.0db4.0000 ARPA Vlan20 Internet 10.2.20.254 - 0012.8034.ae80 ARPA Vlan20 Internet 10.2.10.254 - 0012.8034.ae80 ARPA Vlan10
Host configuration
Host configuration is mainly about having the NIC used for the physical trunk detected by the OS and also the NIC must supports vlan tags (802.1Q).
Mac OS X
The configuration works using 2 different interfaces on OSX, first with the integrated network interface on my Macbook pro (early 2011), a Broadcom BCM5701.
We can set a custom MTU to 9000 (jumbo MTU) using OSX GUI.
Or using the command line.
$ networksetup -listValidMTURange en0 Valid MTU Range: 1280-9000 $ sudo ifconfig en0 mtu 9000 $ ifconfig en0 en0: flags=8963<UP,BROADCAST,SMART,RUNNING,PROMISC,SIMPLEX,MULTICAST> mtu 9000 options=2b<RXCSUM,TXCSUM,VLAN_HWTAGGING,TSO4> ether 3c:07:54:07:e5:33 inet6 fe80::3e07:54ff:fe07:e533%en0 prefixlen 64 scopeid 0x4 media: autoselect (100baseTX <full-duplex>) status: active
Second test, with a USB Ethernet Adapter (ASIX AX88772A chipset). This can be useful to have one if your main Ethernet interface is used for your Internet connection for example.
But this time, setting a MTU over 1500 is impossible.
Even using the command line.
$ networksetup -listValidMTURange en3 Valid MTU Range: 1280-1500 $ sudo ifconfig en3 mtu 9000 ifconfig: ioctl (set mtu): Invalid argument $ ifconfig en3 en3: flags=8963<UP,BROADCAST,SMART,RUNNING,PROMISC,SIMPLEX,MULTICAST> mtu 1500 options=4<VLAN_MTU> ether 00:60:6e:05:0d:a5 inet6 fe80::260:6eff:fe05:da5%en3 prefixlen 64 scopeid 0x8 media: 100baseTX <full-duplex> status: active
Capturing on the Ethernet interface shows CDP in both direction.
ICMP packets are correctly tagged when pinging from R1.
ICMP packets are correctly tagged when pinging from R2 as well, notice the 2 VLAN tags as expected.
Linux
On Linux (Ubuntu 11.10), I used my USB-to-Ethernet adapter detected as eth3.
user@host:~$ lsusb Bus 001 Device 003: ID 0b95:772a ASIX Electronics Corp. user@host:~$ dmesg [ 403.719681] asix 1-2:1.0: eth0: register 'asix' at usb-0000:00:06.0-2, ASIX AX88772 USB 2.0 Ethernet [ 403.720524] usbcore: registered new interface driver asix [ 404.000616] udevd[432]: renamed network interface eth0 to eth3 [ 404.033314] asix 1-2:1.0: eth3: link down [ 404.063245] ADDRCONF(NETDEV_UP): eth3: link is not ready [ 405.343438] ADDRCONF(NETDEV_CHANGE): eth3: link becomes ready [ 405.389150] asix 1-2:1.0: eth3: link up, 100Mbps, full-duplex, lpa 0xC1E1
Then we can set the MTU.
user@host:~$ sudo ifconfig eth3 mtu 9000
user@host:~$ ifconfig eth3
eth3 Link encap:Ethernet HWaddr 00:60:6e:05:0d:a5
inet6 addr: fe80::260:6eff:fe05:da5/64 Scope:Link
UP BROADCAST RUNNING MULTICAST MTU:9000 Metric:1
RX packets:552 errors:0 dropped:0 overruns:0 frame:0
TX packets:56 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:1000
RX bytes:31466 (31.4 KB) TX bytes:12785 (12.7 KB)
Capturing on eth3 shows bidirectional connectivity work. It was too easy!
user@host:~$ sudo tcpdump -vvv -i eth3 vlan 102
tcpdump: WARNING: eth3: no IPv4 address assigned
tcpdump: listening on eth3, link-type EN10MB (Ethernet), capture size 65535 bytes
20:30:37.052792 IP (tos 0x0, ttl 255, id 30, offset 0, flags [none], proto ICMP (1), length 100)
10.2.10.1 > 10.2.10.254: ICMP echo request, id 6, seq 0, length 80
20:30:37.072950 IP (tos 0x0, ttl 255, id 31, offset 0, flags [none], proto ICMP (1), length 100)
10.2.10.1 > 10.2.10.254: ICMP echo request, id 6, seq 1, length 80
20:30:37.094321 IP (tos 0x0, ttl 255, id 32, offset 0, flags [none], proto ICMP (1), length 100)
10.2.10.1 > 10.2.10.254: ICMP echo request, id 6, seq 2, length 80
20:30:37.104457 IP (tos 0x0, ttl 255, id 33, offset 0, flags [none], proto ICMP (1), length 100)
10.2.10.1 > 10.2.10.254: ICMP echo request, id 6, seq 3, length 80
20:30:37.114632 IP (tos 0x0, ttl 255, id 34, offset 0, flags [none], proto ICMP (1), length 100)
10.2.10.1 > 10.2.10.254: ICMP echo request, id 6, seq 4, length 80
Windows
Using the Broadcom integrated Ethernet interface on Windows, I changed the MTU value in the advanced properties for my NIC (accessible via Windows Device Manager). You can also change this MTU using utilities like Broadcom Advanced Control Suite (for Broadcom NICs) or Intel PROset (for Intel cards).
Once the lab build as explained before in the topology section, we can start testing our connectivity.
As you can see, ping failed. Checking the ARP table, we see that the resolution for 10.1.1.254 has failed too, meaning no arp reply has not been received. If you try to ping again while capturing on the Broadcom interface, we see that ARP replies are there but with a closer look, we can see that there is no VLAN tag in the Ethernet frame, consequently our virtual switch just ignore the incoming frames.
This problem is because many drivers strip off “outer” 802.2qQ tags on incoming frames, the Broadcom driver for my NIC is one of them.
Fortunately there is a solution to this issue. A reference page by Wireshark explain the solution for some NIC models (Intel, Broadcom etc.). There is also this older document from Cisco that lists NICs that do not strip 802.1q tags and give some workaround using Windows registry for a few others.
Personally, I used the following Windows registry hack for my Broadcom NetXtreme Gigabit Ethernet NIC:
- Run the Registry Editor (regedit)
- Hit CTRL + Home to go to the top of the registry
- Hit F3 to bring up a search
- Search for TxCoalescingTicks, you should end up in something like HKLM\SYSTEM\CurrentControlSet\control\Class\{4D36E972-E325-11CE-BFC1-08002bE10318}\00xx
- Right-click on the instance number and add a new string value.
- Enter “PreserveVlanInfoInRxPacket” and give it the value “1″.
- Reboot the PC.
Once the registry hacked, I tested again and this time ping and ARP resolution worked!
Capturing on the interface, I could see ARP replies tagged as expected.
Last note for Windows, it was impossible for me to find a way for my USB-to-Ethernet adapter (ASIX AX88772 chipset) to not strip the “outer” 802.2qQ tag. Apparently, this is done in hardware and not changeable via the driver. This is why I do not recommend the use USB-to-Ethernet adapter with the virtual switch option because I found out lot of them are based on ASIX chipsets.
Known issue
It has been reported that IRB (Integrated Routing and Bridging) doesn’t work using a breakout switch (at least with a Cisco 3550), the MAC table is not stable.
04:58:27: %SW_MATM-4-MACFLAP_NOTIF: Host ca0c.0a04.0008 in vlan 102 is flapping between port Fa0/2 and port Fa0/17 04:59:42: %SW_MATM-4-MACFLAP_NOTIF: Host ca0c.0a04.0008 in vlan 102 is flapping between port Fa0/2 and port Fa0/17
IRB requires a unique network interface directly connected to the real switches. You must bypass the breakout switch or you will have MAC table issues.
Pros and cons
The pros:
- A single cable from the PC.
- Works very well with OSX and Linux.
- Can easily switch between CCIE vendor topologies if the breakout switch has enough ports.
- Perfect if using a laptop.
The cons:
- Requires an extra switch for the vlan breakout.
- Need a expensive breakout switch to have layer 2 tunneling working both ways.
- The network interface must support 802.1Q
Conclusion
The virtual switch may be one of the cheapest solution depending of the breakout switch model. For instance a CCNA candidate could use a cheap Cisco 2950 as the breakout switch. This is also the best option if you want to use a laptop and don’t want to tangle yourself in a bunch of USB-to-Ethernet adapters.
