Large Scale Event Networking Case Study

This year Noise Productions is combining the networking requirements of Wings over Wairarapa and Wairarapa TV into a single piece of infrastructure to support all of the events needs as well as the requirements of a live Television Broadcast.

The networking requirements for Wings over Wairarapa have compounded year on year to the point where preparation and installation for the February event begin in early December. Rock solid Internet and networking connectivity are absolutely necessary for a large scale event these days with things like Ticketing, pass-outs, Eftpos, staff communications, audio distribution, video distribution, Live TV up-links, live radio uplinks, internet connectivity for trade stands, gold pass and VIP WiFi access, intercoms, event scheduling and runsheet publishing, pilot messaging, event publicity and media communications all dependent on rock solid and high bandwidth internet connectivity. If the internet stops, the whole event grinds to a hault and 30,000 people are left wondering what happened.

It begins with internet connectivity and because the airfield is outside the Masterton fibre zone, internet connectivity is somewhat of a challenge. We also need the internet to be absolutely bulletproof so it is connected to multiple locations with full redundancy and failover. Spark provide a single VDSL connection into an airfield hangar which provides around 40 megabits of bandwidth, this is bonded with a 300 megabit 23km wireless link to Popoiti Hill where Wairarapa TV is broadcast from. There is no internet at this location so the link bounces another 25km back to Masterton to the Wairarapa TV broadcast racks. At this location a 100 megabit fibre connection is used as the main internet bandwidth for the event. A 2km wireless link from this location to a secondary redundant location provides another 100 megabit fibre connection as a backup just in case the main one goes down. This provides three internet connections from different locations that are bonded together but also allow redundancy and failover if any one of them should go down.

A core firewall / router housed at Wairarapa TV provides all this failover cleverness and controls the entire network from offsite.

The next problem is getting internet and network connectivity spread out around the roughly 5 square kilometers that make up the Wings site. Careful consideration has to be made to ensure that there are no bottlenecks at any point in the network and that if any part of the network goes down it does not impact all or part of the site. One of the biggest mistakes WiFi installers commonly make is chaining links together. This results in the links closest to the start becoming quickly saturated, performance drops quickly and then dropouts cause all of the locations downstream to lose connectivity. This is bad but is also way too common and the reason why most WiFi installations are horrible.

To ensure Wings has awesome WiFi, Noise Productions uses a multi part hub and spoke design with full redundancy of the Hub.

There are effectively two independent 'Hubs' which any spoke of the network can connect to and each 'hub' has an independent link to the core network. The primary hub has the VDSL connection plus a long range wireless link to Popoiti Hill and the secondary hub has a high capacity link back to the primary hub plus another independent long range link to Popoiti. This means either one of the core hubs can go down and the entire system keeps running. If any 'spoke' goes down, only that areas is impacted and no other areas are taken down with it.
This design is much more complicated than an easy 'chain link' design but avoids the common problem in a chain link design of a single failure taking out the entire network or a single point of high load slowing down the entire rest of the network.

Now that we have connections to lots of points around the airfield, we need to provide wired and wireless internet connectivity covering each location. To do this we use extremely powerful industrial grade WiFi access points at each location. These access points have a special handoff capability so that you can walk around the Wings site and remain connected to WiFi the whole time with no dropouts, we also now encroach into a very important aspect of large scale event networking and that is Security.

With 30,000 punters on site, its highly likely that a few of the people visiting are going to have a go at breaking into your network so that they can either wreak havoc or just score themselves unlimited free internet. To avoid this, Noise Productions carves the network up with different layers of security and a special network topology called VLANs or virtual networks. This means that multiple networks for different functions can all run on the same infrastructure without any chance of someone on one network being able to access a more secure part of the network.

For example, we don't want people connected to the guest WiFi network to be able to access the ticketing systems or to be able to access the Wairarapa TV network in order to start their own Television Broadcast. We also wouldn't want a ticketing clerk to start a bitorrent session and download a massive video slowing down the internet for everyone else.

Even though the same wires and hardware are used for each network, they are controlled, monitored and throttled independently so that one idiot can't ruin it for everyone. In total, Wings uses 12 seperate virtual networks all running on the same infrastructure. Each virtual network has its own reservation of internet bandwidth, its own network priority and its own separate security layer. Virtual networks for the Television broadcast and ticketing systems are highly secure and restricted, while networks for Guest WiFi and Media are more open.