Linux at the BBC
By: Damion Yates
Friday, October 29, 1999 11:46:35 AM EST
URL: http://www.linuxplanet.com/linuxplanet/reports/1176/1/
Introduction
I'm sure you have heard of the BBC and will be aware of its long history in technical innovation, as the corporation invented NICAM and was among the first to offer high quality PAL colour television.
The BBC is making pioneering moves in the world of digital media with its digital terrestrial TV system. We are encoding existing and new channels to about 5 megabits per second each in MPEG, and transmitting over normal broadcast frequencies. This means we can fit several channels onto the air where there used to be one, plus allow much more flexibility with new interactive services on the same bandwidth.
My team is responsible for the support of the BBC's Unix systems, used for serving web pages, realmedia and now this new digital text service. We all use Linux at home when on call at night (to some degree) to support this. I personally have FreeBSD, Solaris-x86 and Linux, but rarely leave Linux.
The BBC's General IT teams use Linux all over the internal network. A number of content producers are starting to use Linux rather than Windows/Mac, but this is mainly those producing interactive Website elements such as perl CGI scripts.
In this article I'll look at some of the BBC's practical applications of Linux and why we use Linux rather than alternatives.
Practical Applications
DHCP and DNS
The BBC uses Linux to provide almost all the internal DNS and DHCP services. In fact, if a few stopped, there would be serious trouble. I believe we have about 30,000 machines dependent on Linux-run DHCP/BOOTP for IP addressing and maybe even a few more dependent on Linux-run DNS for address resolution. We use BMC's Software Patrol to monitor the Linux operating system and have written bespoke modules to monitor DNS/DHCP.
Digital Text Service
Along with the new medium of digital TV is a newer text service to replace Ceefax.
Ceefax, I should explain to readers outside the UK, is a system which encodes services in plain text for viewing alongside TV. However, rather than being addressed to a particular recipient and sent over the Internet, it is broadcast along with the TV signals from the BBC and other UK broadcasters in the UK and some European countries. Suitably equipped TVs can decode it and get access to several hundreds of pages of text, typically from the BBC's news service (news/tv-listings/weather). It is quite an old system; the graphics capabilities are limited to a text-only screen, roughly 40x25 charactors in size. The newer Digital text server should be faster with more interactivity and features, such as hyperlinks, real TV-quality image stills, proportional anti-aliased fonts and quarter-screen actual TV image while browsing.
The channel and signals for this service, known as an MHEG stream, are melded into the end broadcast stream before actual transmission by expensive dedicated hardware. With the MPEG video data which provides the TV pictures, the digital text services are transmitted into millions of homes in the UK.
The stream of digital text data that is pumped in to this hardware in TV Centre in London is transmitted over TCP/IP on a lowly 10base2 (thinnet) cable from Linux. It is a dual-redundancy system with a complete double-up of every element.
The Linux boxes are low-cost PCs running a slim Slackware setup, with nothing apart from basic network utils and a single ELF binary called 'spam' (Spoolfile Playout And Management system) that takes parameters and talks over TCP/IP to the dedicated hardware. The main idea is that, if one dies, we can cheaply and quickly get a replacement box installed and running. We can, of course, immediately switch to the fail-over system, as there is full redundancy--but would only be running on one leg until we got the replacement in place. So far the biggest problem has been network cards failing or losing IRQ/IO settings.
If the daemons on both Linux PCs stopped or both boxes went down, it would immediately halt the digital text service. In the Ceefax (Teletext) world this is equivalent to the text completely freezing, the clock not ticking on, the numbers not cycling through any more, etc... This is very serious.
Their daemons take a file called a "spoolfile carousel", which they constantly cycle through, playing out the contents. This file is updated via FTP from an E250 UltraSPARC Sun running Solaris2.6. A number of tcl scripts and C binaries which accept data from various sources in the BBC, mainly (at the moment) from Ceefax/News, convert this into the compiled MHEG spoolfiles before uploading. Our data is always up-to-date, with the latest weather/news/tv schedules, and it is this right-on-time delivery that puts the BBC, as a corporation, ahead of the opposition.
So in this case it is easy to see how the Linux machines are pivotal to the Corporation's core business of providing news...in a highly visible position, too.
Rapidly needed webservers
In a few instances a PC has been quickly converted into a Linux webserver for the development or demoing of a website,where a local drive wouldn't work--for example where SSI (Server Side Includes) or CGIs were used. BBC Online used to do most of the development on a few basic Linux PCs, though we're moving them to Solaris so that the development environment matches what we use on the Internet. The BBC TV show 'Tomorrow's World', which I should explain for non-UK readers is a well-established TV programme about new technology and scientific advances, hosted the Turing Test where three studio guests were shown using a web interface one-to-one chat system. They had to chat for 5 minutes and determine if they were talking to a computer or a human.
The same system had run for a week before the show on the BBC Web site. There was one human and 2 computer programs; the programs were from outside sources, and I wrote the CGI wrapper interface for the public to chat to them or the human. For the show, which went out live, we had a Linux box with apache and the same setup as the BBC servers for the CGI interface. It performed flawlessly on the night, as you'd expect. I'd tried to get Linux on the desktop machines, which would help show that Tomorrows World was a cutting-edge show, but there were complications and not enough time, so in the end it had to be Windows clients.
WebCam
We also use Linux for lots of other little things almost too trival to mention. For instance:
the webcam we have looking towards the Millennium dome;
http://news.bbc.co.uk/hi/english/static/domecam/default.htm also has a Linux box below it running the realencoding software recently released by www.real.com;
the award-winning Osprey cam site from http://www.bbc.co.uk/animalzone/, sadly now no longer in use, used a 4-year-old Linux box and gfx capture card. The linux box would dial up to a local ISP to save money, then ftp the images at regular intervals up to the webserver. A phone line was put in especially for this, as it is in an incredibly remote moor in Scotland--about 3 hours drive from any large city! This ran fine for months and allowed us to diagnose and fix any faults with ease, as we could connect in and have the same control as if it were right next to us.
Advantages of Linux
Clearly we use all kinds of equipment in our
IT strategy, but for particular situations, Linux
is a winner, and I'm about to tell you why.
It's free
The Linux licensing means that there are none of
the financial constraints that might exist if extra
copies of licenses had to be purchased. So
this does encourage us to use it for all kinds
of applications.
It uses standards
When you are working with broadcast media you must always have an eye for standards. The material we send out has to be received by millions of people, so it must be compatible with whatever equipment those people have. Because this is part of our core business, in other areas we have a respect for standards and the transparency they bring to the internal workings of our complex systems.
The support is good
Our best support success story for Linux was my contacting Donald Becker at NASA asking about Driver support for the 3C900B PCI network card. I first did several net searches using the local Usenet Newsserver, then http://www.dejanews.com/, as it was back then, and I asked on IRC #linux channels. I then read the kernel sources for said driver, and found, as I'd expected, detailed comments. After drawing a blank on these usual sources of information I decided to email the author of the actual device driver. His address appears in most peoples' Linux boot sequence, and also in the src, in case you're wondering how I knew. I emailed, asking him to reply to the work mailing list. The question was basically whether this particular brand of new 3com network card was compatible with the existing driver. His C comments in the kernel sources implied that there were some problems with some of the newer ones that he was working on, and he listed some successes and failures. My card, the one that came with our machines, wasn't even listed, as it was so new.
He replied in amazing detail within one hour! Never in a million years had we expected the actual author of the C driver code for the operating system we were using to reply back with such great detail so promptly. I don't think you could _EVER_ get support like that from anywhere else. Due to it being Open source, I'd already been able to vastly surpass any possibility, had it been any other vendor OS, of examining the situation myself. .
It is efficient
Linux will run on low-specification hardware. As
I mentioned above, the digital text service
machines that directly feed the broadcasts
are very basic. The servers for most of our
DNS/DHCP servers are Pentium 166MHz with 32
MBytes of RAM, or similar spec. Even on these,
processor utilisation is low in normal running.
Remote access is easy
Linux boxes, as with all UNIX systems, provide X
or tty access to shells which communicate with
the kernel to provide interactive services.
Either mechanism allows _full_ control of the
system and, better still, has no relation to where
you are based. Controlling a command line shell
from the console is the same as controlling a
command line shell 800 miles away over a modem
in a remote moor in Scotland. You can even
'X' a graphical system to yourself if you don't
feel you can control everything you need from
the command line.
Summary
Linux is used by the BBC behind the scenes in mission-critical positions. It's known about and respected by most technically-oriented BBC staff; as our numbers grow, we may be able to convince shows such as 'Tomorrow's World' to use screenshots of Netscape under Linux systems, instead of MacOS and Windows, to help improve the BBC's technical image.
Ciaran Anscomb, Richard Cooper, Andrew Gorton, Dipen Patel, Oliver Skudder and Andrew Wales.
Damion Yates
Senior Internet Operations Engineer, BBC