Infocrossing and S/390 Linux: An ASP's Story
By: Scott Courtney
Wednesday, February 28, 2001 08:31:59 AM EST
URL: http://www.linuxplanet.com/linuxplanet/reports/3050/1/
A Visit to the Co-Lo Penitentiary
They keep their computers in cages at Infocrossing, like a gigantic prison for miscreant microchips. The cages aren't to keep the computers in, though, but to keep intruders out. Infocrossing, Incorporated, headquartered in Leonia, New Jersey, is in the Application Service Provider (ASP) hosting business. With a strong background in traditional mainframe environments, IBM's System/390 architecture was a natural choice for Infocrossing as they started branching out into the Linux hosting business.
Application Service Providers, for those not familiar with the term, are companies that offer application software to clients on a centrally-located (and centrally-managed) host. The idea is that businesses gain the benefit of running sophisticated applications while avoiding the cost of deploying the needed hardware and software in-house. In the days before the Internet, such arrangements were common and were built using mainframe computers and remote timesharing connections. It was in this environment that Infocrossing was born, over twenty years ago. Now the Internet -- and virtual private networks (VPNs) that exploit its ubiquity -- give companies like Infocrossing a new way to bring their services to customers. At the same time, ASPs are creating new opportunities for mainframe systems, especially now that Linux is a reality at the high end.
Tom Laudati, Senior Vice President for Enterprise Engineering at Infocrossing, says that the company brought mainframe lessons into a distributed world, unlike some companies which moved in the other direction. "All the disciplines we learned as far as backup and recovery and implementing new pieces of software to make sure they work with your other software, we've learned over the past twenty-five years. When you put that PC on the desktop, this all goes out the window." They realized that distributed infrastructures, such as intranets, weren't being managed adequately for mission-critical business needs, and decided this gap represented an opportunity. Over time, Infocrossing started offering managed hosting of UNIX and NT systems, and eventually they branched out to provide colocation facilities as well.
The colocation business, housed in three large, geographically dispersed data centers, is the reason for the cages. They prevent one colocation customer from gaining console access to another customer's equipment, and they are an essential part of physical security at the site. "Customers are allowed access to the facility," says Laudati. "We're a blended environment." The part of the site that handles traditional mainframe (timesharing) outsourcing not accessible to customers, but the colocation facility is. The standard environment has a 7x8 foot cage for each customer's own equipment. The S/390 Linux host, by contrast, is accessible to customers via remote login only. Even when you understand the purpose of those cages, though, the effect is somewhat surreal.
Between their traditional mainframe hosting services, and the new UNIX, Linux, and NT managed hosting, and the colocation facilities, Infocrossing has grown into a US$40 million company with several hundred employees. About three-fourths of the employees are technical people, according to Laudati. The company operates a thirty thousand square foot (about 2800 square meters) facility in Leonia, New Jersey and a fifty-two thousand square foot (about 4800 square meters) facility in Norcrosse, Georgia. They are building an even larger facility in Sterling, Virginia.
Though the company is nominally operating system and hardware agnostic, they have a definite preference for IBM hardware and infrastructural software and are a long-term partner with IBM. Laudati says the mainframe bias is a result of two factors, scalability and reliability, and cites the specific example of ACTS, a company that provides an educational testing and scoring service to many clients, including a statewide public school system. Laudati says this kind of customer needs a very dynamic hosting environment because the workload of the system can vary widely. "One time in the public schools it could be ten students," he remarks, "and the next time it could be ten thousand."
Reliability and Expandibility: The Enabling Framework of Tivoli
Public-access sites can also experience sharp peaks in demand as a result of news events or other external, random factors. Laudati says that Infocrossing customers experiencing sudden, unanticipated peaks, can dynamically purchase additional capacity and then release it when it is no longer needed. "We can monitor [MIPS usage]. When it gets to around eighty or ninety percent, we can dynamically increase the amount of resources provided." For customers using rack-mounted RISC or Intel servers, additional hardware may have to be brought online, but with Linux running on an S/390, the entire process can be handled trivially by a system operator or even automatically by a load management program.
Reliability is the other factor that keeps Infocrossing firmly in the pro-mainframe camp. They have customers using other systems, such as Solaris, Linux, and NT on RISC and Intel hardware. Infocrossing even uses NT (sometimes with Citrix products for remote access) in its own systems. But when the bottom line is depending on the machine never going down, Linux running under IBM's VM hypervisor operating system is the configuration of choice.
Infocrossing can in some ways be thought of as a "fourth tier" provider, according to Tom Laudati. Using the ACTS testing company's deployment as an example, he illustrates Infocrossing's role this way: "We have the end user as the first tier, then the school [as the second], then ACTS is providing the application, then we provide the infrastructure." In other words, Infocrossing provides a place for ASPs to host their applications but is not itself an ASP, at least in this particular context.
Although a long-term (and satisfied) partner with IBM, Infocrossing found that IBM's middleware and management products didn't provide the full depth of capabilities that they wanted to offer their clients. They started with IBM's Tivoli management framework as a base, then added a proprietary upper layer as a value-added proposition for their own managed hosting customers. Says Laudati, "We've actually developed a product called Infocap, using Tivoli as a framework, which will manage...NT or UNIX [or Linux] environments out there." He adds that this was not a trivial effort: "It took a dozen engineers about fifteen or sixteen months to get this integrated."
IBM's Tivoli products, most of which are now available for Linux, include tools for security management (Policy Director, PKI, Secure Way Directory, and others), storage management (Tivoli Storage Manager), trouble ticketing (Tivoli Service Desk), and other infrastructural functions. Robert Graham, Infocrossing's Chief Technical Officer, refers to Tivoli as a "marketecture" umbrella containing products that are not always designed from the ground up to work together. Moreover, the Tivoli suite of products is an enterprise-class framework, and that kind of software requires more than just installation. There is an entire design process in which an enterprise has to decide how they want to use the tools that Tivoli provides. Infocrossing's executives say the company is very satisfied with the Tivoli product suite, but emphasize that Tivoli is an enabling framework, not a simple end-user application program.
Integration is the Key: Putting the Tivoli Pieces Together
Fred DelGaudio is Infocrossing's Senior Vice President for Product Development. He describes the company's large-scale Tivoli deployment as a multiphase process. "The first challenge was to get the products to work individually. The second challenge was to get them to work together. We wrote some code and some APIs, as well as database translation routines, to make them communicate with one another."
"We'll use distributed monitoring to instrument storage utilization of the server environment, which is pretty typical," says DelGaudio, acknowledging that some of Infocrossing's services are a basic feature of all ASP hosting companies. But he emphasizes that they don't stop there. "When utilization reaches a certain percentage, we'll throw out an alert. What distinguishes us is that we've built integration between the various Tivoli products. That alert goes into Tivoli Service Desk, so I don't need a person to open a trouble ticket. It's automatic. The third component that we've integrated is Tivoli Storage Manager, which can dynamically add storage to the user environment. If that whole process was successful, then that trouble ticket and alert can be automatically closed. If the process was unsuccessful, then it escalates from a yellow to a red [status]. Netview is also integrated into Infocap. If Netview notices a problem with one of the components on the network, it will turn the component red [on the console]."
Graham says that his company's enhancement of the Tivoli management products focuses on closing the feedback loop on resolved problems, not just automating the process of solving them one-by-one. The emphasis is on long-term, evolutionary productivity gains. "We've integrated it [Netview] with the Service Desk component so that a trouble ticket gets written as well. We script what gets written into the trouble ticket, so that if it is something that we've seen before and requires manual intervention, the operator sees the script of what to do. The command center folks are also responsible for maintaining the trouble ticketing script. If they followed the steps and it worked, they have nothing else to do. If they had to add a step, it's their responsibility to add that step to the trouble ticket script as well, so that the next time it will be included. They are part of a very large feedback loop."
Tom Laudati says Infocrossing had a number of solid business reasons for choosing Linux on System/390 over other platforms. Although they still support NT and UNIX as well as their legacy mainframe hosting clients, Laudati says that System/390 and Linux are their preferred platform for most new development. "There are some applications that aren't yet ported to Linux for S/390," he says, "so that would dictate where you would run. Linux for S/390 is certainly our preferred platform. It's much easier to administer and manage than the distributed hardware environments."
Ease of administration is mentioned often by early adopters of Linux for S/390, and in fact has been an argument made by traditional mainframe advocates for decades. Laudati says this was a key factor at Infocrossing as well, adding, "We can build a new Linux environment in about fifteen minutes." Laudati also mentioned the issue of reduced floor space, an expensive commodity in a secured raised-floor data center. Even IBM's largest zSeries mainframe is only about the size of a telephone booth, and large mass storage arrays are similarly sized regardless of the CPU type. Gone are the days of mainframe disks the size of clothes washers; modern mainframes use arrays of small-footprint drives not unlike those connected to high-end Intel or RISC servers. Modern CMOS processors have eliminated the need for large water cooling systems with thick hoses trailing under the floor tiles.
Linux for S/390 can run natively on the physical hardware, just as if the S/390 or zSeries were a big PC, or it can run in an LPAR (logical partition) which is basically an allocated set of CPU and memory resources. Linux can also run under IBM's powerful VM operating system, in which case VM manages anywhere from one Linux instance to several thousand such instances and each Linux instance "thinks" it has an entire multiprocessor machine to itself. Because the S/390 architecture includes specific hardware features for virtualizing the machine's resources, there is very little performance penalty (a fraction of a percent, usually) when running one operating system inside another. It is a capability similar to the "virtual 86" mode of Intel's 386 and newer processors, but it is much more sophisticated and (unlike Intel) the virtualization is at the full level of the newest processor generation, not just backward emulation of an older model.
VM and Linux: Room to Grow, Best of Both Worlds
Infocrossing runs their Linux instances under VM, mostly because of the incredible versatility of the VM environment. LPARs allow only a few Linux instances to run on each physical mainframe, but with VM the number of instances is limited only by the loading of the overall system. More instances on one machine means more opportunity for cost savings due to load averaging between instances. Tom Laudati says that economics kept Infocrossing from using IBM's Integrating Facility for Linux (IFL) processors in their dedicated mainframe. "I don't know of anyone who's running native Linux on a mainframe," he says. "The problem with IFL, when we talk with IBM about it, is that they are still going to charge you for another engine that is VM or OS/390 or VSE based." The IFL processors are much cheaper than full S/390 processors and run at the same speed, but they are limited only to running Linux. IBM still requires that the first CPU node in each machine be one of the full-priced S/390 modules. That drives up the entry point price of mainframe hardware for new customers. The advantage, though, is that once they spend this extra money they have a system that can run VM and support thousands of Linux instances instead of just a dozen or so. In the Infocrossing deployment, their existing S/390 machines already had that first processor node and the VM operating system installed but the new dedicated machine did not.
Infocrossing has Linux on three mainframes, in a mix of dedicated and shared configurations. Says Laudati, "Right now the mainframes are located in Leonia, New Jersey. There are three on the floor right now and another on order. One is dedicated right now to Linux under VM. On the one dedicated machine we have now, that's a Multiprise 3000. We have about 25 Linux images running with no significant overhead." Laudati claims the Multiprise 3000, one of IBM's smaller mainframe models, is nowhere near its performance limit and that there is plenty of room to grow.
When asked about performance concerns of the S/390 hardware relative to RISC or Intel racks, Laudati concedes that S/390 is not the right choice for every application. "Anything that is highly CPU intensive, according to IBM, is not a good fit for the S/390. This would be things that do a lot of math calculations. The business applications are typically more I/O bound than CPU bound."
Workforce availability was definitely a factor driving Infocrossing to the Linux operating system, says Laudati, while access to customers' legacy data was a factor keeping them loyal to the S/390 hardware. Linux for S/390 gives Infocrossing's customers the benefits of both worlds. Says Laudati, "If you look at all of the world's data, the predominant amount is stored on a mainframe somewhere. You come along with IP [protocol standards] and all these web technologies to access that data -- that's middleware -- and it has taken off slowly." Laudati feels that the ability to put Linux (which he views as an Internet tool) onto the mainframe itself will accelerate this webification of mainframe data. He adds that his company has observed a fundamental shift in the skill set of college graduates entering the workforce. "You look at the young blood coming out of college now, and they're going to be the buyers of IT in the next five or ten years. And that young blood knows Linux and UNIX."
According to Laudati, this merging of what he calls "IT and IP", meaning business data processing and Internet tools and standards, brings together the best of both. He says he has no doubt IBM is serious about taking advantage of the opportunity, even if it means they have to play by a new set of economic rules in an Open Source world. "I think it makes a lot of sense for them, though it's almost unprecedented that IBM would give up the proprietary rights to their operating system.... It's taking advantage of the horsepower of the S/390, and the technologies that drive the Internet."
Laudati has seen a shift in IBM's attitude toward customers in the past few years as well. "IBM was very inflexible [in the past]. But now IBM is at least as competitive, if not more competitive, than other companies to make a deal with you. They'll work with you to make a deal. It's a much more flexible company over the last four or five years or so." He feels that IBM has learned from its mistakes, from partners, and from competitors. "EMC [a high-end mass storage company] kind of showed IBM the way of being flexible, is my impression. EMC had things like step leases and other things that helped the customer." Step lease involves lower payments the first few months, allowing ramp-up of businesses that are expanding but can't afford the larger lease right away.
With Infocrossing's emphasis on services such as managed hosting, and IBM's increasing reliance on its Global Services consulting and support as a revenue stream (especially in an Open Source environment where software licenses are zero-revenue), it is natural to wonder whether Infocrossing is IBM's partner or its rival. Laudati says they are "probably a little bit of both", adding, "For the most part they're not competing with us." IBM Global Services, he says, focus their efforts on sites where "the MIPS are already on the floor" and the customer wants to add Linux, whereas Infocrossing focuses on bringing new business to the mainframe. When the customer has a mainframe already, they are likely to just add Linux and only a processor module or two.
Linux in a Hybrid Environment: Security Gets a Boost from the VM/Linux Duo
Chief Technical Officer Robert Graham says that while Infocrossing's preference is Linux on S/390, their management environment is by necessity a hybrid of several platforms and many technical components. IBM's SecureWay and Tivoli products underlie much of this infrastructure, and Microsoft's Internet Information Server (IIS) is actually being used as the web server for host administration. Graham describes a typical ASP administrative user's access to the system. "A customer connects to a component of SecureWay called Webseal, using HTTPS, and it authenticates you. For now we're just assigning userid and password security, but it flows over a secured channel. We will be using something like SecurID [in the future]. Webseal effectively acts as a proxy and extends each [HTTP] request header to include the userid and some other parameters. When you are in the IIS environment, you can count on every header containing a trusted user authentication. Webseal also contains some ACLs that can keep users from reaching certain URLs. In our case, we give each customer an acronym and they can only access the pages that are associated with their acronym."
"If the system management product is something that is not web friendly, we use a product called nFuse which is an integration product from Citrix that provides connections between the web environment and the Citrix environment. It allows you to launch a Citrix application from the web and returns a single-use token that enables access." Examples include allowing customers access to Netview. They run Netview's fat Java client under Citrix and let the customer access it using a Citrix protocol called ICA which runs as a plug-in to the browser. "It's not completely seamless," says Graham. "You can tell that you are transitioning from the web over to the Citrix environment, but you do get full functionality." Some of the decision support tools such as Service Desk, Master Cell browser (an event viewer from IT Masters), and Tivoli Decision Support rely on the Citrix environment.
Graham says that while the overall performance of the Tivoli and SecureWay products has been good, Infocrossing has broken some new ground and, in the process, occasionally broken IBM's middleware as well. "We may have been the first one to open up a second network interface card on the fly. SecureWay kind of committed suicide, and we had to report a bug. IBM fixed it; they're good about that."
Though Windows and the related Citrix products are used where they are needed, both Graham and Laudati said Windows is not Infocrossing's preferred platform. They continue to support it as customers demand, but they use Linux on S/390 or on Intel whenever possible. Says Laudati, "We have... several Linux environments running on Intel systems. Why don't we port that to S/390? Because some of the software isn't available there yet." He adds, "Not all of the Tivoli products are available yet on Linux for S/390, so we have to use some other products there. We are committed to migrate back to the Tivoli products as they become available."
When asked about Microsoft's Windows 2000 Data Center Edition, Laudati took a neutral tone, saying, "We don't have enough information to make a solid evaluation of Data Center Edition yet." He also mentioned that this product is very new in the marketplace, without a proven track record, but stopped short of ruling it out entirely for the future.
Robert Graham was spoke very directly to the issue of security concerns for enterprise customers in Microsoft environments. "Who wouldn't be worried about Microsoft?" he asked rhetorically. "We have an alliance with Foundstone, a security consulting firm. They have a neat demo where they come in and show you how quickly they can break into an NT box. Because Microsoft is so prevalent, it's a big target for hackers." Infocrossing has a three-level firewall setup, and it is not running on NT.
Tom Laudati seemed to trust Linux security for the most part, but he says that the VM environment makes Linux even more secure because VM's security is layered on top of the virtual machines that support each customer's Linux instances. Infocrossing allows its customers to remotely access the VM hosts, but each customer only has access to its own Linux instance. Each Linux virtual machine runs under its own VM login session.
IBM and S/390 Linux advocates claim cost savings for the S/390 Linux environment over rack-mount Intel or RISC servers, and these cost savings increase as more applications are added to a single physical mainframe. So what are the numbers like in the real world, and how big does a company need to be to be a customer of Infocrossing? Tom Laudati answers, "We gear ourselves to the larger customers. But we are pretty competitive against Intel box hosting." He says the entry point for managed hosting on a Linux for S/390 virtual server is about $2000 per month. Limited managed services are included, but not the full spectrum offered by the company. As for the cost savings versus racks of Intel hardware, Laudati says the numbers aren't in yet. "We're just starting to capture that information. Because we already had System/390 sitting on the floor, there was no additional hardware cost."
Tom Laudati is optimistic about the future of Linux on the S/390 architecture, including IBM's new zSeries mainframes. "Mainframe Linux is a pretty hot growth area," he says, adding, "If Linux is not the hottest mainframe growth area, it's right up there. It bridges the gap between IT and IP technologies." IBM, as well as mainframe mavens everywhere, are hoping Laudati's prediction comes true.
Resources:
Infocrossing home page
http://www.infocrossing.com/
IBM Linux for S/390 home page
http://www.ibm.com/servers/eserver/zseries/os/linux/
Application Service Provider sites on Internet.com
http://www.internet.com/sections/asp.html