Finding China, Crystal, and Tableware With Linux

By: Rob Reilly
Tuesday, October 3, 2006 10:54:25 AM EST
URL: http://www.linuxplanet.com/linuxplanet/reviews/6316/1/

A Humble Start

Replacements, Ltd. has carved a niche market out of locating and selling hard-to-find china, crystal, and silver tableware and collectibles.

Nearly half of the company's seven million retail customers place orders through the call center, The other half do business over the Web. Orders are filled from an 11 million plus item inventory comprised of more than 250,000 patterns. The company also handles current and active patterns, obtained directly from manufacturers.

Computer technology has been a key factor from the very beginning. As you would expect, Linux now plays a major role in keeping the company's data straight.

Bob Page quit his job as an auditor for the state of North Carolina and started the company 25 years ago in his basement and attic. People questioned his decision. Replacements, Ltd. is now the world's largest retailer of discontinued and active dinnerware.

At first, Page tracked customers orders through thousands of 3x5 index cards. In 1984, he took the company high tech with proprietary Data General computers running the AOS/VS operating system, the INFOS database, and applications developed in DG Cobol.

In 1995, they moved to Unix (on the Data General AViiON hardware line) running DGUX. With the switch, they ported the database over to Oracle. DG clustering protected the database and server-based applications, while also providing high availability. Over time, in classic client/server fashion the client GUI eventually ran on Windows machines, developed under PowerBuilder.

2002 marked a Replacements move over to Solaris (on Sunfire hardware) with Veritas (now Symantec) for the server/cluster side.

By 2003, Replacements had started considering Linux. Although SUSE was considered, Red Hat eventually got the nod. The Replacements IT team felt that Red Hat's professional support and annual delivery cycle differentiated it from other vendors. In Replacements high-volume transaction environment, avoiding downtime was an absolute priority.

Jim Meredith, Replacements Manager of Information Technology, said that convincing the CIO that Linux had the performance and stability to run their business took quite a bit of work, but he ultimately became the strongest advocate.

Motivation, Testing, and Implementation

Meredith said it took a solid two years to complete the testing and migration. The first year was just internal testing by himself and Unix Administrator Eric Singer. Selected developers and users were brought in on the second year, to iron out any problems. "The goal was to make the migration from Sun to Linux as easy as possible," he said.

They also wanted commodity machines that were more open, could reduce costs, and permitted manufacturer independent hardware choices. Over the years, Meredith and Singer found that all flavors of proprietary Unix had quirks, so scripts always needed tweaking with an occasional re-writing of code. They wanted to avoid having to make changes for new versions.

Early on they did some beta testing with Veritas clustering and Meredith liked the fact that it ran on both Sun and various combinations of Linux hardware. Sun's clustering solution only ran on Sun hardware. He wasn't excited about locking into one vendor and so chose Veritas (Symantec) to run the cluster.

Oracle provided an evaluation package of Oracle 10g RAC, while HP offered hardware needed for testing. Red Hat offered an evaluation version of RH Advanced Server for a three-month trial.

In 2005, Replacements purchased a set of HP Proliant 585s for a two-node Veritas/Symantec cluster to do server-based applications, reporting, and batch processing. They used four 385s as the database servers for their Oracle 10g RAC cluster. Since HP provided the hardware, they also provided first-tier software support.

During testing, Meredith created lots of spreadsheets to show progress and promote the case for Linux with the CIO and senior management. Documenting the fact that Brio (now Hyperion) SQR (structured query reporting) worked reliably, over a three month period, definitely helped sell the project.

Veritas/Symantec was also chosen because their file system was more mature and worked better than the other solutions. During the 2003 to 2005 time frame, Meredith found that the performance was roughly twice that of Ext2 or Ext3. These two had write performance issues, which in turn caused batch-processing speed problems.

Fixing The Rough Spots

In spite of all the earlier testing, Meredith was pretty nervous shortly after going live on Linux. A problem showed up in printing.

Replacements does a lot of heavy printing jobs, during the course of filling orders and producing reports. It turned out that Cups (the default printing system for modern Linux distributions) rendered a Post Script file for every single print job. This took up to five times longer to print that with the old lp/lpr printing setup on Solaris.

Fears were quickly calmed by Singer, who wrote an interface for Cups that intercepted the job and added the necessary escape sequences, for a particular printer. The reports don't need a lot of Post Script preprocessing, for this business anyway. Changing fonts, pitch, etc. were done much more easily and faster, directly on the printer.

The team ran into another problem with directories having very high file counts. The command line argument list in scripts that worked just fine under Solaris was was too long for Linux, and many scripts generated "arg list too long" errors. This glitch was solved in the Linux/Veritas environment by adding more directories to the application file structure, each of which held fewer files.

Results and Recommendations

Now that the new Linux/Oracle has been in use for more than a year, Meredith commented that it is running "nice and steady."

He also mentioned that the company is pretty conservative with new technology, so for now they are happy with the implementation.

Meredith recommended several things that integrators might keep in mind for their projects.

Pick vendors that stay up on certifications. Some still are not very good at that. This can be a problem for a company because generally, if an application isn't certified on a certain platform, the vendor won't provide support. Symantec seemed to be right on top of certifications. Also, if you run a proprietary application you might have to stay one or two revisions back (older version of Red Hat Linux) for certification.

Not staying up with certifications for revisions could definitely turn Replacements away from a vendor.

"Test as close to reality as you can," was another suggestion. Replacements, Ltd., tested for two years before they felt comfortable putting the systems into production. This isn't unreasonable considering the company absolutely has to process real transactions in real time.

Lastly, Meredith suggested that companies build good relationships with their vendors. A good working relationship with HP and Red Hat allowed Replacements to get the hardware and software needed to simulate the system that they wanted and work out many of the bugs, before flipping the switch.

Who would have thought that a company the finds dinner plates and china for their customers would ever use Linux in the back office?

The fact that Replacements uses Linux for their mission critical systems today, is a reflection of the original pioneering spirit displayed by Mr. Page all those years ago.

With more than seven million customers and growing, the company continues to make great business decisions.

Rob Reilly is a consultant, trend spotter, and writer. He is a contributing editor for Linux Today. He advises clients on portable business computing and presentation technology integration. You can visit his web page at http://home.earthlink.net/~robreilly.