Open Source in a Sales Management System
NOTE: This short study is a sketch that hardly does justice to the scale of the project — but for commercial reasons most of the details have had to be omitted.
We've long been believers that Open Source products outperform most of their closed-source competitors, even when applied to commercial line-of-business applications. In line-of-business applications, reliability and robustness are the watchword and the systems have to work. Our experience of key components such as Apache and Perl lead us to believe that they are better and safer bets than many of the allegedly commercial alternatives.
We were approached by Bryant Homes in mid 1997 to discuss the possibilities of building a distributed sales tracking and processing system. Bryants are one of the leading UK house builders (here is their website) and like most house builders they have a particular problem: their shop front is mobile and most of their customers come to where they are building the houses. A building site will be active for a certain period before all the plots are sold and the "shop" then is itself sold (the show house): at that point the sales operation is closed down for that site.
A central database is used to track the sales operation, located at their head office in Solihull, West Midlands. It's not realistic to put in expensive fixed lines to temporary offices like the show homes, but obviously it's very important to be able to automate the workflow of the sales process. In fact it's particularly important, because many of the staff who operate the process work part time and need support in managing the continuity of the relationship with their customers.
In mid-97 it was clear that something could probably be done using web technology and making use of a browser as a front-end. Using standard components like those, dial-up access back to head office for the data transactions and some intelligent use of local storage on the sales PCs (images of houses, plans, non transaction-related data) a highly cost effective solution could be built. A crucial issue was to keep the connectivity costs down: initial figures showed that a year's phone bill would dwarf the development budget if all of the sites maintained 8-hour connections. There are a lot of sites out there!
Fairly rapidly an architecture evolved. This general architecture is very well understood nowadays, but remember we were working way back and were treading a much less smoothly-worn path.
The Head Office Database is a large Informix database running on HP 9000 systems. A separate system runs Apache on HP/UX, where the queries to the database are fielded by a FastCGI process which handles those of the CGI scripts that are written in Informix embedded SQL. Fast CGI was necessary because of the delays introduced by connecting to the Informix database; for some reason this operation consumes a lot of time. Fast CGI allows a permanent connection to be maintained and shaves seconds off the creation of each page.
The Informix SQL queries were deliberately kept simple and were based on the analysis of the transaction needs of the system as a whole. They provide a data channel only — the work of making individual queries for specific pages and formatting the results is done by extensive Perl scripts running in Apache's mod_perl service.
A lot of development work went into creating the "build tools" as they are known; these test that the production Perl scripts are syntactically correct and validate the parameters that pass between the Perl and the SQL scripts. The build tools consumed a significant amount of early time in the project but proved their worth when it came to building the deliverable software later.
The on-site systems use Netscape running on Windows. A local webserver and CGI scripts are installed on these systems and automatic uploads of data to the PCs are controlled from the Head Office systems using opportunistic "push"; when the on-site system connects via ISDN, stale data is refreshed from Head Office. Many on-site queries can be resolved by referring to local data (site plans, images of houses and numerous others), so the connection to Head Office is not kept permanently open. From time to time key transactions do occur on the on-site system and these are pushed over to the Head Office systems, causing a connection to be made.
Connections from site to Head Office are via ISDN line, received by a standard Access Port / RADIUS server combination as found in any number of ISPs.
An important part of the design was to store no state at the remote end. Building sites are notoriously prone to theft and the PC systems installed there need to be as hot-swappable as possible. The design successfully achieved the goal apart from the issue of the remote IP address, which is used for validation of incoming calls at the Head Office end.
Three separate servers are used in the development system, listening on different TCP ports. The standard production server listens on port 80; others on port 81 and 82 are used for pre-delivery testing and development in parallel. The software configuration system (based on RCS) is aware of this and tools are used to automate building the various deliverables as well as performing regression testing.
In general the Open Source tools have been absolutely problem-free. The notable exception was the use of Hypermail as the tool to convert inter-site email archives into web-browseable archives. Considering the pain and grief that Hypermail caused us, it would probably have been quicker to have reimplemented it from scratch rather than try to make it work on the scale and volume of messages that it was asked to handle. Several tense project management "moments" were the direct result.
That apart, the project has been a success. It went into production in 1998 and has proved popular with its users from the beginning. Development has continued since then, with increasing demands for further functionality. It's hard in a short study like this to give a feel for the scale of the product — this is not a little "demonstrator" project, it is a full-scale mission-critical application that has had a direct impact on the business processes of its user.