(LinuxWorld) — I once had a professor who claimed he decided to teach computer science because it was the only way he could get to play with the toys without having to deal with users. He's dead now, but every time I find myself in a preliminary project-specification meeting with the local management's nominated users, I get closer to agreeing with him. That's certainly what happened last week when I met with several groups of Nichievo's users, mainly from the sales and analyst communities.

On the positive side, I did get to wander around a couple of working offices and had a chance to meet with several customers. That got me a few nuggets that make the technology choice easier. But the rest of it? What a waste of time.

The sad fact is that early-stage requirements meetings — professionally known as Joint Application Design (JAD) sessions — often waste everyone's time and get projects started off in the wrong direction. You frequently get the wrong people, and the few attendees who contribute to completing the organization's work often let their perceptions of you — and their concerns about how others at the meeting see them — shape their comments to the point of obliterating the value.

At worst, you get stuff that's filtered through technology perceptions; prescriptions for changing the job to fit the person's chosen technology. At best, what you get are self-deceiving prophecies; descriptions of what the people think they ought to be doing.

A self-deceiving prophecy?

You get a self-deceiving prophecy when, during a meeting with their colleagues and bosses present, someone describes how he or she does work. When you observe them at work, they'll try to mold what they do to what they said, and what they said typically describes what they think they ought to be doing (and not what they really do).

For example, I once did a requirements review as part of an audit on a $40 million child-welfare-systems development effort that had gone off the rails. The documentation was precise and surprisingly clear. The JAD session records showed a well-defined workflow with clear opportunities for information support, precisely as implemented.

Workflow diagrams showed caseworkers coming in at about 8:00 A.M. (the official start time is 8:15), reviewing files, verifying appointments, checking out departmental vehicles and heading out, departmental laptops in hand, to make their visits. According to the documentation, they'd start to straggle back around 3:00 P.M. to prepare their reports, update files, and go home around 5:00 P.M. (the official end-of-day is 4:30).

Reality was nothing like this, something that became obvious when I noticed that the departmental parking garage was empty at night. Most case workers worked a split shift, coming in mid-morning to talk to colleagues, file reports and visit homes during the late afternoon and early evening.

Their information needs didn't resemble what they told the interviewers either. What had happened was simple. These were outgoing, socially empathic people who had given the interviewers exactly what they wanted: clearly defined, factually oriented specifications that unfortunately bore only an idealized resemblance to the relationship information they really worked with.

Their primary question before going on a home visit didn't have anything to do with the mother's record or where the kid got sent to school. It boiled down to: has she got a new boyfriend whose presence means I should take a cop along? They didn't want laptops stuffed with forms or wireless access to service agency websites, they wanted handhelds with integrated phones and a panic button that let them check out the new boyfriend's criminal and mental-health records.

Meeting mayhem

What's driving the wastefulness of the meetings is the absence of a sense of crisis to make local management care about the project. Without that, they tend to assign their losers to the requirements team simply because these are the people most easily spared from the real work. In white-collar environments, that means you often get the people who've long since stopped doing their jobs and become high-priced Windows support people instead. Unfortunately, these people not only don't know the job anymore, but they're perfectly willing to waste the whole meeting trying to get you to acknowledge their positions as local computer gurus.

Some of the things they say are just appalling. One solemnly intoned that "the solution needs to be Web-based, that would be the ideal" and got nods all around... although neither he nor anyone else could tell me what that meant. Another wanted to share his deep concern that using Linux would make the firm look unprofessional. A third carefully shunted me outside during a break to tell me that I could just download the entire solution from xmethods.org (it's a dot.hype site pushing SOAP).

Nevertheless, senior management has been trained to believe that requirements meetings are a necessary component of the process, so I go. But I spend as much time as possible wandering around annoying people who don't go to the meetings. What I'm looking for are the behavioral patterns that constitute the way an organization gets its work done. Then I can figure out what the information-support needs are and build a prototype for people to respond to. This works, because people generally don't know what they need, but they can tell you with certainty when you get it wrong.

For example, I didn't find out what the real primary requirement was by asking people in meetings. I found out by asking to go along on a few sales calls so I could understand the overall process better.

That was a mistake. First, the people making sales calls don't see themselves as making sales calls ("We're not underwriters!"). Second, they didn't think the customers wanted to see a strange face. Many of Nichievo's customers are deeply concerned about confidentiality; for them, Nichievo is a trump card they keep hidden from the people they deal with.

The customer's eye view

Luckily, there are always a few special relationships. I got a look at the customer's view of this thing and learned something important right away: most of the larger organizations have traditional EDI setups, and they're years from moving to Web-based e-commerce.

Even more interesting was that, during our travels, the analysts and executives involved started to talk about processes and control issues but ended up expressing their fears about the security and reliability of the system. In some cases, I wondered if their expression of fear, or my perception of it, was actually being inspired by the cab ride. Overall, their focus on the potential for fraud and/or business interruption seemed real enough.

As they see it, computerization removes paper and duplication from the process, thereby making the firm increasingly dependent on electronic records. If those go away or get compromised, the firm is in deep trouble.

In many cases, the concern is overrated. It's simply not true that the average teenage nephew can hack into DOD systems, but whatever the reality of the threat, many of Nichievo's users are deeply worried by the risks implicit in this system. Fraud is the least of their worries, as one put it: "So we get ripped off. Well, it happens... it doesn't mean the firm goes down." Even having the system scramble records or go dead for a few days would be survivable. Publication of confidential customer records wouldn't be; that would threaten not only the firm, but also the relationships its people have with each other, with customers and within their professions.

From a requirements perspective, the bottom line on this is simple: if I can't get the risk of information leakage down to the point of being negligible, this project should not proceed.

Right up front, that's the kiss of death for the .Net approach. I may or may not be able to proceduralize security using a central Linux server and Cocoon, but I can guarantee you that it can't be done using a Windows server with Microsoft tools.

SOAP insecurity

There's a simple and direct reason for this that transcends my usual cynicism about products that have 50 million lines of unrefereed code: SOAP. The Simple Object Access Protocol is nominally just a message envelope, a way of encapsulating and forwarding information. Couple it with an HTTP binding, however, and you have a remote-procedure call (RPC) tool for bypassing firewalls and internal controls. This is great for putting spam directly on a user's desktop, and it's not too shabby as a replacement for cookies either. However, my guess is that it really shines brightest as a tool for getting network-access information from unsuspecting users.

Combine this with the stateless nature of an HTTP connection, and Web services become obviously and irremediably insecure. You can, I think, take palliative actions such as:

1. Using Microsoft's Windows Message Queuing instead of an HTTP connection to get logging and transaction continuity, but this means building and supporting a Windows client to be imposed on customers.

2. Using IBM's MQSeries instead would avoid part of this, but using it would probably push the project scale up a notch or two too far.

3. You could try adding a SOAP content-analyzer to the firewall box, because SOAP messages are just text strings. But the failure rate of anti-spam filters suggests that this may be a lot harder than it sounds.

4. If you have trustworthy, hard-wired machine i.d.'s to work with (Palladium, for example), you could register correspondent machines and control everything that way. This may be the way of the future, but it just isn't practical yet.

5. You could also try to strip all the stuff needed for SOAP to work off the box. That's how you beat the issue on the Linux/Cocoon server: just say no at load time. But this may only barely be possible on Windows 2000 Server.

A recent article on theregister.co.uk reports on a paper ostensibly written by Microsoft people involved with switching Hotmail from FreeBSD to Windows 2000. The paper was purportedly liberated from an insecure Microsoft server. I don't know if the document is authentic or not, but it contains an absolutely wonderful paragraph in the "Advantages of Unix" section:

Image size. The team was unable to reduce the size of the image below 900MB; Windows contains many complex relationships between pieces, and the team was not able to determine with safety how much could be left out of the image. Although disk space on each server was not an issue, the time taken to image thousands of servers across the internal network was significant. By comparison, the equivalent FreeBSD image size is a few tens of MB.

Right, exactly. With Linux and the Apache/Cocoon toolset, I can certainly sidestep SOAP on the server and implement simple safeguards against someone accidentally adding the components. With Windows 2000 Server, someone may be able to disable it, but that will last only until the next systems administrator comes along to install the latest patches or new OS release. Of course, this is version control run backwards; I'm predicting future exposure to a click-and-drool sysadmin, but automated updates across rackmounts full of little boxes makes that a gimme.

This is a problem with all projects that push the envelope: if it doesn't end up in the mainstream, some future sysadmin or local IT manager will silently cause it to fail and later describe that failure as your fault.

For now, however, the bottom line is simple: it's no SOAP. Using Windows for data this sensitive would be utterly irresponsible.

That doesn't mean using a Linux server makes sense either. It may be that the risks involved outweigh the benefits no matter what, unless we dump the whole project back into a traditional EDI framework based on the X.4xx standards. That's classic security through obscurity, but it would actually appeal to a lot of customers and is something I've asked senior management to consider.

Server-side security the Linux/Cocoon way

Just how secure and reliable can I make a Linux/Cocoon setup? I can do pretty well on the server side:

1. Put in two systems, both physically secured but in separate Nichievo offices
2. Set up parallel but independent management for each machine
3. Automate the process of keeping the two databases in sync
4. Use routing to automate fail over
5. Spend a few extra dollars for higher reliability gear and RAID 0+1 (mirroring striped disk arrays) at each end
6. Use the Cocoon-authentication framework under Tomcat/Jakarta
7. Log everything
8. Impose stringent verification processes before issuing logins
9. Institute a series of random failure drills to ensure that the administrators practice the steps needed for recovery from system or network failure of all shapes and sizes.

What I can't do, however, is affect security at the client end inside the firm or among customers. As far as I know, the best I can do is to verify the integrity of the access process and use end-to-end encryption to render most interceptions harmless. If someone who is authorized to access the system abuses the information, whether it's a customer or internal user, there's nothing I can do about it.

Part of what I have to be concerned about here is the database-replication strategy. Leave that open to someone who can spoof a router to run a man-in-the-middle attack (in which both databases think they're connected to each other, but they're really both connected to the a third party system), and everything else is a waste of effort.

PostgresSQL doesn't yet have replication, but there's a serious effort under way (see http://gborg.postgresql.org/), and they're considering using encryption at the database level. That'd be nice, but it's not ready yet. It may be ready by deployment time — a lot happens over six months in the open-source community — but I can't count on it. I could wimp out and put Solaris on that server so I can use interprocess PKI, but that's not trivial to do. Besides, I want this to be a Linux solution.

One way to fake it is to hang those two servers on their own T1 links to a large ISP and then use router-based packet encryption. That's expensive but effective, and it offers the side benefit that I can detect a man-in-the-middle attack easily by monitoring packet-travel time.

From a commercial-software perspective, the slickest replication service is offered by Informix, but I'm uncomfortable about their future with IBM. Oracle and DB2 have good solutions too, but they're expensive and too often invite DBA-parameter twiddling of the kind that eventually brings systems to a stop.

So it comes down to Sybase, PostgresSQL and mySQL. The latter doesn't have replication, but there are apparently some PERL tools around to do the job. Of course, that might have to be done with PostgresSQL too.

On the other hand, mySQL doesn't have internal stored-procedure capability. That's critical in the client-server world, because the database engine is often the only piece of multi-user software in the system. By default, the place-transaction serialization is handled. However, that's a non-issue here. I can safely externalize procedures and leave mySQL to do nothing more than store and retrieve data, which is something it's quite good at.

This is an argument for using mySQL. After all, what isn't there can't be subverted. I know that's paranoia talking, but I've been paranoid ever since I traced an information leak to a guy who had set up logging on his boss's PostScript printer and was happily downloading everything the boss's secretary printed for him.

This is not an obvious decision, but next week, when I attempt a working prototype, I'm going to use PostgreSQL rather than Sybase or mySQL. But I'm not using any stored procedures or placing any bets on PostgreSQL getting replication services. Come deployment time, we'll revisit this.

In the meantime, I'm going to see what it would take to break mySQL as the likely long-term choice.

Feedback so far

So far, I've not received much feedback on the key business issues in the Cocoon articles. The next article discusses the active prototype, and boy do I need feedback before writing that!

• The pros and cons of business-app implemention via open-source software (Part 1)
• Nichievo corporate background
• Open-source vs. Microsoft tools for business-app implementation, Part 2

• Report: Open Source and Linux: 2002 Poster Children for Security
• The Unix Guide to Defenestration
• Understanding Apache Cocoon