A proxy approach to e-mail security

Ian Brown and C. R. Snow

Published in Software - Practice and Experience, 29(12) 1049-1060, October 1999.

Introduction

A wide variety of electronic mail software is used to send messages across the Internet. Two principal protocols – Simple Mail Transfer Protocol (SMTP) and Post Office Protocol (POP) – are used to allow mail clients and servers running in extremely heterogeneous environments to communicate with each other and thus send and receive e-mail.

The one disadvantage of this heterogeneity is the amount of effort required to add functionality to all these different pieces of software (and their frequent upgrades), particularly with complex services such as security. Such services are only valuable when implemented in the majority of applications.

An alternative approach is to examine the protocols and data flow involved in communication between the mail clients and servers. By modifying data as it passes between the two, a proxy can provide functionality to any application that uses those protocols without requiring them to be separately upgraded. We describe the use of this approach to secure electronic mail between sending and receiving hosts.

Encryption and Digital Signatures

Few Internet users realise how vulnerable e-mail is to forgery and interception. Envelopes protect letters; but their digital equivalent, encryption, is rarely used. Electronic messages can be automatically scanned for items of interest to an eavesdropper. It has been widely reported that national intelligence agencies are already performing such searches on a large scale [1, 2, 3, 4, 5].

Encryption has a very long history in military and diplomatic activity. It allows messages to be scrambled so they are only readable by the intended recipient. It can also digitally sign messages so the recipient can be sure of the sender and that the contents have not been altered in transit.

Standard cryptographic techniques use a secret code or key to encrypt a message. Only someone also in possession of the key can then decrypt the message and discover its contents. But how do two parties share a secret key in the first place?

Public key cryptography, invented secretly by Britain’s Government Communications Headquarters [6] and openly by Diffie and Hellman [7], solved this problem. Users create two related keys; one is made public, the other kept very private. Data encrypted with the public key can only be decrypted with the private key. Data can also be signed using a private key. Any other user can check the signature with the associated public key. Users publicise public keys using Web pages, directories and many other methods.

Governments have proved reluctant to allow their citizens the protection encryption affords. Even after cryptographic systems were developed in the open literature, bureaucratic obstruction prevented their deployment. Only the persistence of Phil Zimmerman, in writing and distributing his free PGP encryption software, has allowed any widespread use of these techniques – although still on tiny fractions of Internet mail. As Zimmerman points out, "it would be nice if everyone routinely used encryption for all their e-mail, innocent or not, so that no one drew suspicion by asserting their e-mail privacy with encryption." [8]

In order to reach this position, however, encryption of e-mail must be simple to achieve. The average e-mail user will simply ignore complex or unwieldy encryption facilities.

The Current Situation

PGP was originally written for use with a command line interface. Early versions required users to place their message text in a file, encrypt the file using PGP, and then send the message with the encrypted file as its body. This is clearly highly unsatisfactory for users of window-based mail agents such as Eudora. As a result, a number of window-based front ends appeared.

All of these systems act as "plug-ins" to various mail programs. They provide additional menus and facilities, and require the user to take positive steps to encrypt or sign an outgoing message. Similarly, if an encrypted and/or signed message arrives, the user must take explicit action to decrypt and/or verify the origin of the message. We feel strongly that as long as the average user is required to take additional actions to send a message securely, then they will rarely take those additional steps on the grounds that "the message is not that important, anyway". With such a universally held attitude, the situation will never be reached in which e-mail users are routinely encrypting their messages, with the consequence that all encrypted messages will continue to be regarded with suspicion.

Netscape and Microsoft mail clients have recently included Secure MIME [9] support, with this kind of automatic processing of messages. Due to United States export regulations, only versions with restricted key lengths can be exported. The keys used are so short that it is feasible for messages to be decrypted using the "brute-force" method of trying all possible keys.

True GUI versions of PGP do exist, which allow more automatic working with some mail programs. On the whole, however, the level of user interaction still required to encrypt messages might still be regarded as unacceptably high.

The approach we have taken hides the encryption, decryption, signing and verifying functions. They are used automatically by default, with no user intervention required. By working at the mail handling system level, they work with all mail programs that use POP and SMTP. They will not yet work with systems that use other protocols, such as the Microsoft Mail Post Office, or UNIX systems that write incoming messages into a file that is read directly by a mail agent. However, even in such systems as these, there is a point at which standard protocols are used, and interception could take place at such boundaries.

Architecture

We have chosen to call the secure mail system Enigma, after the World War II German encryption machine.

We assume that an Enigma user will generally be using a personal workstation, and running one of a number of possible mail clients; for instance Eudora or Netscape Communicator. Under normal circumstances, the client will communicate with a different server machine. The communication uses standard mail protocols to send and receive mail. Typically, SMTP [10] is used for sending mail and POP3 [11] for receiving mail.

Enigma achieves its goals by inserting an additional program, or proxy, between the mail client and server, appearing to the mail client as the server and communicating with the real server as though it were the mail client. There are two parts to the program; one to act as an SMTP server for outgoing mail, and another to act as a POP3 server for incoming messages.

Much of the information needed for processing secure incoming mail is contained in the message itself. This, along with the user’s preferences, for which we have tried to define sensible defaults, makes the cryptographic procedures automatic. The only situation in which user intervention may be required is with the handling of the user’s private key, and where a required public key is not available.

A typical mail agent communicates with its server by sending and receiving its messages in cleartext form, susceptible to eavesdropping. It is therefore essential that messages are encrypted before they leave the secure domain where the client is running, usually the same machine. Thus the messages will be encrypted as they leave, and decrypted after they arrive at, the user’s system.

The system configuration is shown in Figure 1. The mail client is configured so that outgoing mail is sent to the local host on the standard port for SMTP service and receives mail from the local machine on the POP3 standard port. The encryption program, which deals with outgoing mail, then connects to the system running the real SMTP server, while the decryption program connects to the system running the real POP3 server.

We are assuming that the user’s workstation represents a secure environment (represented by the outer dotted box in figure 1). If there are any concerns regarding the security of the messages after they been decrypted by Enigma and placed in the user’s mailbox, then these can be addressed by storing the mailbox in an encrypted form. Enigma addresses the problem of communications security, and regards encryption of already received messages as a problem in storage security. This distinction is an important one, and usually has implications in the area of key management [12].

Figure 1

More complex configurations can also be used. For example, some UNIX hosts run a local mail transport agent that then connects to an organisational mail relay. Enigma could be inserted between the mail client and local MTA, or between the local and remote MTA. It may not be possible to use Enigma in the former case, because the Mail User Agent does not necessarily use a protocol such as SMTP to communicate with the local mailboxes. However, provided the local and remote MTAs interact with each other using SMTP, Enigma could act as a proxy for each in the same way as it does in the single user workstation situation. We illustrate this is Figure 2. From the security point of view, this solution is less than satisfactory, since mail sent and received by such a system would typically not be transferred while the user remains at her terminal. This has serious implications with regard to the handling of private keys, as we discuss later.

While this solution is sufficient to ensure that mail is encrypted and verified by signature as it passes across the network, the security issues within the user’s host give rise to serious concern about the value of Enigma in this context.

Figure 2

Operation

To encrypt a message, the public key of all intended recipients is required. Message recipients are enumerated in commands in the SMTP protocol. Enigma identifies those recipients, and tries to find public keys for them all. The program attempts to find public keys by looking in the local public key database and interrogating a public key directory.

A user must identify herself as part of the SMTP protocol, so signing of outgoing messages can also be automatic after obtaining the user’s private key. As protection against theft, this key is generally (although not necessarily) stored in encrypted form, protected by a user-chosen passphrase. When a passphrase is required, Enigma presents a dialog box to request in from the user (see figure 3). Having to enter a passphrase every time an encrypted message is received is tedious, and encourages users to choose short – and thus less secure – passphrases. Our system will cache passphrases for a user-specified time. This allows each user to weigh the risk of a cached passphrase being captured by an attacker against the convenience of not having to continually re-enter it.

Clearly, Enigma is at its most effective when used on a physically secure workstation, and the (unsatisfactory) shared host situation described earlier requires that not only is the private key stored unencrypted in the memory of the machine, but that the private key must also be stored, unprotected by the passphrase, in the file store accessible to Enigma. Mechanisms do exist for the secure transfer of a password or passphrase over an insecure network [13, 14], and such methods could be used to allow the stored version of the private key to remain encrypted. The "APOP" option of the POP3 protocol could also be used to achieve this.

Figure 3

The default situation is that all messages sent will be signed using the sender’s private key, and will be encrypted using the recipients’ public keys. A warning will be issued on any attempt to send a message to a recipient for whom no public key is available. Users may then choose whether the message should be sent unencrypted. There is little point in sending the message encrypted to those users with public keys available but as plaintext to others; attackers will have little trouble finding one of the plaintext versions. Mail clients occasionally interact with a user while a message is being sent, for instance to request a correction if one of the recipients is not a valid e-mail address. Enigma’s requests should be seen by users in the same light.

PGP messages have the property of being self-identifying, in the sense that the body of received messages indicate whether they are signed and/or encrypted. Encrypted messages are decrypted with the user’s private key. If the required private key is not available, either because the user does not possess the key, or chooses not to enter her passphrase, the ciphertext is passed through unaltered to the mail client. If a signature is present and the sender’s public key is available, the signature is checked.

Enigma marks signed messages by inserting a text notification into the body of the message (see figure 4). The name of the user associated with the signature key rather than the contents of the unauthenticated From: field is displayed. If this text is already present in a message, it is removed to prevent an unsigned message appearing to be signed. Full integration with a mail program would allow a slightly better user interface. In particular, non-text parts of a message could be shown as signed. We feel that this small advantage is outweighed by the effort required to provide and maintain such integration in a large number of mail programs.

Figure 4

The message ciphertext can optionally be placed at the end of each decrypted message. This may be useful for signed messages where the recipient needs to be able to prove in future that the message really was signed, for instance if it was a contract. Later versions of Enigma will allow the signature to be extracted and attached as a separate message bodypart, providing a more elegant solution to this problem.

(The user must also consider the matter of trust in relation to the public key. It is possible for anyone to create a key pair and publish the public key as belonging to someone else. However, questions of the trust that can be placed in a public key, though interesting, are the subject of a different discussion, and will not be pursued here.)

Client/Server Interaction

We now consider the interactions that take place between a typical mail client and the servers with which it exchanges mail. As previously stated, we assume that the mail agent will send mail directly via SMTP, and receive mail using the POP3 protocol.

We shall examine the commands and responses that pass between client and server, and describe how these would be modified, if at all, by the presence of Enigma. As with all client/server interactions, the client will generally initiate a conversation by sending commands to the server and the server will respond to those commands.

SMTP

There are certain SMTP commands in which Enigma has no interest, and these are simply passed through to the real SMTP server. One command (RSET) is intended to reset the connection between the client and the server, which requires Enigma to reset a small amount of state information.

The STMP commands that interest Enigma are as follows.

MAIL FROM: <sender> identifies the sender of the message that follows. The <sender> information is used to locate the private PGP key that will be used to sign the message. If not already cached, the passphrase to allow access to the private key will be requested from the user.

Each message recipient is identified with a RCPT TO: <recipient> command. Having obtained the identity of a recipient, Enigma must then attempt to acquire a public key for that user. This may be in the local public key database, or the program may need to request the key from a directory.

If the system fails to obtain a public key for any one of the recipients, the user can choose between sending the message in the clear to all recipients, sending the message only to those recipients whose public keys can be obtained, or not sending the message at all. This choice is resolved by entries in the "preferences" file or by holding a dialogue with the user.

The DATA command is responsible for the transmission of the message text to the SMTP server. Message headers are sent as cleartext, as there is currently no widely used standard for mailers to automatically decrypt and display them in a message index. The message lines are collected together until the entire message has been received, and then signed and encrypted using the keys already identified. The result is passed on to the SMTP server proper.

There are three further commands in the SMTP protocol that appear to pertain to the sending of messages, but that we have chosen to disregard, and which are in any case very rarely used. The SEND command attempts to write a message directly on to the user’s terminal. Such an action clearly depends on the user being logged on, but in any event, a recipient would probably not appreciate having a quantity of encrypted PGP output deposited on his screen. For the same reason, SOML (try to SEND, but place the message in the user’s mailbox if the SEND fails) and SAML (SEND the message, and place it in the user’s mailbox anyway) have been ignored.

POP3

Enigma may need to decrypt and/or authenticate retrieved messages. As with SMTP, there are a number of commands sent to the POP3 server by the client that do not involve the actual transfer of messages. On the whole, these will be passed unchanged through to the real POP3 server by Enigma. Similarly, when the server replies, responses usually pass unchanged through Enigma directly to the client.

Two commands require action as they retrieve some or all of an incoming message. RETR <msg> returns message msg in the user’s mailbox. If the message is encrypted and the recipient’s private key is available, it will be decrypted. If the message is signed and the signer’s public key is available, the signature will be checked. The message is then passed to the mail client.

TOP<msg> <n> is similar to the RETR command, except that only the first <n> lines of the message body are returned to the user agent. Partial messages cannot be decrypted or authenticated, so when Enigma receives a TOP command, it must convert it into a RETR command for forwarding to the POP3 server, and after processing the response, must deliver only <n> lines of the decrypted message. In all other respects, the action of TOP will be identical to that of RETR.

User/Server Interaction

The interactions between Enigma and the user are controlled by a configuration file. The level of user interaction can be reduced to the absolute minimum by setting suitable values in the configuration. It may, for example, be configured so that the user is asked for the passphrase to release the private key on the first occasion that the key is required (either for signing an outgoing message or decrypting an incoming one), and the decrypted key may then be cached for a user-determined length of time. The present version of the system imposes a maximum of one hour on this period, but in principle (apart from the weakened security aspect) there is no reason why this period should not be indefinitely long. Again this will depend on the perceived security of the user’s secure zone (usually the workstation).

The only remaining circumstance under which Enigma requires an interaction with the user is in the case that Enigma is unable to find (unambiguously!) a public key. The minimal interaction is in this case no interaction at all, since Enigma can be configured to send the message in cleartext form. In general, the user may wish to have slightly more control, to the extent that she might wish to be informed of the absence of a key and to be offered the option of sending the message in cleartext, or of aborting the transmission. Additionally, as we point out later, the user may wish to take manual steps to obtain a missing key.

Key Handling

PGP assumes that users are fully aware of the way keys should be managed. It provides facilities for keys to be created, added to and removed from a local database, to be revoked, and a whole host of other operations. Our aim was to hide as much of this detail from the naïve user as possible; the sophisticated user can use other key management tools since Enigma uses standard PGP key formats. Ideally, naïve users will have their key management looked after by a system administrator, as the process is hard to do automatically. Enigma’s key management facilities are only provided to help non-PGP users get started; its real focus is the automation of day-to-day message processing. The program can entirely co-exist with other PGP-compatible tools.

When first set up, Enigma can generate a key pair for a new user. The user interface asks for the user’s name and e-mail address and some random typing to ensure a good-strength key pair, creating a 1024-bit RSA key pair by default to hide cryptographic jargon. The private key of this pair is encrypted with a user-chosen passphrase before both are saved in a local database. The public key is automatically submitted to a nearby public key server; the servers are linked and replicate new keys between themselves, so that the key will become available in all such servers within a short period of time.

The first time one PGP user communicates with another, she needs to add his public key to her key database. This allows her to encrypt messages and verify the signatures of his replies. She can find his key on his home page or by requesting it from him or a public key server.

Enigma can automatically query a public server. If a key is not available for a message recipient, Enigma will check a user-specified directory and store any retrieved key in a local cache. Users without keys are noted so that Enigma does not waste time searching the public key database next time. Keys and the lack thereof are cached for a user-specified time.

Practicalities and Experience

Enigma is written in Java for portability reasons, and also for the availability of a large number of class libraries – including cryptographic code. It has been tested on Windows platforms and a variety of UNIX systems. We have also demonstrated that it is possible for Enigma to inter-work with other PGP systems, such as the PGP plug-in for the Eudora mail agent.

In a recent paper, Young suggested an approach based on mail proxies that act as interfaces between a mail agent and PGP [15]. There is a clear implication that such proxies are closely related to the mail agents for which they act. In fact, Young states "Updates to mail programs are… frequent… Combined with the fact that there are a lot of mail programs in use, there is clearly a potentially significant requirement for on-going effort. There is clearly no technical way around this…"

Our approach, however, depends much less on the mail agents in use, and much more on the mail protocols employed. Certainly it is the case that mail programs will change, by upgrades or by more fundamental modifications, but we would argue that the mail protocols will remain more constant, and hence less effort is required to adapt Enigma to a changing environment.

An Example

We conclude this section with an example, in which we detail the manner whereby a user employing Enigma will interact with another user using the Eudora mail agent equipped with the PGP 5 plug-in.

There are two modes in which the PGP plug-in for Eudora can be used. The more "automatic" of the two allows the composer of an out-going message to indicate (by enabling either or both of two facilities in the message composition window) that the message is to be encrypted and/or signed. Thus the steps required for a user of the Eudora mail agent equipped with a PGP plug-in to send a signed and encrypted message are:

1. Enable both encryption and signature by clicking the appropriate icons in the message attribute bar.
2. Compose the message as normal.
3. Send the message. This will cause a pop-up window to appear asking the user for a passphrase. (Confusingly, if Eudora is configured to queue the message for later transmission, this dialogue does not appear until the actual transmission is attempted, which may be some time after the message was created and queued for delivery.)
4. If the local public key file is unable to resolve unambiguously the key to be used for each recipient, another pop-up window appears, inviting the user to select the key to be used. Again, this does not occur when the message is created, but is delayed until actual transmission is attempted. If a required key cannot be found, the user has no choice but to abort the transmission.

By contrast, using Enigma would not require step 1, and step 2 would take place in the normal way as for unencrypted mail. Step 3 would take place on the first occasion that an encrypted message was sent, and depending on the settings in the configuration file, would not be required again until the cache version of the private key expires. Step 4 would also be avoided in the case where the user chose not to be informed of the unavailability of a public key.

On input using Eudora, the body of a signed and/or encrypted message consists only of an indication that an attachment has been placed in the user’s "Attachments Folder." The icon displayed indicates whether the message is encrypted or only signed (at this stage, it is not revealed whether or not an encrypted message is also signed). Clicking on the icon will invoke the PGP plug-in, causing the signature to be checked using the public key of the sender (extracted from the local public keyring). The status of the signature is displayed, indicating whether the signature was good, unverified, not checkable, etc. The message is then displayed in the message body window.

If the message is encrypted, the private key passphrase is requested, but this again may be cached to avoid the need to type the passphrase on every incoming encrypted message. If the encrypted message is found to contain a signature, this is checked as described above.

Using Enigma, the fact that the message was encrypted might not be apparent to the recipient. The recipient will be asked for the passphrase for the private key, unless this has already been cached. If it has, the decryption takes place by Enigma without the user even being aware that the messge was encrypted. It is necessary for the message to be modified to indicate the status of the signature, as has already been described, but again this occurs without the user being required to take any action.

Because Enigma and the Eudora PGP plug-in (and indeed PGP itself!) use the same data formats to send encrypted and signed messages, messages from one are happily processed by the other. Enigma would obviously be of little use if it only allowed secure communication with other Enigma users. Instead, it can work with any of PGP’s estimated 100,000+ users.

Conclusions

We believe that the Enigma approach to the problem of providing unobtrusive access to encrypted electronic mail has considerable advantages over other solutions proposed. Its main virtue is that it is independent of the mail agent used, saving the effort needed to add security functionality to many different applications. While new versions of Enigma would be required if new protocols for the sending and receiving of mail were introduced, solutions that utilise plug-ins or front-ends would need to be upgraded every time a new version of the mail agent itself is installed. We would contend that the latter event is likely to be more frequent than the former, and hence the problem of keeping Enigma compatible with its environment (i.e. mail agent and server) is less onerous than the plug-in approach. We hope to extend Enigma to work with other mail protocols such as IMAP (Internet Mail Access Protocol) and also News. We have started implementing a plug-in architecture that centralises code common to all protocols, allowing support for new protocols to be added with minimal effort.

We also claim that the use of Enigma involves the user somewhat less than front-end or plug-in solutions. The savings may be small, but we have argued that the principal reason why users are reluctant to use encryption is that they see the value of using encryption as being disproportionate with the effort required to employ it. Any saving, however small, in inconvenience or effort therefore tilts the balance more towards the universal use of encryption technology.

The performance of the system is entirely acceptable. Using Sun’s Just-In-Time Java compiler (now a standard part of the Java interpreter), the cryptographic processing of each message typically takes around 1 to 2 seconds. This slight delay is usually invisible as part of the general process of sending or retrieving a message through a mail server. Cryptix, the underlying cryptographic library, is now available with native code plug-ins that will speed up these operations even further. Many mail agents anyway send and receive mail as a background task, in which case an extra couple of seconds is almost immaterial to a user.

Although Enigma as it now stands is designed purely for the purpose of providing PGP-type encryption in as unobtrusive a manner as possible, and with total independence from the particular user agent in use, it would be possible to envisage other encryption services provided in the same way. It would for example be a trivial matter to append a secure timestamp to a message as it is processed by Enigma. Similarly, Enigma could deal with automatic archiving of both incoming and outgoing messages.

Automatic archiving can be achieved using the mail agents themselves, or separately by an autonomous archiving program. The messages thus archived would still be stored in plaintext form, and the issue of confidentiality by storage encryption would once again be raised. Clearly, Enigma could be extended to perform an archiving function, and this would allow the encrypted version of the messages to be archived. However, as we have pointed out earlier, there are valid security reasons for treating storage encryption differently from communications encryption, and our concern with the present version of Enigma is to provide communications security.

Other work is starting to use the proxy approach. Zenel [16] has implemented proxies that reduce the bandwidth of data sent over a wireless link. Cabell [17] has designed a general-purpose proxy system that works with e-mail, Web page access and is extensible to any Internet protocol. He has incorporated Enigma into this system.

These and other schemes can be characterised as transformers that modify the properties of a data stream as it passes between domains with different requirements. For Zenel, this is between a high-speed wired network and a low-bandwidth wireless link. Enigma changes the security properties of messages as they pass between the relatively trusted local workstation and the untrusted Internet domains. As the Internet becomes ever more heterogeneous in terms of security, bandwidth and many other requirements, we feel this is an important way to maintain the connectivity that has been the overwhelming factor driving its success.

Acknowledgements

We are very grateful to the referees of earlier versions of the paper whose helpful and constructive comments greatly assisted us in the preparation of this revised submission.

References