.in 0 .rm 72 .he //Release Notes/ .fo //-#-/ .ce 100 VMS Software Tools Mail System .sp Fall 1983 DECUS Distribution .sp 2 Joseph Sventek Computer Science and Mathematics Department Lawrence Berkeley Laboratory Berkeley, CA 94720 .ce 0 .sp 3 This document is intended to serve several purposes. First, and foremost, it documents the design of the Software Tools Mail System. The information contained herein should prove useful to system management entities, permitting tuning and problem tracing of the system. Finally, it provides the necessary information required to interface the mail delivery system to another network substrate, as well as the information required to interface other mail composition utilities to the delivery system. .sp The Software Tools Mail System prototype was designed and implemented to test out the extensibility of virtual machine concepts to distributed applications. As such, there was little provision made for "bullet-proofing" the software. It is quite remarkable that the system has worked so well in hostile environments, given this lack of explicit naive-user armor. It is a direct result of this success that the software is being distributed. .sp As with all other public-domain software from LBL, there is NO guarantee of performance or support, implied or expressed. Written bug reports, with fixes, are solicited; all reasonable attempts to incorporate such fixes in future releases will be made. Phone calls concerning this software are discouraged, due to the interrupts caused by such communication. Every effort has been made to include adequate documentation such that installation and maintenance will be straightforward. Correspondence about the system should be addressed to me via the following addresses: .nf Joseph Sventek U.S. Mail Mailstop 50B-3238 Lawrence Berkeley Laboratory Berkeley, CA 94720 lblg!mail-requests.j@lbl-csam Arpanet ucbvax!lbl-unix!lblg!mail-requests.j UUCP .fi .bp .ce Design Issues .sp Examination of existing mail systems indicates that the most successful systems (in terms of extensibilty) are those which partition the mail service into two layers: a delivery layer and an user-interface layer. Such an architecture provides for several types of user-interface paradigms, while providing a constant delivery service, independent of the user composition utility used. This partitioning has been strictly adhered to in this work. .sp A major design flaw of many systems is the inability of the delivery system to do anything with the mail other than store it in a file somewhere. This failing is due to a very narrow interpretation of mail delivery. When one user sends a message to another user, the message is actually destined for the receiver's delivery agent. While typically the delivery agent simply appends the message to the user's mailbox file, one should be able to specify an alternate delivery agent which can dispose of the mail in another manner. Such capabilities as message forwarding, auto-reply, news services, etc. are difficult to achieve without this capability. As a result, this capability is provided by the delivery system. .sp Many mail systems fail to observe any standards with respect to addressing, message format or transport protocol between hosts. Due to an absolute requirement for ARPANET compatibility locally, the appropriate DARPA standards are followed for this system. The relevant Request for Comment documents describing these standards are included with the documentation kit. .sp As mail systems become more integrated into computing environments, the adaptability of the software as needs and/or standards change becomes crucial. The application of accepted software engineering techniques (structured design, virtual machine concepts, etc.) to the system design is one way of assuring the adaptability and maintainability of the software. .sp Adaptability is especially important in the area of network support. Many prior systems were only concerned with, at most, one network, and could therefore implement the network support in an ad hoc manner. Current and future systems will be required to communicate over several network backbones (for example, one of the LBL/CSAM hosts is connected to four different networks), with the result that the modularity of the network support software will be paramount in determining the utility of the delivery system. .sp An important area to explore is the use of the mail system as a network layer upon which to build high-level services. The usefulness of the system in such applications depends upon the degree to which the pieces of the delivery system on separate hosts conform to standards, as well as whether the mail service provides the guarantees necessary to implement realistic high-level services. .sp As in other areas of computer science, the appearance of state-of-the-art software in no way guarantees the elimination of obsolete (but functional) software performing similar services. Since communication requirements demand interaction with mail systems not conforming to the adopted standards, it is necessary to provide gateway functions between existing systems and any new implementation. .bp Delivery System Design .sp The delivery system consists of three processes which are always active on each host, plus separate receiver processes for each type of network substrate which is supported. The general flow of a message through the system is as follows: .sp .in +3 .ti -3 1. A composition utility (or network receiver process) formats a message for submission to the delivery system. .sp .ti -3 2. The message is posted to the process MAILER, which copies the message to a mail system directory, expanding local aliases and resolving the delivery agent for each user; MAILER also encrypts the body of the message to prevent un-authorized perusal of the message while it sits in the mail system directory. This copy of the message is then passed to the DELIVR process. If the message was successfully queued to DELIVR, a success status is passed back to the requestor process; otherwise, an error status is returned, permitting the requestor to take some action to rectify the situation. .sp .ti -3 3. DELIVR handles local deliveries, either by appending the messages to the user's `mymail' file, or by passing it to an appropriate alternate delivery agent process. All messages destined for remote hosts are grouped together by host; each group is then sent using the appropriate protocol for transmission to the appropriate host. .sp .in -3 In addition to this main path, several out-of-band flows can occur. They are: .sp .in +3 .ti -3 1. If any invalid local addresses are detected by MAILER, these addresses, with a copy of the message, are queued up in a file named `~msg/mreturn.xxx', where the xxx is a rotating sequence number. .sp .ti -3 2. If DELIVR is unable to place the mail in the user's file (file is locked, user over allocation, disk off-line) or the alternate user agent could not be found, a file named "~msg/dreturn.xxx" is created with a copy of the message. .sp .ti -3 3. If network delivery was requested to an unknown host, the message is placed in a file named "~msg/fretry.xxx". .sp .ti -3 4. If the network delivery could not be completed to a known host, the message is placed in a file named "~msg/sretry.xxx". .sp .ti -3 5. Some of the gateway modules create files of the form retry.xxx, where is replaced by some distinguishing character other than 's' or 'f'. .sp .ti -3 6. The process MFLUSH runs periodically (this is site-specific, with the default being 15 minutes), and cleans up after the other two processes as follows: .sp .in +3 .ti -3 a. mreturn.xxx, dreturn.xxx and fretry.xxx files are reformatted for return to sender, and queued to MAILER. .ti -3 b. all other retry.xxx files are queued to DELIVR for another attempt at delivery. Note: MFLUSH checks that the file has not been in the retry cycle longer than a site-specific period of time (the default is 7 days). If the message age exceeds this limit, the message is reformatted for return to sender as an aged message. .in -3 .sp .in -3 A variety of other processes can participate in the delivery process: .sp .in +3 .ti -3 1. NOTIFY - if this operating-system specific utility is in the standard search path, DELIVR will invoke it to notify local users of mail received with a message of the form .ce Software Tools Mail has arrived from `user at host' .sp For both DEC systems, the nobroadcast status of the user's terminal determines whether this message breaks through to the user. .sp .ti -3 2. TCADD - this particular alternate delivery agent is provided to perform controlled addition of messages to teleconference topic files used by the `tc' and `htc' utilities. It validates the sender's write access to the topic; if such access is granted, then it enters the message as a new entry in the teleconference database files. It also handles forwarding of messages to the topic guru if the message was addressed to tc_topic-request. .sp .ti -3 3. RMTXEQ - this alternate delivery agent permits users at other hosts to run commands remotely, subject to very severe integrity constraints. Upon reception of a message, RMTXEQ checks to see if the sender is a known super-user of the mail system; if so, then the command will be executed. Failing this test, the first token in the command to execute (the first line of the body of the message) is compared with a list of valid verbs that anyone can execute. If the command is valid, then the first line of the message is executed, with any other lines made available to the command as its standard input. The output of the command is captured, and mailed back to the sender. .in -3 .bp .ce User-Interface Utilities available .sp The following user-interface utilities are available in this distribution. The individual writeups for each can be consulted for more information. In addition, a primer on the use of `msg' may be found in section 4 of the manual. .sp .nf .ls 2 Htc - provide help on teleconference topics Mail - utility for sending mail Mretry - re-submit mail saved by composition programs Msg - utility for manipulating message files Msplit - utility for salvaging message files Postmn - report the presence of mail Resolve - resolve mail system user names Sndmsg - utility for sending mail to other users Tc - display entries in one or more teleconference topics Users - list valid mail recipients .fi .ls 1 .sp 3 .ce Interface between user-interface utilities and the delivery system. .sp The interface consists of two parts: .sp .in +3 .ti -3 1. When a message composition utility has completed composing a message in a disk file, it sends the name of the file to MAILER via the following library call: status = sndmlr(file) If `status' is returned as a value `OK', then the message was successfully processed and queued. If the value `ERR' was returned, then an error occurred; the composition utility will usually log a message to the user and queue the message for future submission via the following library call: status = savmsg(file, savfil) If `status' is returned as `OK', then the contents of `file' were successfully appended to `savfil' for future submission using `mretry'. An error return at this stage is essentially unrecoverable. Fortunately, this should never happen. .sp .ti -3 2. The other portion of the interface is the format of the message as delivered by DELIVR. This format is the same regardless of whether the mail is appended to a disk file or piped into an alternate delivery agent. Since the message format standard (RFC822) specifies ASCII messages only, the messages in a message file are separated by a special sequence which is never expected to be found in a real message - ^A^A^J. In practice, this has never collided with the contents of a real message. The format of each new message delivered by DELIVR is as follows: .sp .nf ^A^A^J Return-path: <...> 0 or more Received: timestamp lines The body of the message as the composition utility posted to MAILER .fi .sp There are a variety of subroutines provided in the library to permit manipulation of this structure. The source code for `postmn', `rmtxeq' and `msg' can be consulted for some actual examples. .sp .in -3 As you have probably gleaned from this discussion, it is the responsibility of each composition utility to guarantee that the message submitted to MAILER conforms to the RFC822 standard. The utilities provided in the distribution are guaranteed to do this; it is essential for users writing their own user-interface utilities to also follow this convention. Consult the source code for `sndmsg' to see the exact format of the files passed to MAILER. .bp .ce Interface to Networks .sp The interface of the delivery system to receive network mail is quite simple: .sp When a remote DELIVR wishes to transmit a message to the local host, it transmits an appropriate connect request for a server process which understands the SMTP protocol. Depending upon the operational characteristics of the network, the receiver process may already be active, awaiting the connect request, or it may be bound to a known socket, such that the connect request causes a copy of the receiver process to be activated. The receiver process issues a connect accept to complete the connection. The remainder of the dialogue between the two processes is in conformance with the SMTP protocol (RFC821). When the sender indicates that the message transfer has been completed, the receiver submits the file containing the message to MAILER, indicating success or failure to the sender depending upon the status received from MAILER. .sp The simplicity of this interface permits new network support (and gateways to other mail systems) to be added quite modularly, thus permitting easy extensibility. Note that the receiver process does not have to validate any addresses, as that is the domain of MAILER, who will return any invalid addresses to the sender. .sp 3 .ce Network Virtual Machine .sp To permit message transmission over a new network substrate, it is necessary that five (5) network-specific functions be implemented: .in +3 .ti -3 1. subroutine xxxfin - called by both sender and receiver routines to close down the virtual circuit between them. .sp .ti -3 2. integer function xxxget(buf) - get the next buffer from the network. `buf' is size MAX_DATA; the function return is the number of characters returned in the buffer or EOF. In addition, upon a successful read, `buf' will be EOS terminated. .sp .ti -3 3. integer function xxxirx(host) - the receiver calls this function to issue the connect accept and to fetch this host's name on this network. The function returns are OK/ERR; upon successful connection acceptance, `host' will be returned with this host's name on this network in the form "@hostname". .sp .ti -3 4. integer function xxxitx(thost, fhost) - the transmitter calls this function to issue the connect request to host `thost', which is in the form "hostname". The function returns are OK/ERR; upon successful connection, `fhost' will be returned with this host's name on this network in the form "@hostname". .sp .ti -3 5. integer function xxxput(buf) - the EOS-terminated buffer `buf' is sent on the virtual circuit. The function returns are number of characters sent or ERR. .sp .in -3 The templates for sender subroutine and receiver process are provided in smtptx.r and smtp.r, respectively. These templates reference the template routines netfin, netget, initrx, inittx and netput. To build versions specific to your network, you must create a file xxxdef containing the following defines: .nf define(initrx,xxxirx) define(inittx,xxxitx) define(netfin,xxxfin) define(netget,xxxget) define(netput,xxxput) define(smtptx,xmtptx) define(NETWORK_TYPE,"XXXNET") .fi .sp In addition, if the receiver process must run as a daemon, then the line .sp define(CONTINUOUS_SERVER,) .sp should be placed in xxxdef, also. .sp The following commands will create the sources for transmitter function and receiver process: ratfor xxxdef smtptx.r >xmtptx.f .br ratfor xxdef smtp.r >xmtprx.f .sp These fortran files can then be compiled, inserting xmtptx.obj into the msg library, and linking xmtprx to the library to build the receiver image file. .sp To cause your version of DELIVR to start using the network, you can follow one of two approaches: .in +3 .ti -3 1. While debugging your network specific routines, you can create a sender process using the following commands: ratfor xxxdef smtpxmt.r >xmtpxmt.f .br fc -v xmtpxmt.f -lmsg # VMS .br bigbld fc xmtpxmt.f # RSX .sp This image can then be placed in one of the directories in DELIVR's search path, and entries of the following form can be placed in the file `~msg/hosts': xhost * |"xmtpxmt xhost ??~msg/xmtpxmt.log" When DELIVR encounters messages destined for xhost, xmtpxmt will be spawned, with the message piped to it on STDIN. If xmtpxmt is unable to transmit the message, appropriate log messages will be found in the log file, and an error status is returned to DELIVR, such that the message will be requeued for retry. .sp .ti -3 2. After the network routines are debugged, and if the traffic over XXXNET warrants faster service, the support for XXXNET tranmission can be included in DELIVR. Simply add the following line to the file `netlist' that you created during MAILGEN: define(p_x,xmtptx) Then rebuild DELIVR in accordance with the build procedure used in MAILGEN. To start supporting XXXNET with this new DELIVR in place, you must add lines of the following form to ~msg/hosts: xhost * xxxnet .sp Upon reception of a message destined for xhost, DELIVR will now perform the transmit side of the SMTP link directly. .in -3 .sp The ability to embody the transmit function in a sender process also contributes greatly to the extensibility of the delivery system, as well as facilitating debugging. Since xmtpxmt is a tool which reads the message from standard input, one can generate a test message and invoke xmtpxmt directly, and can avail oneself of all of the debugging tools at hand. From the author's own experience, it is possible to support a new network easily in ~one day. .bp .ce Conditions Required for Successful Operation .sp .in +4 .ti -3 1. The latest release of the Software Tools Virtual Operating System (Fall 1983) have been built and be available. .sp .ti -3 2. The appropriate operating-system specific definitions be available (e.g. ST_BIN [VMS] or ST0: [RSX]). .sp .ti -3 3. The following known files be found in `~msg/' (and have WORLD read access): .sp .nf a. address b. malias c. hosts d. hnames e. topics f. sufile g. lnames h. msghlp i. rmtxeq.cmd .fi .sp .ti -3 4. The `tc' directory be available and contain the following files: .sp For each entry in `~msg/topics', which is of the form "name root", four files must exist in the `tc' directory: .sp .nf a. root.bnd b. root.ndx c. root.nfo d. root.tc .fi .sp .ti -3 5. The protections on all the files of 3 and 4 above must permit world read access. .sp .ti -3 6. The mail system utilities must be found in `~bin/'. .sp .ti -3 7. An account be defined (with minimal privileges) under which the SMTP remote object can run (not necessary for RSX). .sp .ti -3 8. The object 254 be defined to DECNET via the appropriate NCP commands, using the account in 7. .sp .ti -3 9. If the VMS Mail Gateway software is to be used, the distributed version of mail.com must reside in SYS$SYSTEM to prevent the VMS Mail Utility from being used. To use the VMS Mail Gateway software from RSX, it is necessary to install VMSTOMSG.TSK as VMSMAI, and to define VMSMAI as object number 27 using NCP. .sp .ti -4 10. If the UUCP software is to be used (on VMS), then the images for `rmail' and `bdsh' must be in the standard Eunice search path, the Eunice version of `rmail' must be placed in usr:, and st.com must be in st_usr: .sp 2 .in -4 The priorities of the processes in the startup command files are suggestive of the following requirements: .sp .in +3 .ti -3 1. MAILER should run at a higher priority than user processes, so that it can be effectively called as a subroutine. .sp .ti -3 2. Network processes should have high priority, so that they can quickly complete their utilization of scarce resources. It is suggested that they run at the same priority as MAILER. .sp .ti -3 3. Since DELIVR is the transmit end of a network connection, it should have high priority also. On RSX systems, DELIVR elevates its priority when transmitting over a network, and drops back down when processing local traffic. This is done within the confines of the inittx routines, so that it does not violate the virtual machine interface. .in -3 .bp .ce Creating a New Teleconference .sp To create a new teleconference topic, the following steps must be performed by a system manager: .sp .in +3 .ti -3 1. Decide upon the topic name and the filename root to be used for storing the state of the teleconference topic. .sp .ti -3 2. Execute the following commands .sp .nf cat nl: >$tc/root.bnd cat nl: >$tc/root.ndx cat nl: >$tc/root.tc .fi .sp where $tc is the `tc' directory on your system (~msg/tc [VMS] or ~msg [RSX]) .sp .ti -3 3. ed $tc/root.nfo .sp Place the following information in this file in the following order: .sp .in +3 .ti -3 a. if the READ access to this topic is to be restricted, enter the following: .sp .nf :read address .in +3 .cc * . . . *cc . .in -3 .fi .sp where `address' is of the form `user@host'. .sp .ti -3 b. if WRITE access is to be restricted, add .sp .nf :write address .in +3 .cc * . . . *cc . .in -3 .sp .ti -3 c. If a specific person is to act as the topic guru, add .sp .nf :guru full-address .fi .sp where `full-address' is the complete address required to reach the user from this host. .sp .ti -3 d. Finally, add the help text for display by the `htc' utility: .sp .nf :help Text describing the purpose of the topic, and any other information of general interest about the topic. .fi .sp NOTE: If the READ access list is omitted, anyone can read this topic. If the WRITE access list is omitted, anyone can add entries to this topic. If the GURU is omitted, entries sent to tc_topic-request will be forwarded to the address `postmaster'. .sp .in -3 .ti -3 4. Add the following line to the file `~msg/topics' .sp topic-name root .sp .ti -3 5. Set the protections on these four files to READONLY for the general public. It is necessary to assure that the general public can still read ~msg/topics, also. .in -3 .bp .ce System Utilities for Mail System Maintenance .sp 2 Several utilities are provided for maintenance of the system. Complete manual pages are provided in the (sys) section of the manual. The description below is merely meant to introduce you to them. .sp .in +9 .ti -7 addr - utility for regenerating the user address file. This utility is automatically invoked when the software tools system is started, and may be invoked whenever the system account file is modified. The appropriate invocation is .sp addr | sort >~msg/address .sp .ti -8 mstop - gracefully stop the delivery system. This utility tells MAILER, DELIVR and MFLUSH to clean up and exit. .sp .ti -9 mtrace - change the tracing level for the various log files. The default tracing level is 1, in which internal errors and communication errors are logged by MAILER and DELIVR. `mtrace' can be used to change the level to any of the following values: .sp .in +2 .ti -2 0 no logging at all .ti -2 1 only log internal errors within each program (e.g. could not open a temporary file which was just written) .ti -2 2 log communication errors between processes and over networks .ti -2 3 log enough information such that the history of each message can be traced .ti -2 4 log intermediate result babble, generally resulting in enormous log files. .sp .in -2 It should be noted that setting the tracing level to `n' implies all smaller tracing levels. .sp .ti -9 nalias - indicates to MAILER that it should reload its volatile database. This should be done whenever ~msg/address, ~msg/malias or ~msg/hnames are changed. .sp .ti -9 netdlv - turn network delivery of messages on or off. .sp .ti -9 tcarch - archive old teleconference entries. .in -9 .bp .ce Care and Feeding of your Mail System .sp 2 Several aspects of the system require preventive maintenance for smooth operation. The purpose of this section is to outline those areas that need the maintenance. .sp .in +3 .ti -3 1. About once a week, create new versions of the log files and peruse the old ones. Scanning the old ones might indicate a problem with a network connection that you are not aware of, due to the system's persistence to achieve delivery. After perusal, these old versions can be archived in the appropriate manner for your site. .sp .ti -3 2. At least daily, list the contents of the ~msg directory. If the system is experiencing any difficulties, this directory will get clogged with temporary files. Depending upon the amount of mail traffic, you might wish to check more frequently. If there are problems, see the section below on how to recover from problem situations. .sp .ti -3 3. Once a day, I bounce messages to myself off each machine to which we are connected via addresses of the form .ce tools at lblg via X .sp where X is replaced by the target host. By creating an alias for these addresses in ~/malias (home directory), this can be made quite simple. The arrival of the messages assures that the delivery systems on the other hosts are in working order. .in -3 .bp .ce Healing a Sick Delivery System .sp 2 Occasionally, the delivery system may get constipated, usually due to circumstances outside of it's control. First I will characterize the symptoms of a constipated system, how to effectively treat the symptoms, and describe possible causes. .sp 2 .ul Symptoms .sp .in +3 .ti -3 1. You notice that not all three (3) of the server processes are active. .sp .ti -3 2. ~msg seems to be filling up with files named tdelivr.XXX. .sp .ti -3 3. A large number of sretry.XXX files queue up in ~msg. .sp .ti -3 4. People have perfectly good messages returned as being too old. .sp .ti -3 5. Perfectly good messages are returned with "cannot access file" error messages. .sp 2 .ti -3 .ul Treatment: .sp .ti -3 Regardless of the symptom, .sp .ti -3 1. Stop the mail system using `mstop'. .sp .ti -3 2. If that fails, kill the processes in a system-dependent manner. .sp .ti -3 3. Use the tool `tail' on the appropriate log files to see if the process diagnosed its own problem; especially concentrate on delivr.log, since accumulating files in ~msg is usually due to delivery problems. .sp .ti -3 4. tdelivr.XXX files have the same format as sretry.XXX files; therefore rename them to sretry.sav;* so that they will be requeued to DELIVR when the system is restarted. .sp .ti -3 5. Delivr.log sould be able to tell you why sretry.XXX files are piling up in ~msg. Check ~msg/hosts to make sure that your definition is correct. .sp .ti -3 6. If you have specified a default network delivery action to take on hosts which are not explicitly known, check to see if the user has a typo in the .br RCPT TO:<...> .br line of the file. If so, this can be corrected with an editor. .sp .ti -3 7. Check that the system time has not been set to something absurd. .sp .ti -3 8. Check that the address file correctly maps users to home devices and directories. Also make sure the appropriate disk is mounted correctly. .sp 2 .ti -3 .ul Possible Causes: .sp .ti -3 1. If you are debugging new network support, DELIVR can often be destroyed. .sp .ti -3 2. This usually occurs because DELIVR is hanging. This can happen if new network support does not enforce any timeout requirements. (See decnet.r and support routines for how to handle this problem if the low-level network code does not provide timeout services.) Since SMTP is a lock-step protocol, if the receiver process does not respond, and timeout support is not provided, DELIVR will wait forever until it receives it's acknowledgement. .sp .ti -3 3. The primary reasons for queued up sretry files are severe hardware problems on the remote machine, such that it is unavailable; they should queue up for exactly that reason. Unfortunately, the `default' host support can cause sretry's to non-existent hosts due to typographical errors. This can also occur if new network support software returns the incorrect function returns to netdlv in DELIVR. .sp .ti -3 4. If the system date is incorrect, messages can be erroneously returned. .sp .ti -3 5. If the address file does not have the correct information, or if the disk is full or offline, the messages will be returned. .in -3 .bp .ce Timing Considerations .sp 2 The timing constraints discussed above are reflected in the timeout values used in the distributed software. The values used are as follows: .sp .in +3 .ti -3 1. If the link is totally between non-interactive processes on the two machines, then each netput or netget must complete in 60 seconds. .sp .ti -3 2. If one end of the link is dependent upon human interaction (VMS mail receiver talking to a user sending VMS mail), the restrictions are relaxed to 30 minutes (1800 seconds) for each netput or netget. .in -3