EXECSYMB John Osudar Electronics Department Argonne National Laboratory 205 A-051 9700 South Cass Avenue Argonne, IL 60439-4837 Phone numbers: FTS: 972-7505 (312) 972-7505 Electronic mail addresses: Bitnet: B35049 at ANLCMT MFENET: B35049@AN2 ***** THIS DIRECTORY CONTAINS AN UPDATE TO THE EXECSYMB SOFTWARE THAT WAS ***** ***** ORIGINALLY SUBMITTED TO THE SPRING 1987 SIGTAPE. THIS IS >>>NOT<<< ***** ***** A COMPLETE RESUBMISSION!!! ***** Introduction ------------ EXECSYMB is a VAX/VMS server symbiont that feeds queue entry information to detached processes that are designed to execute specific operations (hence the name, Executive Symbiont). It is multi-threaded (it can handle up to 16 queues at the same time); it requires one detached process and one mailbox for each queue, plus one mailbox for status information and one for detecting process exit status for each copy of EXECSYMB (i.e. one per 16 queues). EXECSYMB can be used to implement a wide variety of VMS queue-based operations. Among its present applications are: (1) Remote queuing of print jobs (and limited remote queuing of batch jobs) (2) Pre-processing and post-processing of jobs destined for other server symbionts (e.g. performing MFENET gateway operations on files queued to MFENET inbound-file symbionts) (3) Queued file transfer for DECnet and other networks This program was written with one major goal in mind: to simplify the system's interface to a user-written queue processor, eliminating (or at least hiding) the quirks that are present in the job controller-to-symbiont interface supplied by Digital. Under VMS V4, symbionts operate in a rather strange environment. A symbiont is a detached process (unlike VMS V3, where it was a subprocess of the job controller). However, it lacks several useful things that most processes have: symbionts have no CLI, which means that they can't spawn commands; symbionts are always created with username SYSTEM and one privilege (SETPRV); and symbionts cannot access process-permanent logicals (SYS$INPUT, SYS$OUTPUT, SYS$ERROR) the way other processes can. A symbiont's SYS$INPUT points to a mailbox to which the job controller writes its request messages, and SYS$OUTPUT points to another mailbox from which the job controller reads status messages from the symbiont. (Incidentally, the latter is the job controller's "achilles heel"; if your symbiont accidentally writes a bunch of stuff to SYS$OUTPUT, the job controller will respond with a series of console error messages about "invalid mailbox message received", followed by a fatal "end-of-file on job controller mailbox" that aborts the queue manager, stopping ALL queues on your system and leaving active batch jobs in VMS limbo. VMS V4.4 allows the queue manager to restart after such an abort, but this usually leads to a recurrence of the same problem, and a hard abort on the part of the queue manager.) EXECSYMB was designed to make the symbiont environment easier to use. Of course, EXECSYMB itself has to deal with the restrictions listed above. However, a queue handled by EXECSYMB is processed by an EXECSYMB detached process, which is created with the DCL CLI, a real command procedure as SYS$INPUT, a log file as SYS$OUTPUT, and a pair of mailboxes for talking to EXECSYMB. The protocol for passing information between the executive symbiont and the detached process involves transferring "items", which EXECSYMB parses from the job controller's request message, through a mailbox. One of the "items" is created by EXECSYMB itself -- it's called "EXEC_STEP", which is an "execute trigger", telling the detached process that a string of items has been completely transferred and an operation (execute, reset, exit) is to be performed. The detached process returns a completion status through the other mailbox, which provides EXECSYMB with a task status to return to the job controller. As a result, it is relatively easy to write a DCL command procedure to process a queue. There are just a few requirements for such a procedure, which are outlined below, in the section "sample queue processing procedure". Features of EXECSYMB -------------------- In addition to providing a safer and less hostile environment for queue processing, EXECSYMB adds some features that aren't available to ordinary user-written symbionts without additional coding. (1) EXECSYMB is multi-threaded, so that it can handle a number of different queues at the same time. The code required to do this correctly is rather complex, so concentrating it in EXECSYMB is quite practical. (2) EXECSYMB handles the AST-level delivery of job controller messages for things like stop and pause requests. (3) EXECSYMB eliminates the need for calling the symbiont support routines to parse, copy, and re-copy the request message components (request items), as is done in most user-written symbionts. EXECSYMB parses the request message itself, and keeps pointers to the individual items; only those items required by a particular queue are actually passed to the queue processor. (4) EXECSYMB provides a requeue-on-error capability that can be selected on a per-queue basis. If a queue specifies a retry interval, and a task is aborted with an error status, the job is automatically requeued to try again after the specified interval. It is also possible to specify whether the job is to be restarted from the task that aborted, or from the beginning. (Note that, if this feature is selected for a particular queue, the queue processor must do some reasonable error detection and recovery, returning an "error" status to EXECSYMB only if it really wants the job to be retried.) (5) EXECSYMB allows a queue processor to handle all copies, the first copy only, or the last copy only, for jobs that specify multiple job copies or file copies. (This is especially important when the queue processor is transferring one copy of the file, and forwarding the number-of-copies parameter for handling elsewhere.) (6) EXECSYMB allows the queue processor to be started dynamically, when the queue actually has work available. This option is selectable on a per-queue basis, by specifying a "timeout" interval. If the queue processor detached process exists but is idle, and no work is queued within the timeout period, EXECSYMB will tell the process to exit (while keeping the queue in the active state, as far as the VMS job controller/queue manager is concerned.) Running EXECSYMB ---------------- Here's what you need to know to run EXECSYMB. (Remember that a symbiont image must reside in SYS$SYSTEM.) The EXECSYMB process will change its name to SYMBIONT_EXEC_n, where n is a sequence number that starts at 1. Since each process handles up to 16 queues, you shouldn't see sequence numbers above 1 unless you have more than 16 queues processed by EXECSYMB at the same time. EXECSYMB sets its base priority from the /BASE_PRIORITY=n qualifier on the first queue that it starts; if EXECSYMB's base priority is important to you, keep this in mind. EXECSYMB sends important messages, such as startup and shutdown of the main process, to the CENTRAL operator. Other information is written to the log file, which is named SYS$MANAGER:EXECSYMBn.LOG (where the process name is SYMBIONT_EXEC_n). EXECSYMB remains active as long as it has at least one active queue to process; upon stopping the last queue (and waiting for its queue processing detached process, if any, to exit), EXECSYMB will exit. EXECSYMB can send items to the detached process in ASCII or binary form. The ASCII form uses two writes per item, one to send the item name, the other to send its value. The binary form does one write per item, and includes the item's binary code number as the first four bytes of the record. In the ASCII form, many item values are converted from their raw form to an edited ASCII form; e.g. the UIC item will be converted from four binary bytes to a string of the form [gggggg,mmmmmm]. The binary form does no editing on item values. In addition, EXECSYMB can be told to skip sending items that have a null string as a value. This can save on a lot of processing. EXECSYMB terminates the list of items for a particular task with an item that it makes up itself. This item has the name "EXEC_STEP", binary code 0, and can have one of three ASCII values: "EXECUTE", which means that the task's item list has been completely transferred, and the task is ready for execution; "RESET", which means that a partial item list was sent, which should be discarded, and the detached process should return to its initial state; and "EXIT", which means that the detached process should exit. The "EXECUTE" command expects the detached process to return a completion status, and then perform a "RESET" automatically. You can tell EXECSYMB how to process a queue through two queue qualifiers: the command procedure that is to be used is specified through the /LIBRARY qualifier, and the list of processing options is specified as a character string in the /SEPARATE=RESET qualifier (which corresponds to the item JOB_RESET_MODULES). Since device control libraries and job reset modules aren't too meaningful in the context of server symbionts, use of these two qualifiers works out quite well. The /LIBRARY qualifier takes a filename as its value. Although device control libraries are required to reside in SYS$SHARE and have the extension .TLB, the /LIBRARY qualifier does not place these restrictions on the value you specify. (In fact, the job controller sticks a SYS$SHARE: on the front of this value, and a .TLB on the end, but EXECSYMB strips those off!) Thus, /LIBRARY=SYS_PROCS:EXAMINEQUEUE.COM is a valid specification. Note that this field is restricted to a total of 39 characters (since the job controller thinks this is the filename part of a library name.) The /SEPARATE=RESET qualifier takes an arbitrary character string as its value. EXECSYMB requires this string to be a comma-separated list of options from the following list: TIME="d hh:mm:ss.cc" This specifies how long to wait before retrying aborted jobs. If not specified, aborted jobs are not retried. DYN="d hh:mm:ss.cc" This specifies the timeout period for a queue with a dynamically-started queue processor. If not specified, the queue processor is always active while the queue is started. ITEMS="itemlist" This specifies the list of item codes to be passed to the queue processor. Codes are represented as decimal numbers, and the syntax mm:nn can be used for a range of codes. SPOOL=[spooldir] This specifies a directory into which spool files can be entered, if it is possible for the queue to include spool files (i.e. files created from output to a spooled device.) Such a directory MUST exist on all disks that are used as secondary devices for spooled devices. ASCII or BINary This specifies the format for passing item values to the detached process. PRINTer or SERVER This specifies the queue type, and is only used to set the indicator for SHOW QUEUE (printer queues print, server queues process) NONULl or NULL This specifies whether items whose values are the null string are passed to the detached process CHECK or NOCHECK This specifies whether the queue can checkpoint jobs, i.e. whether a job consisting of several tasks in which one task aborts is restarted from the point of abort (CHECK) or from the beginning of the job (NOCHECK) COPY={ ALL, This specifies which copy of a multi-copy job, First, and which copy of a multi-copy file in a job, or Last} is to be processed by the detached process. ALL gets every copy, FIRST gets the first file copy in the first job copy, and LAST gets the last file copy in the last job copy. (Note that certain flags in the SEPARATION_CONTROL item may be set only on the FIRST or LAST copy.) USER=username If specified, this sets the username under which the queue processor detached process will run. (The default is SYSTEM.) This can be useful for accounting, or for processors that use proxies into other DECnet nodes. Note that the specified username need NOT be a valid one on the system (i.e. there doesn't have to be an entry in SYSUAF for that username)! Defaults, if a particular option is not specified, are: no requeue, permanent queue processor, no ITEMS, no SPOOL directory, ASCII, SERVER, NULL, CHECK, COPY=ALL, USER=SYSTEM A little more needs to be said about spool file handling. EXECSYMB can be used to process queues from spooled devices (such as spooled null printers, used for remote printer support). When output is written to a spooled device, a spool file is created on the device's secondary device (disk), and queued to the device's queue. This file is identified only by file ID, and has no directory entry. In some applications (e.g. where the file will be copied to another node with a network copy program), the file must have a full filename to be effective. EXECSYMB will detect spool files for queues that specify the SPOOL=directory option, and will enter such files in the specified directory on their device. Once such a file has been processed (and is about to be deleted automatically by the job controller), EXECSYMB removes the directory entry. Sample queue processing procedure --------------------------------- The following is a sample queue processing procedure that can be used with EXECSYMB. It is reproduced here to illustrate the required features of such a procedure; it can also be found, without comments, in this directory in the file EXAMINEQUEUE.COM (which can be used as-is with EXECSYMB.) This procedure does nothing "useful"; it produces a log file containing the items passed for each request it processes, and it indicates successful completion of each task. First, you can specify any setup commands you require. SET NOVERIFY and SET NOON are recommended. $ SET NOVERIFY $ SET NOON Next, the procedure must locate the mailboxes that it uses to talk to EXECSYMB. The input mailbox (written by EXECSYMB, read by the procedure) has a logical name of CMD_MBX_pid where "pid" is the process ID of the detached process running this procedure. The output mailbox (written by the procedure, read by EXECSYMB) is used for returning status, and has the name STAT_W_MBX_pid (where "pid", again, is the process ID). We can also determine the name of the queue we are processing by translating the logical name QUEUE_NAME_pid. Thus, the procedure must determine its own PID, and then use it to access this information. $ PID==F$GETJPI("","PID") $ QUEUENAME=F$TRNLNM("QUEUE_NAME_''PID'") The items sent by EXECSYMB all have names that can be used as DCL symbols. That is the technique used here. (EXECSYMB has an alternate format for items, in which a fixed-length binary item code precedes the item value; that format is intended for use by compiled programs that run in the detached process.) The item "EXEC_STEP" is used by EXECSYMB to guide the operation of the detached process; EXECSYMB can assign it the values "EXECUTE", "RESET", and "EXIT". Here, we initialize the symbol to the value "INPUT", and use the symbol as the destination label of a GOTO. This is a rather simple way of implementing the control operation. $ EXEC_STEP="INPUT" Before we do any real work, we must open the input and output mailboxes for talking with EXECSYMB. $ OPEN/WRITE/ERR=NOMBX STAT STAT_W_MBX_'PID': $ OPEN/READ/ERR=NOMBX INPUT CMD_MBX_'PID': Now comes the good part of the procedure. It starts with the label that was initially assigned to EXEC_STEP. The first command reads the NAME of the symbol (i.e. the item name) being passed, and the second command reads its VALUE. Thus, EXECSYMB can tell the process which item is being passed, and its value, and a symbol of that name with that value is created. $INPUT: $ READ/END=ENDMBX INPUT SYMBOL $ READ/END=ENDMBX INPUT 'SYMBOL For demonstration purposes, we echo the information to SYS$OUTPUT, which is the log file. $ WRITE SYS$OUTPUT SYMBOL $ WRITE SYS$OUTPUT 'SYMBOL Having completed the value assignment, we go to the location pointed to by EXEC_STEP. This was initially INPUT, and will remain so until EXECSYMB sends the item EXEC_STEP and a new value (EXECUTE, RESET, or EXIT) for it. $ GOTO 'EXEC_STEP When EXECSYMB says "EXECUTE", it indicates that the item sequence for this task is complete, and should be acted upon. A real symbiont processor would do something with the item list at this point (e.g. do something with the file specified in item FILE_SPECIFICATION). Once the operation is complete, EXECSYMB expects a status to be returned via the mailbox. This example always returns a success status. $EXECUTE: $ WRITE STAT "%X00000001" After an "EXECUTE" operation is complete, or when EXECSYMB explicitly requests a "RESET" (such as when a job is deleted while being processed), the procedure should reset controlling variables to initial values, and return to its input processing loop. This procedure has one controlling variable, EXEC_STEP, which is now reset to point back to the INPUT loop. (It also writes out a separator line to the log file, to mark the boundary between requests.) $RESET: $ WRITE SYS$OUTPUT "========================================" $ EXEC_STEP="INPUT" $ GOTO INPUT Finally, if an "EXIT" operation is requested, EXECSYMB wants the detached process to exit. One way to do this is to log off; you can also do an EXIT, a LOGOUT, or whatever else you desire (including, for example, sending operator notification that the detached process is exiting.) $EXIT: $ LOGOFF