.RIGHT MARGIN 80 .TITLE;RSX-11M-Plus System Accounting and Datatrieve .SUBTITLE;B. Z. Lederman I.T.T. World Communications .NOTE;Abstract One of the features of RSX-11M-Plus is system accounting, which allows the accumulation of statistics about the system (such as the number of logins, disk I/O activity, print queue submissions, and so on). One of the methods supplied to extract this information is the conversion of the system accounting file to a form readable by Datatrieve, and a command file to produce reports with Datatrieve. This paper will show how these commands have been changed to suit our installation, with an explanation of why each change was made, and the benefits derived from the change. Some options which may be of value to other installations will also be shown. The intent is to show how easily the reports may be tailored to individual needs by using Datatrieve as the report generator. .END NOTE .PARAGRAPH After we had installed M-Plus and were satisfied that it was running correctly, I became interested in the system accounting feature, which was one of the features which led us to convert from RSX-11M. System accounting is started and stopped with privileged commands, and in our system it is always started by the [1,2]STARTUP.CMD file, so that our system is always accumulating statistics. These statistics become useful only when they are output in human-readable form, and the first step in this process is to convert the data from the file in which it is stored by the system (normally [1,6]ACNTRN.SYS) with the command: .BLANK SHOW ACCOUNTING/DATATRIEVE ACCOUNT.DAT .BLANK This places the data in a file (in this example named ACCOUNT.DAT) where it may be read by Datatrieve (or by other programs). The next step can be to use the command file supplied by DEC, [126,24]ACNTRN.CMD, which will produce a report for each type of transaction recorded by system accounting. It is invoked by the command: .BLANK DTR @[126,24]ACNTRN.CMD .BLANK and it will prompt the user for the name of each report. .PARAGRAPH I immediately found a number of features in this command file which I did not like. First, it creates the neccesary record definitions, procedures, etc. in order to create the reports, but then immediately after using them once, it deletes them. This is inefficient, because it requires both time and computer resources (especially disk I/O) to create and delete all of these structures, as they will be needed the next time a report is desired. It did not make sense to me as I planned to obtain reports at regular intervals. There is also the problem that whenever something is defined in Datatrieve, it takes up space in the dictionary (a file usually named QUERY.DIC), and when it is deleted, the space is not re-used, so that each time the command file is invoked to obtain reports, the dictionary file will get larger and larger, with an increasing amount of space allocated on disk but not re-useable until the dictionary compression utility is used. Therefore, the first change I made was to edit the file and remove all lines beginning with the word "DELETE", so the file would create the definitions and not delete them. I also removed all lines beginning with a colon such as ":SS-REPORT", as these are commands to invoke the procedures. I thus have a file which will set up all of the data structures once, and now I can invoke the individual procedures whenever I want one particular report. After invoking the modified command file, the dictionary contents are as follows: .BLANK.NO JUSTIFY.NO FILL.TEST PAGE 13 DTR> show all Domains: RESOURCE-INFO SYSTEM-INFO USER-INFO Records: RESOURCE-INFO-REC SYSTEM-INFO-REC USER-INFO-REC Procedures: ALL-REPORT CRH-REPORT DEA-REPORT DMO-REPORT DST-REPORT INP-REPORT INV-REPORT LOG-REPORT MOU-REPORT PRT-REPORT RTP-REPORT SAB1-REPORT SAB2-REPORT SS-REPORT TAB1-REPORT TAB2-REPORT TIM-REPORT UAB1-REPORT UAB2-REPORT Tables: .BLANK.JUSTIFY.FILL Each of the procedures generates a single report on a particular resource, and each may be invoked when wanted. For example, the LOG-REPORT looks like this: .BLANK.NO JUSTIFY.NO FILL.TEST PAGE 20 L O G I N T R A N S A C T I O N 29-Sep-83 Page 1 NAME UID DEV ACNT UIC LOGON TIME B LEDERMAN PNA4 TT5 0003 [300,003] 6-Sep-83 13:47:13 B LEDERMAN PNA51 TT7 0003 [300,003] 13-Sep-83 8:19:1 B LEDERMAN PNA53 TT7 0003 [300,003] 13-Sep-83 8:20:29 B LEDERMAN PNA55 TT7 0003 [300,003] 13-Sep-83 8:21:49 B LEDERMAN PNA57 TT7 0003 [300,003] 13-Sep-83 8:23:56 B LEDERMAN PNA63 TT7 0003 [300,003] 13-Sep-83 8:30:39 B SYSTEM SYS1 TT17 9901 [001,001] 29-Aug-83 8:29:7 B SYSTEM SYS2 TT16 9901 [001,001] 29-Aug-83 8:29:13 B SYSTEM SYS3 TT3 9901 [001,001] 29-Aug-83 9:8:26 B SYSTEM SYS15 TT4 9901 [001,001] 6-Sep-83 8:11:47 C SCOTT PNA104 TT7 0011 [300,011] 13-Sep-83 10:19:5 C SCOTT PNA105 TT7 0011 [300,011] 13-Sep-83 10:19:48 C SCOTT PNA112 TT16 0011 [300,011] 13-Sep-83 10:35:23 .JUSTIFY.FILL.PARAGRAPH I then started looking at the way the reports were generated, and I found an area where an immediate improvement could be made. Each report was generated with the command sequence: .BLANK.TEST PAGE 3 FIND xxx WITH CODE=nn .BREAK REPORT CURRENT ON *."DEVICE OR FILE" .BLANK The first line finds all of the records in a domain (a collection of data) with a transaction code of nn (each transaction type has it's own code), and places it by default in a collection named "CURRENT". The second line reports this collection on a device or places it in a file, depending upon the answer received from the user; when Datatrieve sees an asterisk, it means that the user is to be promped for an answer. This certainly works, but it requires Datatrieve to go through the data once to find the records with the proper code, and a second time to do the report. It is more efficient to do both at once with a command such as: .BLANK REPORT xxx WITH CODE=nn ON *"Device or File" .BLANK and so I changed all of the procedures to do this. .PARAGRAPH The next step was to examine the record definitions. A record describes how each data field is arranged within each record of a data file, and is the basis for all data retrieval and processing. The record definitions supplied put related data records (such as all user information) in one record definition, making three large records in all; one each for system information, user information, and general system resources. My objection to this was that it makes each record definition very large, and, in my installation, I am interested in only one information type at a time. The problem with a large record definition is that it uses up pool (it is very important to remember that this refers to Datatrieve's internal working pool, NOT RSX system pool), and running out of pool in Datatrieve is rather like running out of system pool in RSX, only you lose just one job rather than the whole system. Conserving pool space is a goal in Datatrieve which is persued with just as much fervor (and with just as much reason) as conserving system pool in RSX, so a record definition which is unneccesarily large is something to avoid. Another reason is that, if you forget the names of the individual fields (data items), and try to do a SHOW#FIELDS command to see the names, the first fields will scroll off your CRT screen long before the last fields come up, which is annoying. I decided, therefore, to set up one record definition and one domain for each transaction type, so I could deal individually with each one as I needed it. One of these cut down records would look something like this: .BLANK.NO JUSTIFY.NO FILL.TEST PAGE 14 01 ACCOUNTING-TRANSACTION-REC. 10 CODE PIC IS 99 USAGE IS COMP. 10 LOG. 15 LOGNAM PIC IS X(16). 15 LOGUID PIC IS X(10). 15 LOGDEV PIC IS X(6). 15 LOGACC PIC IS 9(4) USAGE IS COMP. 15 LOGUIC PIC IS X(10). 15 LOGDAT USAGE IS DATE EDIT-STRING IS DD-MMM-YY. 15 LOGHUR PIC IS 9(2) USAGE IS COMP. 15 LOGMIN PIC IS 9(2) USAGE IS COMP. 15 LOGSEC PIC IS 9(2) USAGE IS COMP. ; .JUSTIFY.FILL.BLANK and so on until all fields are defined. All records start with CODE, which is the two digit code identifying the packet type: the "USAGE#IS#COMP" signifies that the field is a one word binary integer (like INTEGER*2 in Fortran). Similarly, "USAGE#IS#DATE" is an 8 byte date format increasingly used by DEC. The record types were logically organized by the developer at DEC by naming each field with 6 characters: the first three refer to the packet type (in this case the Login Transaction Block), and the last three to a field (DEV is Device, UIC is the familiar User Identification Code, and so on). This is a very logical method of organizing the data, and there is nothing "wrong" with it, but I chose to re-organize it in a different manner, for two reasons. First, if the report procedures are examined, it may be seen that all of the fields are referenced in a manner such as: .BLANK PRINT LOGNAM("NAME"), LOGUID("UID"), LOGDEV("DEV"), LOGACC("ACNT"), LOGUIC("UIC"), .BLANK The reason is that the field name must be given to identify the data, and Datatrieve, by default, uses the field name to identify that column at the top of the page, so the additional reference such as '("NAME")' must be made to override the default and have the word NAME print at the top of the column rather than LOGNAM. There is nothing wrong with this, but it requires extra working space in the report writer. This could also be done by placing the additional qualifier "QUERY-HEADER#IS#NAME" in the record definition of the LOGNAM field, so that Datatrieve would use the word NAME to head a column whenever LOGNAM is referenced: this would reduce the pool space used in the report, but increase the pool space used by the record definition. My other reason for the change is that there is a name field in most of the packets, and rather than having to know a different field name for each packet, I decided to name all of the fields in all of the records uniformly: thus the device field in all records is DEVICE, the UIC field in all records is UIC, and so on. That way, I don't have to know a lot of field names, and the correct header will print out whenever I do a query or report. I also squeezed down the definitions a little by removing the word "IS", which is optional, and by removing unneccesary edit strings. I then had a set of uniform record definitions, which look like this: .BLANK.NO JUSTIFY.NO FILL.TEST PAGE 24 DEFINE RECORD BASE-LOGIN-REC USING 01 BASE-LOGIN-REC. 10 CODE PIC 99 USAGE COMP. 10 FI PIC XX . 10 NAME PIC X(14). 10 DEPT PIC XXX. 10 NBR PIC X(7). 10 DEVICE PIC X(6). 10 ACC PIC 9(4) USAGE COMP. 10 UIC PIC X(10). 10 DATE USAGE DATE. 10 TIM. 20 FILLER PIC 99 USAGE COMP. 20 FILLER PIC 99 USAGE COMP. 20 FILLER PIC 99 USAGE COMP. 10 TIMR REDEFINES TIM. 20 LH PIC 99 USAGE COMP. 20 LM PIC 99 USAGE COMP. 20 LS PIC 99 USAGE COMP. 10 TIME PIC 9(6) COMPUTED BY (LH * 10000) + (LM * 100) + LS EDIT-STRING 99,99,99. 10 FILLER PIC X(60). ; .BLANK.JUSTIFY.FILL This definition has some features not in the DEC supplied records, and deserves some explanation. First, the UID field of the original definition has been replaced. This field contained data of the form "BZL0013", where BZL would be the three letter mnemonic defined in the system account file for that user's account, and the number is the login sequence for that particular login. As it happens, my company has been using three letter mnemonics to designate different departments for many years, and these were used to designate the department for each user who has an account on the system. I therefore decided to access the data in two fields, the first named "DEPT" to recall the first three letters, and "NBR", which is the login number. Similarly, I have set up the name field to separate the first inital (and blank space) from the last name, which is what I will be using in my reports. The time field require some explanation. The hour, minute, and second are recorded in separate binary integers. Since the ususal way to print out a time in an RSX system is "hh:mm:ss", each DEC report concatenated the fields by printing them in a manner such as this: .BLANK PRINT LOGHUR||":"||LOGMIN||":"||LOGSEC("TIME")USING X(8) .BLANK but there is a problem: when integer fields are concatenated, leading zeroes are suppressed, and I could find no way to stop this with edit strings, so that a time of 8 hours, 14 minutes, 3 seconds would print out as "8:14:3", and not "08:14:03" as would be expected. This looked messy to me when a column of times appeared on a report. The best I could do was to combine the fields using the COMPUTED#BY feature. TIM is a group name, which allows a number of fields to be referenced together. The three FILLER pictures allocate the correct amount of space in the record definition for the time data, but FILLER never prints out, nor does it appear in a SHOW#FIELDS command: it is a way to allocate space in a record without using it. The next entry, "TIMR#REDEFINES#TIM" means I want to give Datatrieve an alternative method of accessing the space reserved in the record definition, and now I put in the three real fields containing the hour, minute, and second. When Datatrieve prints out a record, and there is a REDEFINES, the first field in the list is taken by default: in this case, the FILLER is first, so it does not print out, and neither does the TIMR fields, but they are always there, and may be accessed explicitly by name whenever they are wanted. What will print out instead is the COMPUTED#BY field: this type of field does not actually take any space in the record, but is computed by Datatrieve when the information is required. The computation is to move the hour left 4 digits by multiplying by 10000, and minutes left two places by multiplying by 100, adding the hour, minute, and seconds together, and printing it out with commas (as there is not provision to put colons in an edit string. The result for the time given above would be "08,14,03", and while commas are less familiar than colons, all time fields line up one over the other when printing on a report, and the time is printed this way every time, including queries as well as reports without having to put in the concatenation operation, so I decided that I would put up with commas to gain the other advantages. I repeated this procedure for all of the transaction types, creating a record definition named BASE-xx-REC and a domain named BASE-xx, where xx corresponded to some transaction type such as LOGIN, RESET, CRASH, etc. The one exception to this is that I combined the MOUNT, DISMOUNT, ALLOCATE, and DEALLOCATE transactions into one record called BASE-AM-REC and one domain called BASE-AM: this is possible as the transaction packets are the same for these four transactions, and I will be reporting them all together, as will be shown later. .PARAGRAPH I also created a new report procedure for each transaction type, similar to this one: .BLANK.NO JUSTIFY.NO FILL.TEST PAGE 15 DEFINE PROCEDURE LOGIN-REPORT READY BASE-LOGIN REPORT BASE-LOGIN WITH CODE=19 SORTED BY DEPT, NAME, DATE, TIMR ON LOGIN.RPT SET COLUMNS-PAGE=80 SET REPORT-NAME="Login Transaction" AT TOP OF NAME PRINT FI|NAME PRINT DEVICE, UIC, DATE, TIME USING 99,99,99 AT BOTTOM OF NAME PRINT SKIP, COL 1, "Number of logins for", SPACE 1, FI|NAME, SPACE 1, COUNT, SKIP AT BOTTOM OF DEPT PRINT SKIP, COL 1, "Number of logins for", SPACE 1, DEPT, SPACE 1, COUNT, SKIP 2 END-REPORT FINISH END-PROCEDURE .BLANK.JUSTIFY.FILL Each of these reports is set to come out on a specific file, as I know I want my reports in files, and not on TI:. Note that it is possible to sort by TIMR, the re-defined time field, even though it does not normally print out. Sorting by the group name results in all fields in the group being sorted, which is what is desired. In my case, I want to sort by DEPARTMENT and NAME, but sorting by DEVICE or any other field, is also possible. A portion of the report generated by the above procedure looks like this: .BLANK.NO JUSTIFY.NO FILL.TEST PAGE 35 Login Transaction 13-Sep-83 Page 1 DEPT DEVICE UIC DATE TIME B LEDERMAN TT5 [300,003] 6-Sep-83 13,47,13 Number of logins for B LEDERMAN 1 W ROACH TT7 [300,002] 6-Sep-83 10,08,26 TT4 [300,002] 6-Sep-83 11,39,05 TT3 [300,002] 6-Sep-83 13,35,12 TT6 [300,002] 6-Sep-83 13,47,22 TT7 [300,002] 6-Sep-83 14,17,26 TT3 [300,002] 6-Sep-83 14,20,10 TT4 [300,002] 6-Sep-83 14,29,58 TT6 [300,002] 8-Sep-83 15,08,59 TT3 [300,002] 9-Sep-83 08,45,09 TT5 [300,002] 9-Sep-83 09,41,47 TT3 [300,002] 9-Sep-83 11,21,10 TT4 [300,002] 9-Sep-83 12,00,50 TT5 [300,002] 9-Sep-83 13,54,50 TT16 [300,002] 9-Sep-83 14,14,22 TT16 [300,002] 9-Sep-83 14,16,17 TT6 [300,002] 9-Sep-83 14,33,30 TT5 [300,002] 9-Sep-83 15,43,03 Number of logins for W ROACH 17 Number of logins for PNA 18 .BLANK.JUSTIFY.FILL It is also very easy to get only the summary of the data, rather than all of the details simply by not printing the details. Where I want this, I create another procedure, which looks almost the same as the previous one: .BLANK.NO JUSTIFY.NO FILL.TEST PAGE 12 DEFINE PROCEDURE LOGIN-SUMMARY READY BASE-LOGIN REPORT BASE-LOGIN WITH CODE=19 SORTED BY DEPT, NAME, DATE, TIMR ON LOGINSUM.RPT SET COLUMNS-PAGE=80 SET REPORT-NAME = "Login Summary" AT BOTTOM OF NAME PRINT "Number of logins for ", NAME, SPACE 1, COUNT AT BOTTOM OF DEPT PRINT SKIP, "Number of logins for Department ", DEPT, SPACE 1, COUNT, SKIP 2 END-REPORT FINISH END-PROCEDURE .BLANK.JUSTIFY.FILL As may be seen, the only significant difference is that there is no unqualified PRINT statement, so the details do not print. It is not necessary to print details in order to have breaks on a field, or to summarize (COUNT) on a field. This report looks like this: .BLANK.NO JUSTIFY.NO FILL.TEST PAGE 11 Login Summary 12-Sep-83 Page 1 NAME DEPT Number of logins for LEDERMAN 1 Number of logins for ROACH 17 Number of logins for Department PNA 18 .JUSTIFY.FILL.PARAGRAPH So far, I consider what I did to be mostly re-shaping the DEC-supplied command file to a form I consider more acceptable in terms of efficiency or ease of use. I then started to consider making changes which would give me either reports not supplied by DEC, or which would improve performance. The first change is improving the retrieval of data from the data file. It every case, there is a selection of the CODE field, to extract those records of the type under consideration: with the existing sequential file, it is necessary to read the entire data file completely to find the records wanted for each report. It would be more efficient to put the data in an indexed file, with CODE as the primary key: then any reference to the data would quickly retrieve only those records with the correct CODE for that report. While it would be possible to read the sequential file and write to an indexed file with Datatrieve, the RMS utility IFL will do the same job, and is several orders of magnitude faster for this kind of operation. What I did was use the RMS DEF utility to define an appropriate file: the command file to create the data file is as follows. .BLANK.NO JUSTIFY.NO FILL.TEST PAGE 36 ;THE FIRST QUESTION ASKS FOR THE FILE SPECIFICATION. ; the name is ACCOUNT.DOM, and will be created ACCOUNT.DOM YES ;THE NEXT QUESTIONS DEAL WITH FILE ORGANIZATION & RECORD ATTRIBUTES ; an indexed file, fixed length records 120 bytes long ; carriage control IDX FIX 120 YES ;THE FOLLOWING QUESTIONS DEAL WITH KEYS ; the key is an integer, two bytes long named CODE ; and duplicates are allowed INT 0 2 CODE YES NO ;THE NEXT QUESTIONS DEAL WITH ALLOCATION AND PLACEMENT ATTRIBUTES ; no placement control, initial allocation 30 blocks, extend 5 blocks NO NO 30 5 NO ;THE NEXT QUESTIONS ASK ABOUT FILL SIZES FOR KEYS ; fill the areas completely (there will be no future insertions) 512 512 ;THE FOLLOWING QUESTIONS DEAL WITH FILE PROTECTION RWED RWED RWED RWED .BLANK.JUSTIFY.FILL The data can then be inserted in the file with the following commands (which assume the RMS utilities are installed under their normal names): .BLANK SHOW ACCOUNTING/DATATRIEVE:ACNTRN.SYS ACCOUNT.SEQ .BREAK DEF @ACCOUNT.DEF (the definition file given above) .BREAK IFL ACCOUNT.DOM=ACCOUNT.SEQ .BLANK It is true that it takes some time to insert the records in the indexed file, but from then on time is saved by the faster access, so if several reports are to be generated, or the data is going to be examined to find specific events, the net savings justifies creating the indexed file. No change is necessary in any of the record definitions or procedures, but the domain definition reference to ACCOUNT.DAT should be changed to ACCOUNT.DOM to access the new data file. .PARAGRAPH One of the reports I wanted to generate was for device mounts, dismounts, etc. The DEC-supplied reports give only one operation per report, and I wanted all mounts, dismounts, allocations, and de-allocations to be in a single report, and to be identified by name rather than by code number. To do this, I first created a domain with all of the operation codes. .BLANK.NO JUSTIFY.NO FILL DEFINE RECORD CODES-REC USING 01 CODES-REC. 03 KEY PIC 99 USAGE COMP. 03 TYPE PIC X(16). ; DEFINE DOMAIN CODES USING CODES-REC ON CODES.DOM; .BREAK DEFINE FILE FOR CODES KEY=KEY; .BLANK.JUSTIFY.FILL Note that CODES uses an indexed file, so the data can be quickly retrieved by the KEY field. The data in CODES looks like this: .BLANK.NO JUSTIFY.NO FILL.TEST PAGE 19 KEY TYPE 01 System 02 User 03 Task 09 Startup 10 Invalid Login 11 Time Change 12 Allocate 13 Deallocate 14 Mount 15 Dismount 16 Print 19 Login 20 Crash 21 Device Stats 22 Reset 27 Card Reader .BLANK.JUSTIFY.FILL This gives me a domain with all of the valid code types and their english translation: at the moment, I'm only using 12, 13, 14 and 15, but it was just as easy to put them all in, in case I want them later. It is also easy to ready and print CODES whenever I forget the number of a particular transaction type. Next, I defined something called a VIEW, which is simply a way to combine the information from more than one domain into what looks like a single domain. .BLANK.NO JUSTIFY.NO FILL.TEST PAGE 15 DEFINE DOMAIN AM OF BASE-AM, CODES USING 01 AM OCCURS FOR BASE-AM WITH CODE=12,13,14,15. 10 TYPES OCCURS FOR CODES WITH KEY=CODE. 20 TYPE FROM CODES. 10 DEPT FROM BASE-AM. 10 NBR FROM BASE-AM. 10 TERMINAL FROM BASE-AM. 10 ACNT FROM BASE-AM. 10 DATE FROM BASE-AM. 10 TIM FROM BASE-AM. 10 TIMR FROM BASE-AM. 10 TIME FROM BASE-AM. 10 DEVICE FROM BASE-AM. ; .BLANK.JUSTIFY.FILL The first line simply states that I am combining data from BASE-AM and CODES. The next means I only want data with a CODE of 12, 13, 14 or 15, which are the transactions I am interested in. By doing this, I will get only the proper transactions, and I don't have to remember to select the proper codes each time I access the domains, which is a great convenience. The next line is the one which matches up the code in BASE-AM with the corresponding record in CODE: thus the TYPE on the next line will contain the proper text for that transaction type (Mount, Deallocate, etc.). The rest of the view is simply the information I want to carry over from BASE-AM. Now, when AM is printed, each line will contain the proper TYPE for that transaction, and all four types of transactions, and only those transactions, will appear. This is very handy not only for reports, but also for direct query using Datatrieve: one can go in and print out the transactions for a particular user, or device, or combination. Having a view which automatically selects the proper records, so that one does not have to enter the "WITH CODE=nn" each time, is such a convenience that I created several other views for those transactions that I expect to be accessing for inquiries, such as Invalid login transactions, login transactions, print queue transactions, and user activity. Other transactions, which I expect to use only in reports, do not need a view as the "CODE=nn" can be fixed in the report procedure, and there is some extra overhead in using a view. Reporting a view is also possible: .BLANK.NO JUSTIFY.NO FILL.TEST PAGE 10 DEFINE PROCEDURE AM-REPORT READY AM REPORT AM SORTED BY DATE, TIMR ON AM.RPT SET COLUMNS-PAGE=80 SET REPORT-NAME="Allocate / Mount"/"De-allocate / Dismount" PRINT TYPES, DEPT, TERMINAL, DATE, DEVICE END-REPORT FINISH END-PROCEDURE .BLANK.JUSTIFY.FILL A sample of the generated report follows. .BLANK.NO JUSTIFY.NO FILL.TEST PAGE 20 Allocate / Mount 12-Sep-83 De-allocate / Dismount Page 1 TYPE DEPT TERMINAL DATE DEVICE Mount $SY CO0 8-Sep-83 DU0 Mount $SY CO0 8-Sep-83 MT0 Mount $SY CO0 8-Sep-83 MT1 Mount $SY CO0 8-Sep-83 DB1 Dismount SYS TT7 9-Sep-83 DU0 Mount SYS TT7 9-Sep-83 DU0 Dismount SYS TT16 9-Sep-83 MT0 Dismount SYS TT16 9-Sep-83 MT1 Allocate SYS TT16 9-Sep-83 MT1 Mount SYS TT16 9-Sep-83 MT1 Deallocate SYS TT16 9-Sep-83 MT1 Dismount SYS TT16 9-Sep-83 MT1 Mount SYS TT16 9-Sep-83 MT1 .BLANK.JUSTIFY.FILL Note that there is no "CODE=" qualifier in the REPORT command, but it is now necessary to sort by date and time, because the insertion of the data into the indexed file sorts the data by code only. It would have been very helpful if some uniformity in the record layout could have been followed by DEC, but as it is, it appears that a different team developed the packet for each different transaction, and they didn't talk to each other. Thus, fields which are common to all (or almost all) transactions, such as the date and time, appear in entirely different places in different records, and there doesn't seem to be anything the user can do about it now. It should also be noted that I am sorting on TIMR and not TIME: TIME is a COMPUTED-BY field, and Datatrieve-11 cannot sort a COMPUTED-BY field. I also like to create reports which are 80 columns wide, when possible, rather than 132 columns, as the paper is easier to handle, and it looks better than a few fields spread across a wide page with a lot of blank space between them. .PARAGRAPH My next major effort was with the device statistics. We have systems which place a heavy load on disk transactions, and balancing the load between disks appeared to be a way to improve throughput, and one of the reasons we went to M-Plus was to get the statistics to find out just what was happening on the disks. When I looked at the device report, however, I was a little disappointed to find the statistics recorded as running totals, rather than the total for the last scan interval. I can understand the reason for it, but I want the reports to show me what happened in the past hour (or other survey period), and not show me hours when nothing happened. It is possible for Datatrieve to do this through the use of an intermediate domain to store temporary data, the definitions for which are as follows: .BLANK.NO JUSTIFY.NO FILL.TEST PAGE 12 DEFINE RECORD DEVICE-SUM-REC USING 01 DEVICE-SUM-REC. 10 DATE PIC X(9). 10 TIME PIC 9(6) EDIT-STRING 99,99,99. 10 DEVICE PIC X(6). 10 IO-SUM PIC 9(9) USAGE COMP EDIT-STRING ZZZ,ZZZ,ZZ9 QUERY-HEADER "I/O COUNT". 10 WORD-SUM PIC 9(9) USAGE COMP EDIT-STRING ZZZ,ZZZ,ZZ9 QUERY-HEADER "WORD COUNT". 10 CYL-SUM PIC 9(9) USAGE COMP EDIT-STRING ZZZ,ZZZ,ZZ9 QUERY-HEADER "CYLINDERS"/"CROSSED". ; .BLANK DEFINE DOMAIN DEVICE-SUM USING DEVICE-SUM-REC ON DEVSUM.SEQ; .BLANK.JUSTIFY.FILL This domain simply serves as a temporary storage place for the particular fields I want to appear in the final report. The procedure to do the actual conversion is: .BLANK.NO JUSTIFY.NO FILL.TEST PAGE 11 DEFINE PROCEDURE DEVICE-SUM-REPORT DECLARE IO PIC 9(9) USAGE COMP. DECLARE WORD PIC 9(9) USAGE COMP. DECLARE CYL PIC 9(9) USAGE COMP. IO = 0 WORD = 0 CYL = 0 DEFINE FILE FOR DEVICE-SUM; READY DEVICE-SUM WRITE READY BASE-DEVICE .TEST PAGE 14 FOR BASE-DEVICE WITH CODE = 21 SORTED BY DEVICE, DATE, TIMR BEGIN STORE DEVICE-SUM USING BEGIN DATE = DATE TIME = TIME DEVICE = DEVICE IO-SUM = IO-COUNT - IO WORD-SUM = WORD-COUNT - WORD CYL-SUM = CYL-CROSSED - CYL END IO = IO-COUNT WORD = WORD-COUNT CYL = CYL-CROSSED END FINISH BASE-DEVICE RELEASE IO RELEASE WORD RELEASE CYL .TEST PAGE 10 REPORT DEVICE-SUM WITH IO-SUM > 0 ON DEVSUM.RPT SET COLUMNS-PAGE=80 SET REPORT-NAME = "Device Statistics Summary" PRINT DEVICE-SUM-REC AT BOTTOM OF DEVICE PRINT SKIP, "Maximums:", MAX(IO-SUM), MAX(WORD-SUM), MAX(CYL-SUM), SKIP 2 END-REPORT FINISH END-PROCEDURE .BLANK.JUSTIFY.FILL The procedure simply goes through the device statistics data (sorted in the order I want the report, which is by device, date, and time), subtracts the current hour from the previous hour, and stores the difference. This leaves DEVICE-SUM containing the statistics for each hour sorted by the device, which is exactly what I want in the final report. The DEFINE#FILE statement ensures that I start with a new data file, rather than picking up the data left over from the last time I ran the report: it is faster than deleting the old data, but it is necessary to remember to purge the old DEVSUM.SEQ files occasionally. Also note that it is necessary to declare all variables which are to be used, and that they must be initialized to zero. Once the data is converted, only the records where the activity is greater than zero will be reported to eliminate hours when nothing happened. The report looks like this (some detail lines were deleted just to save space, so the maximums reflect lines not on this page): .BLANK.NO JUSTIFY.NO FILL .PAGE Device Statistics Summary 12-Sep-83 Page 1 CYLINDERS DATE TIME DEVICE I/O COUNT WORD COUNT CROSSED 11-Sep-83 09,13,19 DB0 125,060 32,017,672 1,784,389 11-Sep-83 10,13,19 DB0 125,002 32,005,200 1,778,782 11-Sep-83 11,13,19 DB0 124,997 32,001,544 1,778,216 11-Sep-83 12,13,19 DB0 94,077 24,120,632 1,357,430 11-Sep-83 13,13,19 DB0 241 61,696 26,304 11-Sep-83 14,13,19 DB0 241 61,696 26,304 11-Sep-83 15,13,19 DB0 241 61,696 26,304 Maximums: 297,604 122,120,108 6,326,075 11-Sep-83 07,13,19 DB1 124,054 31,757,824 2,204,479 11-Sep-83 08,13,19 DB1 124,077 31,763,712 2,215,187 11-Sep-83 09,13,19 DB1 124,139 31,779,584 2,204,196 11-Sep-83 10,13,19 DB1 124,060 31,759,360 2,224,216 11-Sep-83 11,13,19 DB1 124,106 31,771,136 2,220,454 11-Sep-83 12,13,19 DB1 93,336 23,894,016 1,657,114 Maximums: 124,358 247,766,154 2,951,050 11-Sep-83 09,13,19 DU0 160,856 41,179,136 0 11-Sep-83 10,13,19 DU0 160,811 41,167,616 0 11-Sep-83 11,13,19 DU0 160,683 41,134,848 0 11-Sep-83 12,13,19 DU0 120,714 30,902,784 0 Maximums: 160,856 41,179,136 0 11-Sep-83 09,13,19 MT1 28,138 9,540,096 0 11-Sep-83 10,13,19 MT1 28,122 9,534,464 0 11-Sep-83 11,13,19 MT1 28,109 9,529,856 0 Maximums: 28,138 9,540,096 0 .BLANK.JUSTIFY.FILL Users should note that this procedure will issue a warning message, "Proceeding to report unsorted records", because there is an "AT#BOTTOM" statement for a field for which there is no sort parameter in the REPORT command. This is a warning message only, and may be ignored as the file being reported is a sequential file which is already sorted in the proper order for the desired report. It may also be noted that in this particular case, since DEVICE-SUM is always written in order, and reported in that same order, it can use a sequential file to store data, and no performance advantage would be obtained by making DEVSUM.SEQ into an indexed file. .PARAGRAPH I found an interesting statistic the first time I ran this report, which was that the minimum disk activity for the system disk never went below 240 QIO's, even at night when I knew no one was on the system. I concluded that this is the activity of the system accounting package itself, accumulating data. This activity was measured without task accounting, and without DECnet, with a 1 hour scan interval: changes to any of these factors would change the base level of activity. If this activity was not wanted on the final report, the qualifier "WITH#IO-SUM#>#0" could be changed to one which would reject activity below a given level. .PARAGRAPH Another report not provided by DEC, but one I think will be of interest, shows a calculated charge amount for each user and department based on the system resources used. The starting point for this is the User Account transactions record, with an additional charge field: .BLANK.NO JUSTIFY.NO FILL.TEST PAGE 48 DEFINE RECORD CHARGE-REC USING 01 CHARGE-REC. 10 CODE PIC 99 USAGE COMP. 10 DEPT PIC XXX. 10 NBR PIC X(7). 10 DEVICE PIC X(6). 10 ACNT PIC 9(4) USAGE COMP. 10 UIC PIC X(10). 10 LOGIN-DATE USAGE DATE. 10 LITIM. 20 FILLER PIC X(6). 10 LITMR REDEFINES LITIM. 20 LIH PIC 99 USAGE COMP. 20 LIM PIC 99 USAGE COMP. 20 LIS PIC 99 USAGE COMP. 20 LID PIC 9(18) COMPUTED BY (0.0000001 * LOGIN-DATE). 10 LOGIN-TIME PIC 9(6) COMPUTED BY (LIH * 10000) + (LIM * 100) + LIS EDIT-STRING 99,99,99. 10 LOGOUT-DATE USAGE DATE. 10 LOTIM. 20 FILLER PIC X(6). 10 LOTMR REDEFINES LOTIM. 20 LOH PIC 99 USAGE COMP. 20 LOM PIC 99 USAGE COMP. 20 LOS PIC 99 USAGE COMP. 10 LOGOUT-TIME PIC 9(6) COMPUTED BY (LOH * 10000) + (LOM * 100) + LOS EDIT-STRING 99,99,99. 10 BILLING-DATE USAGE DATE. 10 BITIM. 20 FILLER PIC X(6). 10 BITMR REDEFINES BITIM. 20 BIH PIC 99 USAGE COMP. 20 BIM PIC 99 USAGE COMP. 20 BIS PIC 99 USAGE COMP. 20 BID PIC 9(18) COMPUTED BY (0.0000001 * BILLING-DATE). 10 BILLING-TIME PIC 9(6) COMPUTED BY (BIH * 10000) + (BIM * 100) + BIS EDIT-STRING 99,99,99. 10 ELAPSED PIC 9(12) EDIT-STRING ZZZ,ZZ9 COMPUTED BY (BID - LID). 10 CPU-TIME PIC 9(9) USAGE COMP EDIT-STRING ZZZ,ZZZ,ZZ9. 10 TASKS-ACTIVE PIC 999 USAGE COMP EDIT-STRING ZZ9. 10 TASKS-RUN PIC 9(9) USAGE COMP EDIT-STRING ZZZ,ZZZ,ZZ9. 10 DIRECTIVES PIC 9(9) USAGE COMP EDIT-STRING ZZZ,ZZZ,ZZ9. 10 QIOS PIC 9(9) USAGE COMP EDIT-STRING ZZZ,ZZZ,ZZ9. 10 FILLER PIC X(30). 10 CHARGES PIC 99999V99 COMPUTED BY (CPU-TIME * 0.001) + (DIRECTIVES * 0.0003) + (QIOS * 0.0008) EDIT-STRING $$,$$$.Z9. ; .BLANK.JUSTIFY.FILL Most of these fields are not needed in the report I generate, but I have left them in, should I want them later for a different report. The field of importance is the CHARGES, which uses the Datatrieve COMPUTED#BY feature much as the time was computed earlier: in this case, I have decided to charge for use of the CPU and for all directives and QIOs generated, but the charge could also be based on the number of tasks run, etc. I have also added three other fields, LID, computed from the LOGIN-DATE, and BID, computed from the BILLING-DATE, and ELAPSED, which is the difference between the two. Normally, Datatrieve-11 puts the date and not the time in DATE fields, but the system accounting package actually fills in the complete date and time (in clunks) when it stores the data. These can be scaled down from clunks to seconds as I have done here, and can then be used to calculate the field ELAPSED, which is the amount of time from when the user logged in to when either the user logged out or system accounting was shut down, in seconds. Using these date fields ensures that the proper elapsed time will be calculated even if the period a user was logged in covers more than one day: simply subtracting the logout time from the login time would not take the date change into account. If LOGOUT-DATE had been used instead of BILLING-DATE, then any one who was logged in when the system was shut down (for example, the system manager must be logged in to stop accounting or shut down the system) would have a LOGOUT-DATE of zero, and the times would not be calculated properly. This field could be used to charge for the amount of time a user was logged on the system by including it in the computation of CHARGES; I simply decided not to charge for it in my example. One other minor change which may be noted is in the LITIM, LOTIM and BITIM groups, where I used one FILLER which is 6 charaters long to skip over the three time fields: it is not necessary for FILLER to be the same type of field at that which it replaces, nor for there to be the same number of fields; it is only necessary that the FILLER be the proper total length. The report generator is very much like the other reports: .BLANK.NO JUSTIFY.NO FILL.TEST PAGE 15 DEFINE PROCEDURE CHARGE2-REPORT READY CHARGE REPORT CHARGE WITH CODE=02 SORTED BY DEPT, UIC ON CHARGE2.RPT SET COLUMNS-PAGE=78 SET REPORT-NAME = "System Charges by Department"/"and User" AT BOTTOM OF UIC PRINT "USER", SPACE 1, UIC(" "), TOTAL(CPU-TIME) USING ZZZ,ZZZ,ZZ9., TOTAL(DIRECTIVES) USING ZZZ,ZZZ,ZZ9., TOTAL(QIOS) USING ZZZ,ZZZ,ZZ9., TOTAL(CHARGES) USING $$$,$$$,$$9.99, SKIP AT BOTTOM OF DEPT PRINT SKIP 1, "DEPT", SPACE 1, DEPT(" "), TOTAL(CPU-TIME) USING ZZZ,ZZZ,ZZ9., TOTAL(DIRECTIVES) USING ZZZ,ZZZ,ZZ9., TOTAL(QIOS) USING ZZZ,ZZZ,ZZ9., TOTAL(CHARGES) USING $$$,$$$,$$9.99, SKIP 2 END-REPORT FINISH END-PROCEDURE .BLANK.JUSTIFY.FILL The report generated is: .BLANK.NO JUSTIFY.NO FILL.TEST PAGE 34 System Charges by Department 13-Sep-83 and User Page 1 CPU TIME DIRECTIVES QIOS CHARGES USER [001,001] 380,324. 1,080,404. 185,690. $852.99 USER [200,001] 226,423. 975,196. 142,910. $633.30 DEPT 606,747. 2,055,600. 328,600. $1,486.30 USER [000,000] 1,871,658. 17,319,746. 1,992,107. $8,661.26 DEPT $SY 1,871,658. 17,319,746. 1,992,107. $8,661.26 USER [300,002] 22,600. 55,529. 31,486. $64.44 USER [300,003] 175. 510. 161. $0.45 DEPT PNA 22,775. 56,039. 31,647. $64.90 USER [001,001] 12,987,024. 117,267,216. 30,962,109. $72,936.87 DEPT SYS 12,987,024. 117,267,216. 30,962,109. $72,936.87 .BLANK.JUSTIFY.FILL One could easily have each department printed on separate pages, or print department totals only, or include the details and give the charges for each login session. We don't charge users on our system, so I have not done much work on these reports, but I'm now prepared if we do start charging. .PARAGRAPH When working with the above reports, I noticed that although system accounting records all jobs on print queues, it does not record jobs on batch queues. I also became curious about what statistics could be accumulated with Datatrieve without using the system accounting package. Since the start and end of each batch job is logged on the system console, and the console logging package also allows this information to also be stored in a file on disk, I decided to try to read this file with Datatrieve and see what information I could retrieve. Typically, a console log would contain information such as this: .BLANK.NO JUSTIFY.NO FILL.TEST PAGE 17 08:06:17 COT -- Date is 26-SEP-83 08:06:17 ERRLOG -- Error Logging initialized 08:13:07 Login user ROACH [300,2] TT16: 08:15:23 Login user SYSTEM [1,1] TT3: 09:03:52 *** MT0: -- Dismount complete 09:03:52 Logout user ROACH [300,2] TT17: 09:04:20 Logout user ROACH [300,2] TT16: 09:19:22 Login user ROACH [300,2] TT17: 15:34:30 Batch job - BACKUP started on VT1: 15:34:40 Login user SYSTEM [1,1] HT0: 15:35:18 Logout user SYSTEM [1,1] HT0: 15:45:25 *** DB1: -- Dismount complete 15:45:27 Batch job - BACKUP completed on VT1: 15:45:40 Login user SYSTEM [1,1] TT16: 15:52:19 ERRLOG -- Error Logging stopped 15:52:46 ERRLOG -- Error Logging initialized .BLANK.JUSTIFY.FILL Note the dismount messages: on the console, they line up with the other records, but the first two characters in the record are nulls, and these must be counted in the record definition. Because Datatrieve prefers fixed-length records, and some of the other console records can be very long (especially DECnet and task-termination messages), I decided to first move the data from the log file to one with fixed-length records by defining a file with RMSDEF, as I did for system accounting: .BLANK.NO JUSTIFY.NO FILL.TEST PAGE 20 ; The first question asks for the file specification. CONSOLE.SEQ NO ; The next questions deal with file organization _& record attributes FIX 56. ; The next questions deal with allocation and placement attributes 30 10 ; The following questions deal with file protection RWED RWED RWED R .BLANK.JUSTIFY.FILL Data is moved from one file to the other with the command: .BLANK CNV#CONSOLE.SEQ/AP/TR/PD:40#=#CONSOLE.LOG .BLANK The /AP (append) switch places the new data at the end of the CONSOLE.SEQ file and must be present even if CONSOLE.SEQ is empty, and both the /TR (truncate) and /PD (pad) switches must be present. Now that the data is in fixed-length records, I can write a record definition for the fields I want: .BLANK.NO JUSTIFY.NO FILL.TEST PAGE 40 DEFINE RECORD CON-REC USING 01 CON-REC. 10 BASE. 20 TEXT PIC X(56). 10 COT REDEFINES BASE. 20 COTIME PIC X(8). 20 FILLER PIC X(2). 20 COTID PIC X(12). 20 FILLER PIC X(3). 20 COTDATE PIC X(9). 10 LOGIN REDEFINES BASE. 20 LITIM PIC X(8). 20 FILLER PIC XX. 20 LOGID PIC X(6). 20 FILLER PIC X(6). 20 LOGNAM PIC X(14). 10 DMO REDEFINES BASE. 20 FILLER PIC XX. 20 DMOTIM PIC X(8). 20 FILLER PIC X(6). 20 DMODEV PIC X(4). 20 FILLER PIC X(5). 20 DMOID PIC X(8). 10 BATCH REDEFINES BASE. 20 BATIM PIC X(8). 20 FILLER PIC XX. 20 BATID PIC X(5). 20 FILLER PIC X(7). 20 BATNAM PIC X(9). 20 FILLER PIC X. 20 BATTYP PIC X(7). 20 FILLER PIC X(4). 20 DEVICE PIC X(6). 20 D1 REDEFINES DEVICE. 30 DEV1 PIC X(4). 20 D2 REDEFINES DEVICE. 30 FILLER PIC XX. 30 DEV2 PIC X(4). ; .JUSTIFY.FILL.BLANK This definition looks very much like one of the DEC-supplied system accounting definitions. In this case, the different pages must all be in one definition as there is no way to tell one record from another, except by examining the contents of a field within the record, which I will show in the following procedure. I decided the next step would be to read this file, and create another which would have a more uniform field layout, and which would have the date in every record. I therefore defined a second domain into which the console log data will be placed. .BLANK.NO JUSTIFY.NO FILL.TEST PAGE 9 DEFINE RECORD C-REC USING 01 C-REC. 10 DATE USAGE IS DATE. 10 TIME PIC X(8). 10 TYPE PIC X(12). 10 DEVICE PIC X(4). 10 TEXT PIC X(16) QUERY-HEADER IS " ". ; .JUSTIFY.FILL.BLANK This simple definition puts the fields in fixed locations, which will simplify processing it later. To get the data in, one must read the file created above, and store the data in the new domain. .NO JUSTIFY.NO FILL.BLANK DEFINE PROCEDURE CONVERT READY CONSOLE READ READY C WRITE DECLARE S-DATE USAGE IS DATE. FOR CONSOLE BEGIN IF COTID = "COT -- Date " S-DATE = COTDATE IF LOGID = "Login ", "Logout" STORE C USING BEGIN DATE = S-DATE TIME = LITIM TYPE = LOGID TEXT = LOGNAM END IF DMOID = "Dismount" STORE C USING BEGIN DATE = S-DATE TIME = DMOTIM TYPE = "Dismount" DEVICE = DMODEV END IF BATID = "Batch" STORE C USING BEGIN DATE = S-DATE TIME = BATIM IF BATTYP = "started" BEGIN DEVICE = DEV1 TYPE = "Batch Start" END IF BATTYP = "complet" BEGIN DEVICE = DEV2 TYPE = "Batch Stop" END TEXT = BATNAM END END FINISH END-PROCEDURE .BLANK.JUSTIFY.FILL Note that each IF statement is required in order to determine what type of record is involved: this is why all of the pages must be present in one record definition. The two device fields are required in the batch record because they appear in different positions in those two records. The temporary field S-DATE is used to hold the date whenever COT logs a date change, so that all records which follow will have the current date. The C domain now contains data in a uniform format. .BLANK.NO JUSTIFY.NO FILL DATE TIME TYPE DEVICE 21-Sep-83 09:47:12 Logout SCOTT 21-Sep-83 09:48:25 Dismount MT1: 21-Sep-83 09:52:15 Batch Stop VT1: BRU 21-Sep-83 09:53:41 Dismount DB1: 21-Sep-83 09:53:47 Dismount DU0: 21-Sep-83 09:57:44 Login SCOTT 21-Sep-83 09:57:56 Batch Start VT1: BRUDB1 21-Sep-83 09:59:08 Logout SYSTEM 21-Sep-83 10:05:50 Login ROACH 21-Sep-83 10:10:05 Login SYSTEM 21-Sep-83 10:11:08 Logout ROACH 21-Sep-83 10:14:45 Login ROACH 21-Sep-83 10:18:15 Dismount DB1: 21-Sep-83 10:18:16 Dismount DU0: 21-Sep-83 10:18:24 Batch Stop VT1: BRUDB1 21-Sep-83 10:22:41 Login ANALYSIS 21-Sep-83 10:26:52 Logout ANALYSIS .BLANK.JUSTIFY.FILL With the data in this condition, it is much easier to examine it, or create reports. The only report I am interested in at the present concerns the batch jobs: .BLANK.NO JUSTIFY.NO FILL.TEST PAGE 9 DEFINE PROCEDURE BATCH-REPORT READY C READ REPORT C WITH TYPE CONT "Batch" ON BATCH.RPT SET REPORT-NAME = "Batch Jobs" SET COLUMNS-PAGE = 80 PRINT C-REC END-REPORT END-PROCEDURE .BLANK.JUSTIFY.FILL The report is: .BLANK.NO JUSTIFY.NO FILL.TEST PAGE 18 Batch Jobs 23-Sep-83 Page 1 DATE TIME TYPE DEVICE 17-Sep-83 00:00:24 Batch Start VT1: BRU 17-Sep-83 00:02:53 Batch Stop VT1: BRU 20-Sep-83 13:46:02 Batch Start VT1: BRU 20-Sep-83 13:46:18 Batch Stop VT1: BRU 20-Sep-83 13:46:44 Batch Start VT1: BRU 20-Sep-83 13:56:54 Batch Stop VT1: BRU 21-Sep-83 08:21:21 Batch Start VT1: BRU 21-Sep-83 09:41:22 Batch Start VT1: BRU 21-Sep-83 09:52:15 Batch Stop VT1: BRU 21-Sep-83 09:57:56 Batch Start VT1: BRUDB1 21-Sep-83 10:18:24 Batch Stop VT1: BRUDB1 .BLANK.JUSTIFY.FILL By adding more pages to the record definition, it would also be possible to extract task-termination messages, error-log warnings that a device has exceeded it's hard-error limit, or any other message which appears on the console. This technique may be useful for RSX-11M systems which do not have system accounting. .PARAGRAPH At this point, the reader will hopefully have a better idea of the type of information and reports which may be obtained using Datatrieve to process the system accounting information. It is my belief that the information gathered is potentially of considerable value to a system manager, and that Datatrieve is the easiest way to report this information in such a way as to be of greatest use. I have attempted to show how the reports can be re-formatted to suit the individual site, and to extract information in ways not provided for in the package supplied by DEC, and it is hoped that readers will now be better prepared to tailor the system accounting reports suit their own applications. The author welcomes comments or suggestions about this paper and the material presented.