ANU News Patch Patch ID: 930302_newsfiles.c!saul@hnrc.tufts.edu Date: 2-Mar-1993 Author: Saul Tannenbaum saul_sy@hnrc.tufts.edu News Version: 6.1beta6 News Files: NewsFiles.C,NewsDelete.C Description: A number of messages to this group have requested a "Turbo" skim command for NEWS, in order to speed up processing. While a full Turbo-Skim isn't possible, it is possible to use a number of the RMS performance options to speed up processing of NEWS and, in particular, the SKIM run. The following is a description of the effect of the patches and an important warning about their use: These patches are controlled by 3 logical names: ANU_NEWS_ITEMFILE_BUFFERCOUNT ANU_NEWS_GROUPFILE_BUFFERCOUNT ANU_NEWS_HISTORYFILE_BUFFERCOUNT These logicals are translated via normal system routines, so may be defined in any system, group, job, or process table, or may be defined differently for different contexts. If these patches are installed, and those logicals are not defined, NEWS behaves as before. If these patches are installed, and the logicals ARE defined, the following happens to RMS processing of the NEWS.ITEMS, NEWS.GROUPS, and HISTORY.V60 files: 1) RMS deferred write is turned on - Normally, RMS will write a buffer as soon as it is modified. Deferred write delays the writing of the buffer until RMS needs to reuse the buffer space. 2) The file is opened with "cathedral windows", mapping all extents and obviating window turns. 3) If the logical translates to a valid integer, that value is used as the RMS multibuffering count. *WARNING* *WARNING* RMS uses a very conservative approach to file and buffer management, and for good reasons. If you define these logicals, and are, for example, running a SKIM when your system crashes, you will most likely corrupt your NEWS.ITEMS file, as there will be many modified buffers that will be unwritten. Life is full of tradeoffs. This is but one of them. Also, if you define these logicals to be large numbers, be sure the accounts that will be using them have sufficient process quota to make good use. Locks are taken out for each buffer - be sure that ENQLM is large enough, NEWS dies horribly if you run out of locks. Also, be sure that there is a large enough working set to contain these buffers. *ADVICE* The biggest, and safest, advantage I've seen from these patches is to be sure that the NEWS.GROUPS file is entirely buffered throughout a SKIM run. As no new records are written to NEWS.GROUPS, only item counts changed, this file is safe from corruption. Using SET FILE/STATISTICS and MONITOR RMS to see the effect, if any, of this tuning is well advised. As always, use at your own risk. Feedback and further suggestions are encouraged. *** newsfiles_orig.c --- newsfiles.c ************** *** 28,33 ** V6.1b6 28-Feb-1993 mark.martinec@ijs.si ** - fix strncmp match for generic TCP (broken by V6.0-4, ** so that /PROTOCOL=TCP matched TCPWARE instead of generic TCP) ** V6.1b7 27-May-93 munroe@dmc.com ** rms$errno isn't returned properly from most of the rms_ routines. I ** need the information in order to debug something that is preventing --- 28,36 ----- ** V6.1b6 28-Feb-1993 mark.martinec@ijs.si ** - fix strncmp match for generic TCP (broken by V6.0-4, ** so that /PROTOCOL=TCP matched TCPWARE instead of generic TCP) + ** V6.1b6 2-Mar-1993 saul@hnrc.tufts.edu + ** - RMS optimizations + ** (see FAST_SKIM.PATCH / 930302_NEWSFILES.PATCH for details) ** V6.1b7 27-May-93 munroe@dmc.com ** rms$errno isn't returned properly from most of the rms_ routines. I ** need the information in order to debug something that is preventing ************** *** 103,108 } i->r = cc$rms_rab; i->r.rab$l_fab = &(i->f); if ((status=sys$open(&(i->f))) & 1) { if ((status=sys$connect(&(i->r))) & 1) return(i); --- 106,116 ----- } i->r = cc$rms_rab; i->r.rab$l_fab = &(i->f); + + /* ST 2/2/93 - turn on multibuffering and read ahead */ + i -> r.rab$l_rop |= RAB$M_RAH; + i -> r.rab$b_mbf = 2; + if ((status=sys$open(&(i->f))) & 1) { if ((status=sys$connect(&(i->r))) & 1) return(i); ************** *** 165,170 i->r = cc$rms_rab; i->r.rab$l_fab = &(i->f); sysprv(); if ((status=sys$create(&(i->f))) & 1) { if ((status=sys$connect(&(i->r))) & 1) { --- 173,182 ----- i->r = cc$rms_rab; i->r.rab$l_fab = &(i->f); + /* ST 2/2/93 - turn on multibuffering and write behind */ + i -> r.rab$l_rop |= RAB$M_WBH; + i -> r.rab$b_mbf = 2; + sysprv(); if ((status=sys$create(&(i->f))) & 1) { if ((status=sys$connect(&(i->r))) & 1) { ************** *** 363,368 } } else { kid_valid = 0; if (init_lock()) exit(1); kid_valid = 1; --- 375,390 ----- } } else { + /* ST 2/2/93 RMS Performance Patch */ + int buff_count=0; + char *buff_count_ptr; + int item_performance = FALSE; + buff_count_ptr = news_getenv("ANU_NEWS_ITEMFILE_BUFFERCOUNT",0); + if (buff_count_ptr != NULL) + { + sscanf(buff_count_ptr,"%d",&buff_count); + item_performance = TRUE; + } kid_valid = 0; if (init_lock()) exit(1); kid_valid = 1; ************** *** 372,377 itmfab.fab$l_fna = ITM_FILENAME; itmfab.fab$b_fns = strlen(itmfab.fab$l_fna); itmfab.fab$l_fop = FAB$M_CIF; itmfab.fab$w_mrs = sizeof newsitm; itmfab.fab$b_org = FAB$C_IDX; itmfab.fab$b_rat = FAB$M_CR; --- 394,407 ----- itmfab.fab$l_fna = ITM_FILENAME; itmfab.fab$b_fns = strlen(itmfab.fab$l_fna); itmfab.fab$l_fop = FAB$M_CIF; + + /* ST 2/2/93 if the buffer count logical was defined, turn on + a number of RMS performance features */ + if (item_performance) + { + itmfab.fab$l_fop |= FAB$M_DFW; /* deferred write */ + itmfab.fab$b_rtv = 255; /* map the entire file */ + } itmfab.fab$w_mrs = sizeof newsitm; itmfab.fab$b_org = FAB$C_IDX; itmfab.fab$b_rat = FAB$M_CR; ************** *** 409,414 itmrab.rab$l_rbf = (char *) &newsitm; itmrab.rab$w_rsz = sizeof newsitm; sysprv(); if (!sys_open_nofnf(&itmfab)) { #ifndef V58_UPGRADE_MANUAL --- 439,448 ----- itmrab.rab$l_rbf = (char *) &newsitm; itmrab.rab$w_rsz = sizeof newsitm; + /* ST 3/2/93 - use up to 127 buffers for the item file */ + if (item_performance) + itmrab.rab$b_mbf = (buff_count <= 127) ? buff_count : 127; + sysprv(); if (!sys_open_nofnf(&itmfab)) { #ifndef V58_UPGRADE_MANUAL ************** *** 460,465 nosysprv(); /* open the index newsgroup descriptor file for full access */ grpfab = cc$rms_fab; grpfab.fab$b_bks = 3; grpfab.fab$b_fac = FAB$M_GET | FAB$M_PUT | FAB$M_UPD | FAB$M_DEL ; --- 494,511 ----- nosysprv(); /* open the index newsgroup descriptor file for full access */ + /* ST 3/2/93 - RMS Performance Patches */ + { + int buff_count=0; + char *buff_count_ptr; + int group_performance = FALSE; + buff_count_ptr = news_getenv("ANU_NEWS_GROUPFILE_BUFFERCOUNT",0); + if (buff_count_ptr != NULL) + { + sscanf(buff_count_ptr,"%d",&buff_count); + group_performance = TRUE; + } + grpfab = cc$rms_fab; grpfab.fab$b_bks = 3; grpfab.fab$b_fac = FAB$M_GET | FAB$M_PUT | FAB$M_UPD | FAB$M_DEL ; ************** *** 466,471 grpfab.fab$l_fna = GRP_FILENAME; grpfab.fab$b_fns = strlen(grpfab.fab$l_fna); grpfab.fab$l_fop = FAB$M_CIF; grpfab.fab$w_mrs = NEWS_GRPFIL_RSZ; grpfab.fab$b_org = FAB$C_IDX; grpfab.fab$b_rat = FAB$M_CR; --- 512,525 ----- grpfab.fab$l_fna = GRP_FILENAME; grpfab.fab$b_fns = strlen(grpfab.fab$l_fna); grpfab.fab$l_fop = FAB$M_CIF; + + /* ST 3/2/93 if the buffer count logical was defined, turn on + a number of RMS performance features */ + if (group_performance) + { + grpfab.fab$l_fop |= FAB$M_DFW; /* deferred write */ + grpfab.fab$b_rtv = 255; /* map the entire file */ + } grpfab.fab$w_mrs = NEWS_GRPFIL_RSZ; grpfab.fab$b_org = FAB$C_IDX; grpfab.fab$b_rat = FAB$M_CR; ************** *** 498,503 grprab.rab$w_usz = NEWS_GRPFIL_RSZ; grprab.rab$l_rbf = (char *) &newsgrp; grprab.rab$w_rsz = NEWS_GRPFIL_RSZ; sysprv(); if (no_priv()) sys_open(&grpfab); --- 552,562 ----- grprab.rab$w_usz = NEWS_GRPFIL_RSZ; grprab.rab$l_rbf = (char *) &newsgrp; grprab.rab$w_rsz = NEWS_GRPFIL_RSZ; + + /* ST 3/2/93 RMS Performance */ + if (group_performance) + grprab.rab$b_mbf = (buff_count <= 127) ? buff_count : 127; + } sysprv(); if (no_priv()) sys_open(&grpfab); *** newsdelete_orig.c --- newsdelete.c ************** *** 152,157 open_hist_file() { FILE *fpw; if (nntp_client) return(0); if (hist_off) return(0); --- 152,161 ----- open_hist_file() { FILE *fpw; + /* ST 3/2/93 - RMS Performance Options for History file access */ + int buff_count=0; + char *buff_count_ptr; + int hist_performance = FALSE; if (nntp_client) return(0); ************** *** 153,158 { FILE *fpw; if (nntp_client) return(0); if (hist_off) return(0); sysprv(); --- 157,163 ----- char *buff_count_ptr; int hist_performance = FALSE; + if (nntp_client) return(0); if (hist_off) return(0); sysprv(); ************** *** 172,177 histfab.fab$b_fns = strlen(histfab.fab$l_fna); histfab.fab$b_shr = FAB$M_SHRDEL | FAB$M_SHRGET | FAB$M_SHRPUT | FAB$M_SHRUPD; if (!sys_open_nofnf(&histfab)) { histfab.fab$b_bks = 4; histfab.fab$l_fop = FAB$M_CIF; --- 177,192 ----- histfab.fab$b_fns = strlen(histfab.fab$l_fna); histfab.fab$b_shr = FAB$M_SHRDEL | FAB$M_SHRGET | FAB$M_SHRPUT | FAB$M_SHRUPD; + /* ST 3/2/93 RMS Performance Patches */ + buff_count_ptr = news_getenv("ANU_NEWS_HISTORYFILE_BUFFERCOUNT",0); + if (buff_count_ptr != NULL) + { + sscanf(buff_count_ptr,"%d",&buff_count); + hist_performance = TRUE; + histfab.fab$l_fop |= FAB$M_DFW; /* deferred write */ + histfab.fab$b_rtv = 255; /* map the entire file */ + } + if (!sys_open_nofnf(&histfab)) { histfab.fab$b_bks = 4; histfab.fab$l_fop = FAB$M_CIF; ************** *** 209,214 histrab.rab$w_usz = sizeof newshist; histrab.rab$l_rbf = (char *) &newshist; histrab.rab$w_rsz = sizeof newshist; if (!sys_connect(&histrab)) { sys_close(&histfab); --- 224,233 ----- histrab.rab$w_usz = sizeof newshist; histrab.rab$l_rbf = (char *) &newshist; histrab.rab$w_rsz = sizeof newshist; + + /* ST 3/2/93 RMS Performance Patches */ + if (hist_performance) + itmrab.rab$b_mbf = (buff_count <= 127) ? buff_count : 127; if (!sys_connect(&histrab)) { sys_close(&histfab); *** patchlist.h;-1 --- patchlist.h ************** *** 1,1 =+=+= End =+=+= --- 1,2 ----- + 930302_newsfiles.c!saul@hnrc.tufts.edu =+=+= End =+=+=