MVS TOOLS AND TRICKS OF THE TRADE January 1991 Sam Golob MVS Systems Programmer sbgolob@cbttape.org "THE ARCHIVER" PART II - MORE ABOUT AN INTRIGUING NEW PRODUCT. Last month, we introduced the new free product called "The ARCHIVER". The ARCHIVER stores and compresses non-VSAM data of almost any type, into one single VSAM cluster. This compressed data is item-by-item exportable, still compressed, onto a tape file. All the data can be faithfully reconstituted to its original form from either the "archive" VSAM cluster or from the "archive export" tape. "The ARCHIVER", originally designed as a vehicle for sysprogs to store "all their stuff in one place", has the potential to eventually revolutionize small-scale in-house backups, and to allow far more efficient packaging of software collections and products. As a personal example, I have stored the contents of 274 SMP-related pds libraries, and 15 large sequential datasets, in one VSAM "archive" cluster. After reorganization, the VSAM cluster occupied around 10000 tracks of 3380 disk space. The "archive export" tape, which contained all this data, occupied barely more than one and a half 3480 cartridges. My previous, less efficient IEBCOPY backups took about eight cartridges on average. "The ARCHIVER" Version 4.0 or above, can be obtained on a NEW CBT MVS Tape at version level 321 or higher. The ARCHIVER is on File 147. Please use Version 4.0 or higher of The ARCHIVER, because The ARCHIVER was extensively reworked at that level. Previous versions don't have the great data compression. VSAM files created with previous "ARCHIVER" versions are upwardly convertible using the CONVERT command of the current "ARCHIVER" version. This month, I'll explain some further details of ARCHIVER workings to add to what we said last month, and I'll show some more examples. I hope that you can exploit The ARCHIVER creatively for your advantage, and to the advantage of your shop. A QUICK REVIEW. The ARCHIVER is written by Rick Fochtman from Chicago. It stores almost any kind of data in a VSAM cluster. A stored data item has 5 keys--the first four, member name, group, subgroup, and type--have 10 characters apiece. The last key, the version number, is a binary integer that ranges from 1 to a large value (16 Meg). Keyword parameters are used in almost all the control statements. Individual entries are referred to as "items". Storing an item in a VSAM Archive cluster is called UNLOADing. Comments in a control statement start with an asterisk in column 1. Dynamic allocation of required files is done either by a SET control statement or a DSN or DDN keyword parameter. We'll give an example. To UNLOAD all members of a PDS into an archive, we may use control statements such as: SET VSAM1DSN=TST.SAMPLE.ARCHIVE SET DSN=TST.OUR.SAMPLE.PDS UNLOAD ITEM=(*,SAMPLEPDS,11/09/90,PRODUCTION) or more easily: SET VSAM1DSN=TST.SAMPLE.ARCHIVE UNLOAD ITEM=(*,SAMPLEPDS,11/09/90,PRODUCTION),DSN=TST.OUR.SAMPLE.PDS These statements will cause all members of the dataset: TST.OUR.SAMPLE.PDS to be unloaded (and compressed) into the previously defined VSAM cluster: TST.SAMPLE.ARCHIVE . See Figure One for a complete example. See Figure Two for a list of ARCHIVER control statements and what they do. This material was discussed in detail last month. Now, we'll pick up where we left off, and we'll talk about how The ARCHIVER dynamically allocates most of the files that it needs. THE ARCHIVER'S DYNAMIC FILE ALLOCATION. File allocation when using The ARCHIVER is regulated by the "SET" control statement. There are three forms of SET. These are: SET VSAM2DDN=ddname or SET VSAM2DSN=cluster name SET VSAM1DDN=ddname or SET VSAM1DSN=cluster name SET DDN=ddname or SET DSN=non-VSAM dsname VSAM2DDN or VSAM2DSN is used only in a situation when there are two clusters involved in the operation. In that case, VSAM2D-N would refer to the "from" cluster and VSAM1D-N would refer to the "to" cluster. Examples are the "COPY" or "CONVERT" control statements. ("CONVERT" moves items from an old-format ARCHIVE cluster to a current-format cluster.) These operations copy or convert items from one ARCHIVE cluster to a second. During most other ARCHIVER operations, when only one VSAM cluster is involved, VSAM1DDN or VSAM1DSN is used to point to the cluster. Non-VSAM datasets are always pointed to by SET DDN or SET DSN statements. Of course, a SET DDN statement must refer to a ddname specified in your JCL. A SET DSN statement will automatically invoke dynamic allocation to get to a cataloged non-VSAM dataset. Currently, SET DDN statements pointing to ddnames, are the only way for The ARCHIVER to get to non-cataloged non-VSAM datasets. VSAM cluster values or non-VSAM dataset names that are specified by SET operands will remain in effect until overridden by another SET operand. In addition, the VSAM2DDN, VSAM1DDN, DSN etc. statements can be appended to the operands of any other ARCHIVER command. For example: UNLOAD ITEM=(*,*,LINKLIB,*),DSN=TSTBSSG.TEST.LINKLIB would be the equivalent of: SET DSN=TSTBSSG.TEST.LINKLIB UNLOAD ITEM=(*,*,LINKLIB,*) . That's pretty much the story when it comes to The ARCHIVER's dynamic allocation. It's quite straightforward and simple. Currently, there's one "gotcha" in the matter. "LOADT", which copies items from an item-selectable ARCHIVE tape dataset to a pds or sequential dataset, performs a similar function to "LOAD", which goes from a VSAM archive dataset to the non-VSAM dataset. In other words, "LOAD" and "LOADT" are parallel functions. Currently, the input to "LOADT" is the same as the input to "LOAD", namely VSAM1DDN or VSAM1DSN, even though the item-selectable tape dataset is not a VSAM dataset. Normally, a VSAM1DSN statement would be inappropriate to point to a non-VSAM dataset, and would cause an error. If you would run a "LISTT" against the item-selectable tape dataset, VSAM1DSN or VSAM1DDN wouldn't work. Since the dataset is really non-VSAM, you need to use DSN or DDN to do the dynamic allocation for "LISTT". This matter is somewhat confusing, but I think it is the only significant "SET" control statement problem. Rick may correct this situation in the future. If he does, he will inform you. This slight problem in use, is greatly outweighed by The ARCHIVER's enormous data storage advantages. CHANGING EXISTING ENTRIES IN THE VSAM CLUSTER. The ARCHIVER service functions can control entries in a VSAM archive dataset very closely. The "ALTER" control statement, also termed "RESPECIFY" in older ARCHIVER versions, accomplishes a renaming and re-keying service for archived items. Syntax for ALTER is: ALTER ITEM=(.......),NEW=(.......) where the NEW operand has the same syntax as the ITEM operand with an important exception. In the NEW operand, generic characters such as "*" and "%" are treated as ordinary characters, and can be a part of the key names. Only the "*" character alone will be treated in the NEW operand as a generic, to signify that the corresponding old value will remain unchanged. "RESPECIFY" still remains a valid alias for the ALTER control statement. Let's give an example of how ALTER might be used. Suppose two pds'es have been unloaded into an archive. One of them is SYS1.LINKLIB and the other is PPS.LINKLIB, a program product library. The UNLOAD statements might read: UNLOAD ITEM=(*,SYS1LINKLB,11/15/90,MVSRES),DSN=SYS1.LINKLIB UNLOAD ITEM=(*,PPSLINKLIB,11/15/90,VEND01),DSN=PPS.LINKLIB You might want to change the some of the modules in PPS.LINKLIB to have a special designation. Suppose all modules beginning with the designation "ISR" are to be singled out into a category called ISPF/PDF. Our ALTER statement might then read: ALTER ITEM=(ISR*,PPSLINKLIB,*,*),NEW=(*,ISPF/PDF,*,*) It is clear that a lot of item rearranging may be accomplished from within the VSAM archive, after many items have already been unloaded into it. Because of key arrangement, however, CA and CI splits and space expansion may occur through extensive use of ALTER. If you are constantly manipulating a VSAM archive file, it is wise to reorganize the file frequently. Normal IDCAMS EXPORT-IMPORT or REPRO functions may be used to reorganize and compactify an actively used VSAM archive cluster. Reorganization may also be necessary several times during many large UNLOADs into a VSAM archive. You'll usually find that reorgs will save you a lot of disk space, and will be helpful, especially if you want a VSAM archive to stay around on disk for some time. ADDING NOTES. SPECIAL DOCUMENTATION THAT IS VERY CONVENIENT. Any archived item is capable of having NOTEDATA attached to it. Notes can be added to individual items separately, or they can be added generically. Notes may be used to further qualify or clarify the nature of the items being archived, and they can be of great convenience to their creator, or to any user of the archive. Notes in an archive can be shown with a LIST command, if the keyword NOTE=YES is used. NOTE=YES is not the default. There is a small amount of learning necessary to use the NOTE facility, and this concerns the input dataset, where the notes are fed into The ARCHIVER. The note input dataset is an 80-byte card-image dataset, in which columns 73-80 contain key data for the notes. Text data is in columns 1-72. The key data is used to determine which note images are appended to which items. The data for many notes may be mixed up on one input file, but all is dependent on key values in columns 73-80. Blanks in columns 73-80 constitute comments in the note input data file. Let's consider an example. Suppose the ARCHIVER source pds has been unloaded on the archive cluster TST.TEST001.ARCHIVE with the statement: UNLOAD ITEM=(*,ARCHIVE,UTILITY,SOURCE),DSN=TSTBSSG.ARCHV402.SOURCE We want to mark all the resultant items in the archive with copyright information from the author. This might be done with the statement: ADDNOTE ITEM=(*,ARCHIVE,UTILITY,SOURCE),KEY=ARCHIVER At this point, we don't know how KEY=ARCHIVER is fed in. This comes from the note input file, which is allocated by a ddname, and a control statement like: SET DDN=NOTEFILE The note file will have the word "ARCHIVER" in columns 73-80, opposite the statements: ---------------------------------------------------------------------- ARCHIVER THIS IS A PART OF THE VSAM ARCHIVER PROGRAM, WRITTEN BY RICHARD ARCHIVER A. FOCHTMAN, JR. AND COPYRIGHTED 1990. EXCEPT WHERE NOTED, ALL ARCHIVER PARTS OF THE ARCHIVER ARE REENTRANT, REUSABLE, AND REFRESHABLE. ARCHIVER ---------------------------------------------------------------------- ARCHIVER All items in the archive which satisfy the addnote item qualifications, will get the keyed remarks appended to them as notes. These notes can be listed by a statement: LIST ITEM=(*,ARCHIVE,UTILITY,SOURCE),NOTE=YES Now the DYNAM program in the ARCHIVER source pds is public domain, and is therefore not copyrightable. We can add an additional note to the DYNAM item, to say that DYNAM is public domain. Please notice that adding the additional note does not erase any old notes. Old notes must be completely deleted and then replaced. Let's add the special note for the public-domain module with the keyword of "DYNAM" in columns 73-80 of the note input file, so that: ****************************************************************** DYNAM * THIS PARTICULAR SUBROUTINE WAS OBTAINED FROM THE CBT MODS TAPE * DYNAM * AND IS PUBLIC DOMAIN SOFTWARE, NOT SUBJECT TO COPYRIGHT. * DYNAM ****************************************************************** DYNAM This note might be added with the control statement: ADDNOTE ITEM=(DYNAM,ARCHIVE,UTILITY,SOURCE),KEY=DYNAM so that the DYNAM item will have a note that reads: ---------------------------------------------------------------------- THIS IS A PART OF THE VSAM ARCHIVER PROGRAM, WRITTEN BY RICHARD A. FOCHTMAN, JR. AND COPYRIGHTED 1990. EXCEPT WHERE NOTED, ALL PARTS OF THE ARCHIVER ARE REENTRANT, REUSABLE, AND REFRESHABLE. ---------------------------------------------------------------------- ****************************************************************** * THIS PARTICULAR SUBROUTINE WAS OBTAINED FROM THE CBT MODS TAPE * * AND IS PUBLIC DOMAIN SOFTWARE, NOT SUBJECT TO COPYRIGHT. * ****************************************************************** You get the picture. A FEW "UNOFFICIAL BENCHMARKS". I personally spent some hours testing "The ARCHIVER" before the time of this writing. I did an "UNLOAD" of 63 system libraries into a VSAM archive on a work pack. These included SYS1.LINKLIB on an XA 2.2.3 system, SYS1.LPALIB, SYS1.NUCLEUS, our company's program product library, SYS1.MACLIB, AMODGEN, panel libraries, PSF libraries, and so forth. There were a total of over 15,000 non-alias items loaded. Wall clock time for the whole thing on a heavily loaded 3090 model 180 was about 2 hours and 44 minutes. The VSAM ARCHIVE occupied 400 cylinders on a 3380 disk before it was reorganized, and 200 cylinders after it was reorganized. The "LIST" function for all the items in this ARCHIVE was near-instantaneous, taking 14 wall-clock seconds to list 15,938 items. "LIST" only has to read the VSAM header records for each item. There are not too many EXCP's. Creating an item-selectable "ARCHIVE EXPORT" tape from this ARCHIVE took 47 minutes of wall clock time, to 3480 cartridge. A TAPEMAP run (File 299 of CBT tape version 321) showed that the cartridge was almost exactly half full--253 feet was used for the backup of all these libraries! A full cartridge has about 500 feet, as measured by the TAPEMAP program. A "LISTT" run to list the contents of the item-selectable ARCHIVE tape, took about two minutes of wall clock time after the tape was mounted. This was a cartridge. A "LOADT" was done to restore all LINKLIB load modules from the item-selectable ARCHIVE tape to a junk load library on a work pack. This load library restore was followed by a csect compare to the original SYS1.LINKLIB. It showed exact comparison and no loss of members or aliases except for two things which were normal: different note-list ttrs in the overlay modules, and the restore of orphaned aliases to non-alias status. To me, that restore sounds like a success. I restored the archived members of SYS1.FONTLIB (PSF fonts) from the archive tape to a junk disk copy of the library. I got an exact match, and no loss of members. Restore of SYS1.MACLIB produced an exact compare. This isn't bad. FUTURES. Rick Fochtman is planning a whole series of enhancements for The ARCHIVER product. To be fully informed, order new CBT tapes, look at File 147, and keep in touch with Rick or with me. We hope that soon, the CBT Tape itself will be converted to ARCHIVER item-by-item EXPORT format on tape, because of the great compression factor, and the uniformity in storing items from diverse sources. I hope you found this interesting and useful. If you use the CBT tapes, it will definitely be useful. Good luck. See you next month. * * * * * * * * * * * * * * * * * * * * * * * FIGURE ONE. Sample ARCHIVER Job to UNLOAD all members of partitioned dataset TSTBSSG.ARCHV402.SOURCE into VSAM archive dataset TST.TEST001.ARCHIVE . //TST2ARCV JOB (TS,2322),'TECH.SUPPORT',CLASS=M,NOTIFY=TSTBSSG, // MSGLEVEL=(1,1),MSGCLASS=T TYPRUN=HOLD //* //COPYSTEP EXEC PGM=ARCHIVER,REGION=5900K //STEPLIB DD DISP=SHR,DSN=TSTBSSG.ARCHV40.LOAD //SYSPRINT DD SYSOUT=* //SYSUDUMP DD SYSOUT=* //DISKA DD DISP=SHR,DSN=TST.TEST001.ARCHIVE //SYSIN DD * SET VSAM1DD=DISKA UNLOAD ITEM=(*,ARCHIVER,UTILITY,SOURCE),DSN=TSTBSSG.ARCHV402.SOURCE /* // * * * * * * * * * * * * * * * * * * * * * * * FIGURE TWO. ARCHIVER Control Statements and their Function. ADDNOTE - Append comment notes to an item from a note file. ALTER - Change item key entries for data in a VSAM archive. CONVERT - Convert entries from an old-format archive to current. COPY - Copy items from the VSAM2 Cluster to the VSAM1 Cluster. DELETE - Delete items from a VSAM archive cluster. EXPORT - Item-by-item export from VSAM cluster to sequential ds. IMPORT - Reverse of EXPORT. Sequential export ds to cluster. LIST - Show items on a VSAM archive cluster. LISTT - Show items on an ARCHIVER item-by-item EXPORT seq. ds. LOAD - Reconstitute a pds or sequential dataset from archive. LOADT - Reconstitute same, from the EXPORT tape or seq. ds. SET - Point to ddname or dsname for dynamic allocation. UNLOAD - Archive an item from a non-VSAM dataset to VSAM archive.