MVS TOOLS AND TRICKS OF THE TRADE December 1990 Sam Golob MVS Systems Programmer sbgolob@cbttape.org "THE ARCHIVER" PART I - A NEW PRODUCT AND CONCEPT. Hold onto your hats, folks. This month, we've got something new and amazing for systems programmers to talk about. It's a software package called, "The ARCHIVER". "The ARCHIVER" has a "new idea", or perhaps we can call it "an old idea done in a novel way". The new technique that "The ARCHIVER" makes possible for us, will have far-reaching consequences for the way we sysprogs do our jobs. A whole new opportunity will be opened up. People who follow this column regularly should pay close attention this month. It is probable that in the near future, the CBT Mods Tape will be converted to and distributed in "ARCHIVER" format. A current-sized CBT tape will then fit into less than half of a cartridge. "The ARCHIVER" was written, and is being constantly improved upon, by Rick Fochtman, who lives in the Chicago area. "The ARCHIVER" can dump ALL of a system programmer's software tools (source, load modules, CLISTs, etc.) INTO ONE VSAM FILE, in a very compressed form. That file can then be offloaded to a tape, VERY COMPACTLY. Individual items or groups of items can be flexibly selected from either the VSAM "ARCHIVE" cluster or the "ARCHIVE" tape and loaded directly into a standard partitioned dataset or sequential file. The level of "The ARCHIVER" about which I am writing is Version 4.0 or higher. "The ARCHIVER" was heavily reworked at the 4.0 level. Earlier versions of "The ARCHIVER" do not have the Huffman data compression feature and the current easy-to-use control card format. If you are using an earlier version of "The ARCHIVER", the old files are easily convertible to 4.0 format. "The ARCHIVER" has a "CONVERT" control statement, and is intended to remain upwardly compatible. By the time you are reading this, "The ARCHIVER" Version 4.0 or higher will be available on the current CBT tape, starting at CBT Tape version 321. Watch out for it! THE ARCHIVER'S ORIGINAL REQUIREMENTS. Rick Fochtman is a systems programmer who wanted to put ALL his software tools into ONE place. A programmer's tools can be in the form of source code, object code, load modules, or data. These different data types cannot normally be stored in one single dataset. Rick wanted to invent ONE DATASET that would contain any or all of this data, stored together. Another problem. Several versions of the same program might be kept around because of some need. You might be enhancing the program and you have "the old reliable version" plus "several new versions under test". How does one supply "version number support" for each copy? One might want to include comment data and "crib notes" in one's software collection. We use that info to jog the memory. It sounds tough to satisfy all these requirements with a single product solution. Rick has succeeded. Rick chose to use VSAM for his product's data storage vehicle, because he felt that VSAM was IBM's data management direction for the future. Rick's product can store various kinds of data IN ONE SINGLE VSAM CLUSTER. A QUICK OVERVIEW OF ARCHIVER FUNCTIONING. Rick chose a five-leveled name key that would solve several problems. First, one could distinguish a source program from a load module having the same name. Also, one could keep various modification levels of same-named source or load modules completely separate from one another. With this scheme, it is trivial to separate same-named items that come from completely different originating sources. Everything is stored together in the same "ARCHIVE" dataset. Here's a short description of how the keys work. The first four keys can each contain a 10-byte name, with the first key being special. It corresponds to the original item's "member" name. The other three keys are used to further qualify the data by type, or by telltale descriptors of the user's choosing. The second, third, and fourth keys are officially called: "group", "subgroup", and "type" respectively. The fifth key is a numeric version number that can go from 1 to a large number (16 Meg). All of these key components are handled by the service functions of the "ARCHIVER" program. Now how about the data that corresponds to each key? There are two components to the data. There is the actual data, and there are optional user comment records to describe the nature and origins of the data. How are all of these various data records stored? The data that corresponds to each key is stored across control intervals of the VSAM cluster, no matter what the data type. Directory information that would be necessary to reconstitute the item as a pds member, is gathered before the data is stored. The directory information is stored together with the data. For example, load module structure, alias information, and overlay note lists are preserved and stored before a load module is "archived". ISPF statistics and other directory user information are preserved for source-type data and stored together with that data. The "archive process" also keeps track of DCB attributes in the original dataset. All this information is later used to restore "archived" data to its original condition. The ordinary IDCAMS EXPORT function will make an "ARCHIVE" VSAM cluster transportable to another system or to another disk pack. IDCAMS EXPORT will back up THE ENTIRE ARCHIVE VSAM FILE as one entity. However, Rick also has provided his own "item-by-item EXPORT" function. Rick's "ARCHIVER EXPORT" function is part of The ARCHIVE product, and is different from IDCAMS EXPORT. A tape created by Rick's ARCHIVER EXPORT function, is also organized by key, but items are directly selectable from the tape. You can pick a part of your collection to be restored, a piece at a time. With Rick's ARCHIVER EXPORT to tape, you will not need the VSAM cluster any more. Rick added data compression algorithms that compressed the data as it was being stored. His latest version deletes repeated characters, and then uses the "Huffman Tree" algorithm, which attains a large compression ratio. Rick's "EXPORT" function, which copies an "ARCHIVE" cluster to tape, preserves the data on tape as compressed in the cluster. The storage efficiency achieved, borders on the enormous. An immediate "vast improvement" possibility is to put IBM "Optional Materials" program source on ARCHIVER tapes. Not only will individual programs be selectable by NAME and by PRODUCT LEVEL, but the whole collection will take up far less tape and will be much more compact and usable. "ARCHIVER" FUNCTIONS. How do "The ARCHIVER" service functions work? If you have software items to archive, you must allocate a suitable VSAM "archive" cluster on disk first. (See Figure One for sample IDCAMS statements to do so.) Then you "UNLOAD" items into the cluster as you want to archive them. You do not have to care about data type. The ARCHIVER takes care of almost all data types. The ARCHIVER dynamically allocates all the files, too. The only restriction (currently) is that all dynamically allocated files have to be cataloged. The "SET" control statement is used to bring about the dynamic allocation of a file. When you want to get your software or data back, you merely point to the "archive cluster" with a "SET" control statement, point to a target pds or sequential dataset with another "SET" statement, and "LOAD" selected items into the target dataset or library. "The ARCHIVER" dynamically allocates all the files. This is simple, neat, and easy. An "ARCHIVER" item-selectable tape is not just a tape backup for the VSAM cluster. The tape can be used to restore data directly to a non-VSAM dataset, without need of the original VSAM archive. You can create an item-selectable ARCHIVER tape using the "EXPORT" function of "The ARCHIVER". Merely use the proper "SET" statements, and "EXPORT" the items you want to select. The "IMPORT" control statement performs an opposite function to "EXPORT", loading items from the item-selectable ARCHIVE tape back into an ARCHIVE cluster. Figure Two will give some examples for doing these functions. To get your non-VSAM data back from an ARCHIVER item-selectable tape, just use the "LOADT" control statement with the proper "SET" statements to point to the files. Merely select the items you want to get back. See Figure Three for an example. You can find out what items are on a VSAM ARCHIVE cluster using the "LIST" control statement. Again, files are pointed to, using the proper "SET" statements, and The ARCHIVER dynamically allocates them. The "LISTT" control statement will perform the same function for an item-selectable ARCHIVE tape. An example is shown in Figure Four. Selected items can be deleted from an ARCHIVE VSAM cluster using the "DELETE" control statement, pointing to the selected items. Item key entries can be changed within an ARCHIVE VSAM cluster by means of the "ALTER" control statement. Items can not be deleted from a tape file, but they may be selected from the "archive" tape for reload to the VSAM cluster ("IMPORT" control statement) or to a dataset ("LOADT" control statement). A simplified format of ARCHIVER control statements is: statemnt ITEM=(member,group,subgroup,type,version) ,DSN=dataset.name ,DDN=ddname The version parameter is usually left out. If a same-named member already exists on the VSAM archive cluster, The ARCHIVER merely saves the new copy and increases the version number of the second copy by one automatically. The thinking is, that it's easier to delete a member later, than to try and recover a lost or overlaid member. If the version number is specifically stated in the "ITEM" keyword, then the default is to overlay an existing member on the ARCHIVE cluster that has the exact same five keys. Member, group, subgroup, and type may contain the wildcard indicator, asterisk (*) or the wildcard placeholder, percent sign (%). These have their usual meanings. The statement: ITEM=(*,*,*,*) means that all items are selected. Comment material can be associated with any item or group of items using the "ADDNOTE" control statement. The general format of all the control statements (except for "SET") follows the pattern stated above. TO BE CONTINUED. We've seen the basic externals of what "The ARCHIVER" is set up to do. We've seen how we can get it to hold our data and move it to an archive tape. Next month, I'd like to follow up with some more practical details, and I'll describe some real examples and evidence of how all of us can employ and exploit this new tool. Hope you found all of this interesting. Good luck. See you next month. * * * * * * * * * * * * * * * * * * * * * * * Figure One. //TSTBARCH JOB (TS,2322),'TECH.SUPP-SAM.GOLOB',CLASS=M,NOTIFY=TSTBSSG, // MSGLEVEL=(1,1),MSGCLASS=T TYPRUN=HOLD //******************************************************************// //* JOB TO DEFINE AN ARCHIVE VSAM CLUSTER - SAMPLE PARAMETERS *// //******************************************************************// //IDC EXEC PGM=IDCAMS,REGION=4096K //SYSPRINT DD SYSOUT=* //SYSIN DD * /*-----------------------------------------------------------*/ /* CHANGE THE VALUES FOR THE VARIOUS NAMES AND THE VOLSER */ /* TO FIT YOUR INSTALLATION BEFORE TRYING TO RUN THIS JOB */ /*-----------------------------------------------------------*/ DEFINE CLUSTER(NAME(TST.SYS22305.ARCHIVE) - RECORDSIZE(200 32000) - VOL( WORK02 ) - BUFFERSPACE(262144) - KEYS(49 0)) - DATA(NAME(TST.SYS22305.ARCHIVE.DATA) - CONTROLINTERVALSIZE(24576) - CYLINDERS( 100 100 )) - INDEX(NAME(TST.SYS22305.ARCHIVE.INDEX) - CONTROLINTERVALSIZE(8192) - CYLINDERS( 10 10 )) /* // * * * * * * * * * * * * * * * * * * * * * * * Figure Two. "ARCHIVER EXPORT" and "ARCHIVER IMPORT" Jobs to move "archived" data between the VSAM archive cluster and the item-selectable EXPORT tape. //TSTBARCV JOB (TS,2322),'TECH.SUPPORT',CLASS=M,NOTIFY=TSTBSSG, // MSGLEVEL=(1,1),MSGCLASS=T TYPRUN=HOLD //******************************************************************// //* Archiver EXPORT Job - VSAM Cluster to Tape *// //******************************************************************// //COPYSTEP EXEC PGM=ARCHIVER,REGION=5900K //STEPLIB DD DISP=SHR,DSN=TSTBSSG.ARCHV40.LOAD //SYSPRINT DD SYSOUT=* //SYSUDUMP DD SYSOUT=* //TAPE DD DISP=(,CATLG),DSN=TST.SYS22305.ARCT,UNIT=CART, // LABEL=(1,SL,EXPDT=99000) //SYSIN DD * SET VSAM1DSN=TST.SYS22305.ARCHIVE SET DDN=TAPE LIST ITEM=(*,*,*,*) EXPORT ITEM=(*,*,*,*) /* // //TST2ARCV JOB (TS,2322),'TECH.SUPPORT',CLASS=M,NOTIFY=TST2SSG, // MSGLEVEL=(1,1),MSGCLASS=T TYPRUN=HOLD //******************************************************************// //* Archiver IMPORT Job - Tape to VSAM Cluster *// //******************************************************************// //COPYSTEP EXEC PGM=ARCHIVER,REGION=5900K //STEPLIB DD DSN=TST2SSG.ARCHV40.LOAD, // DISP=SHR //SYSPRINT DD SYSOUT=* //SYSUDUMP DD SYSOUT=* //TAPE DD DSN=TST.PRIV001.ARCT, <=== Item-selectable // DISP=SHR <=== ARCHIVE EXPORT tape //SYSIN DD * * Allocate VSAM "archive" cluster (this is a comment) SET VSAM1DSN=TST.PRIVX01.ARCHIVE * Allocate EXPORT tape SET DDN=TAPE * Put all the items that were on the tape into the cluster. IMPORT ITEM=(*,*,*,*) /* // * * * * * * * * * * * * * * * * * * * * * * * Figure Three. ARCHIVER Restore of a Library from an Item-selectable "ARCHIVE EXPORT" tape. //TSTBARCV JOB (TS,2322),'TECH.SUPPORT',CLASS=M,NOTIFY=TSTBSSG, // MSGLEVEL=(1,1),MSGCLASS=T TYPRUN=HOLD //******************************************************************// //* Restore of SYS1.MACLIB from an Item-selectable Tape *// //******************************************************************// //COPYSTEP EXEC PGM=ARCHIVER,REGION=5900K //STEPLIB DD DISP=SHR,DSN=TSTBSSG.ARCHV40.LOAD //SYSPRINT DD SYSOUT=* //SYSUDUMP DD SYSOUT=* //TAPE DD DISP=SHR,DSN=TST.SYS22305.ARCT <== Item-selectable tape //SYSIN DD * * Allocate the EXPORT tape. It's non-VSAM but its allocate is * syntaxed parallel to the way LOAD would be, from a VSAM archive. SET VSAM1DD=TAPE * Allocate a PDS similar to SYS1.MACLIB and fill it from the tape. LOADT ITEM=(*,MACLIB,*,*),DSN=TSTBSSG.SYS1.MACLIB /* // * * * * * * * * * * * * * * * * * * * * * * * Figure Four. Illustration of Listing ITEMS from an Item-selectable ARCHIVE EXPORT Tape. //TSTBARCV JOB (TS,2322),'TECH.SUPPORT',CLASS=M,NOTIFY=TSTBSSG, // MSGLEVEL=(1,1),MSGCLASS=T TYPRUN=HOLD //******************************************************************// //* LISTT - Listing ITEMS from an Item-selectable Tape *// //******************************************************************// //COPYSTEP EXEC PGM=ARCHIVER,REGION=5900K //STEPLIB DD DISP=SHR,DSN=TSTBSSG.ARCHV40.LOAD //SYSPRINT DD SYSOUT=* //SYSUDUMP DD SYSOUT=* //SYSIN DD * * Dynamically allocate the "ARCHIVE EXPORT tape". Note it's Non-VSAM. SET DSN=TST.SYS22305.ARCT * List all the ITEMS on the tape. LISTT ITEM=(*,*,*,*) /*