MVS TOOLS AND TRICKS OF THE TRADE February 1998 Sam Golob MVS Systems Programmer P.O. Box 906 Tallman, New York 10982 Sam Golob is a Senior Systems Programmer. Sam can be reached at sbgolob@cbttape.org. ASSORTED UTILITIES - PART 1 In our field, there are several marks of a veteran. The first is a general knowledge and conceptual picture of the different parts of the MVS (or OS/390) operating system. Such knowledge usually takes years to accumulate. Second is an ability (acquired by experience) to quickly isolate the component that is causing a problem, and to know where to get help, or how to make an adjustment to fix the problem. Third, a veteran should know how to set up the MVS operating system from scratch, starting from the way IBM ships it. Finally, the veteran is constantly gathering an ever-increasing arsenal of tools, always learning how to use them better. You don't measure a veteran's experience in years, but rather in technical savvy. Every MVS (or OS/390) installation is different, even though they all have the operating system in common. MVS shops are of different sizes, and especially, different budgets. It all depends on what they are trying to accomplish from a business perspective. Some shops are what we'd call "large", which support huge businesses, and they buy a lot of vendor software and hardware products to satisfy their large needs. Other shops can be very small--either medium-sized businesses or software development shops, with relatively low budgets. And there are the "medium-sized" shops in-between, supporting large businesses, who buy some vendor software to fit their needs, but they aren't replete with a lot of "system programmer fancy tools", as the large shops may be. This makes for differences in how the systems programmer, who takes care of that installation's system, approaches the job. In a large shop, the systems programming tasks are segmented, and divided among a considerable number of people. It helps to have a few veterans there, who can cut through the occasional tough problems quickly, and generally guide the installation's progress in hardware and software. The veterans can also bring the more junior personnel up to snuff, eventually making them veterans also. A small shop is run by one or two people possibly, and they have to be generalists. The medium-sized shop, which is in-between, should always have at least one veteran there, and if not, at least one systems programmer who knows how to get access to lots of help. Tool-wise, there are many differences between small and large shops. Even with tiered pricing, the smaller and medium-sized shops can only afford a limited number of vendor-supplied tools. Thus, the systems programmers there, have to "make do with less". My recommendation for this, which is being followed by more and more people nowadays, is to learn how to use free software tools. MVS systems programmers are fortunate to have access to many free tools and utilities, which are parts of huge software collections such as the CBT MVS Utilities Tape. The CBT Tape, which is independently produced, is distributed by several sources, including NaSPA. Contents of the current version of the CBT Tape are now posted on the Internet, at location: www.cbttape.org A version of the CBT Tape is also part of the NaSPA cd-rom, which has been updated approximately once a year. The idea of the CBT Tape is that if one systems programmer solves a problem, why should all the others have to re-invent the wheel? Many interesting software solutions to various problems can be found there. The CBT Tape is literally an MVS systems programmer's encyclopedia. Today, we shall discuss one particular collection of assorted tools, which was put together by a clever man, John V. Hooper. John's tools can be found on File 019 of the CBT Tape, and I'm sure that at least one or more of John's dozen-or-so programs, can benefit any MVS shop. This collection was updated on Version 415 of the CBT Tape, and is running on an OS/390 Version 1.2 system with JES2 at the appropriate level. WHAT'S IN THE COLLECTION John Hooper's collection of programs covers various needs of his medium-size shop. From a glance at our descriptions, you'll quickly see that a lot of key areas in the maintenance of an MVS environment are covered. John has several programs which monitor the blocksize of datasets globally, throughout the entire shop. This has been done to improve I/O efficiency, and has helped to eliminate bottlenecks in the company's batch window. A second group of programs constitutes a general job performance and throughput analysis package. Other programs do dataset analysis and monitor DASD volume contents, using the appropriate SMF records. Of course, larger shops may accomplish this by buying SAS and Barry Merrill's SMF record analysis package, but John Hooper's place doesn't have SAS. Therefore, John wrote some assembler programs to cover this ground. John has a console automation tool, which helps his shop to simplify system startup and shutdown, without using a formal automations package. These programs are all free, are publicly available, and have been running on an OS/390 system. Do they whet your appetite? You can try them too. There's more. There's an operator command to simplify the control of the 3990-3 Cache Subsystem, which normally requires IDCAMS subcommands that are quite complicated. There's a program to report Private Region size below the 16 meg line, right after IPL time, and to make sure it isn't below a critical threshold. If the Private Region is too small, an alarm is sounded, in the form of a non-rollable message to the operator console, and you're not stuck in the middle of a production day, with a job or CICS region that won't run. John has a general map of the main storage areas in an MVS system, and even though the large performance monitors also have such a map, this one can also be run as a batch program to produce a hardcopy report. Another of John's programs will track a load module in the link list or LPA libraries, displaying which library the program is found in, or where in storage (if LPA) the program is loaded. This program also displays the beginning few bytes of each module it finds, so you can get an idea of which version it is. Finally, John can trace, through analyzing LISTCAT ALL outputs for all catalogs, which DASD volume must contain which datasets. This is useful in an SMS environment where it's not clear on which exact volume a dataset has been placed. One can begin to recover, when a particular DASD volume goes bad. At least you know what datasets should have been on it. Some, or all of these tools, may help you. Now I'll go into a bit more detail, so you can better judge for yourselves. Please bear in mind that these descriptions are only a beginning. John Hooper's utilities have much more depth than I can indicate here. HERE ARE THE UTILITIES FLSMFDSN and BLKSCAN: This combination of programs was designed to eliminate bad block sizes of datasets in the entire shop. The FLSMFDSN program analyzes SMF type 14 and 15 records to cover non-VSAM datasets, and type 64 records to cover VSAM datasets. FLSMFDSN has a large collection of sort and selection parameters, making it extremely flexible. FLSMFDSN can be used to flag all the datasets, in the entire shop, which have inefficient block sizes. After these have been pinpointed, the BLKSCAN program is used to scan all the production and test JCL, showing the origin of the inefficient block sizes. All bad JCL can be corrected, and those jobs will run more efficiently. FLSMFJOB: The FLSMFJOB program gets its information from SMF type 30 job and step records, reporting on the resources used by a job step or an entire job. A tremendous variety of execution and sort parameters makes this program extremely useful in pinpointing jobs which are hogging CPU or I/O. Once these jobs, or job steps, have been flagged, something can be done about the programming, to make them more efficient. This program has made it possible to delay a processor upgrade in John's shop, saving a considerable amount of money. COMMAND: At the time this program was written, John's installation didn't have a formal automated operations package, but its operation is large enough (it's a big chain of supermarkets) so that system IPL and system shutdown are nevertheless quite complicated. Without the COMMAND program, complete IPL and system shutdown procedures might take close to half an hour apiece. With this program, each takes only a few minutes. COMMAND accomplishes this by not only allowing for automatic issuing of MVS and JES2 commands, but by including DELAY and message REPLY subcommands also, so that the proper components can come up (or down) in the proper order, at the proper time, and with console WTOR messages properly replied to. This capability allows any shop to recover from an emergency outage, with a very minimum of down time. Business is saved, and money is saved. John told me that even with a console automation package, the COMMAND program is useful for backups of an individual application. Using the COMMAND program, a single procedure can be written, which can shut down the application, take the backup, and automatically restart the application. Auto ops packages aren't frequently used for that. CHECKPVT: The CHECKPVT program is designed to run immediately after IPL, and to report the maximum private area available for started tasks, jobs, and TSO users. After slight system maintenance, it is possible for CSA to be allocated on a different boundary, an entire meg lower, without you're even knowing it. The private area can be cut by an entire meg of virtual storage, making some large jobs or CICS regions abend, when they are started later in the day. This is not particularly pleasant, and can necessitate an IPL for the backout of maintenance in the middle of the day. CHECKPVT will check the private region size at IPL time, and will immediately write a non-rollable message to the operator's console if it is less than a pre-determined threshold value. The production disturbance will be nipped in the bud. FLSMFCAT: The FLSMFCAT program gets information from SMF records to show ICF catalog activity at the dataset level. Sometimes, when a dataset is created or deleted, and the application fails, it is not easy to tell who created or deleted the dataset. The SMF ICF catalog information is the best way to tell what happened. FLSMFSRT: The FLSMFSRT program flexibly produces reports from SYNCSORT SMF records. This program allows John's installation to monitor the larger sorts in production and test jobs. Of course, the program is only useful if your shop uses SYNCSORT as its sorting utility product. FLVOLLST: The FLVOLLST program is designed to print a report listing all of the datasets on a volume based upon information from the system catalogs. This list could be critical in case of a DASD failure which destroys the VTOC on the volume. With volume pooling now available through the use of DFSMS and other program products, it is not always easy to determine the datasets which are on a specific volume. The input to this program must be the output from an IDCAMS LISTCAT command. It is expected that an IDCAMS 'LISTCAT VOL CAT(user.catalog.name)' command will be executed for each catalog in the system. The contents of these reports can then be passed to this utility program to produce the report by volume and dataset name. MODLOOK: The MODLOOK program is designed to run as a TSO command, started task or a batch job to look up the selected module(s) in the system link list or link pack area. If the module is in the link list, the link list library name will be displayed. If the module is in the link pack area, its address will be displayed along with the name of the resident area in which it is located such as PLPA, FLPA, ECSA, etc. The first part of each module is displayed also, since it can contain date, time, or copyright information which may be of interest. With many libraries now in the system link list, it may not always be apparent which dataset contains which program or even more importantly, it may be difficult to determine which library contains a module if duplicate module names exist. SMAP: The SMAP program is designed to print the starting address, ending address, and size of each of the main storage areas in the MVS system. This information can be displayed using most of the popular monitors currently available, but not everyone has one. Also, this program can run as a batch job producing a hardcopy report. JES$LF: The JES$LF program is a JES2 Exit 5 module, designed to process the $LF command when entered. It will give detailed information at the output group level for jobs awaiting print. This is, in effect, a detailed version of the $DF command. In summary, you see that one person can come up with very inventive tools to help the shop, and if he is gracious enough to make the tools generally available, we all can benefit. The CBT Tape is an enormous collection of many such contributions. I hope that the mention of these utilities will encourage you to look at them. Good luck. We'll have more of these tools to show you, next month.