MVS TOOLS AND TRICKS OF THE TRADE May 1996 Sam Golob MVS Systems Programmer sbgolob@cbttape.org Sam Golob is a Senior Systems Programmer working EYES - PART 1 I am of the opinion that Systems Programmers should learn about the system from the system itself. Documentation can be extremely helpful for general knowledge and diagnosis, but the bottom line is always to find out which bytes the system is "looking at". In my old days as an application programmer, I always wanted to look at the exact image of the actual data that my program was looking at. With an application program, you can't accurately diagnose a problem unless you see the data that the program was trying to read. I think the principle is the same with the system itself, even though the operating system is far more complicated. In dealing with the operating system, why don't people think this way? Because they don't have the "eyes", or tools to see what is actually going on, and therefore they have to resort to guessing and documentation. You may ask, what would most people do if they got the eyes? Probably, the smarter ones would learn to use them sooner, while others would take longer in doing so. Some might never learn. It helps if the eyes, or tools, are manufactured by IBM. Often, when an improvement is made to ISPF, many people eventually learn to use it. I've noticed this from looking over other people's shoulders while they were working. Vendor tools, which are less standard, will acquire their following. However, it might take longer for a high percentage of the programmers to start using their features heavily. If an installation acquires a diagnosis tool, often they will send its potential users to a class. Nowadays, with tight budgets, classes are getting rarer, and the exploitation of a vendor's tool might take a long time, or might never occur. What if the eyes are free? Last time, we discussed some of the management aspects of this. However, each systems programmer can acquire and learn to use such available tools. It just takes ambition. Today, we will look at examples of some free tools we can use as eyes to see the operating system. These tools can be used separately, in conjunction with each other, or together with vendor tools you have. Our aim is to encourage you in developing a growth to your "visual technique" so you can see what is going on, instead of having to guess. TYPES OF EYES. We might say generally that any tool which displays some function of the operating system to us, might properly be called an "eye" to the system. However, we need many types of "eyes". A performance monitor, which exhibits many system functions, and the current status of jobs or tasks, is one type of eye. A "core browser" or storage browser, is another type of eye. A JES2 or JES3 spool browser is a third type. A program such as "Fullscreen ZAP", which can examine the entire content of a disk pack, is a fourth type. A multi-purpose tool-box to examine files or data, is a fifth type. An EDP Auditor's tool is a sixth type. And there are other types of eyes in addition to these. The usual work we need to do, in diagnosing some problem or in evaluating performance, generally involves the skilful use of several types of eyes in succession. My friend Rick Fochtman phrased it very well. He told me: "The problem is knowing which eyes to select." We'll start addressing this issue right now. To begin, I do not wish to restrict our discussion to free tools, vendor tools, or IBM tools only. Any tool you have, if you learn how to use its features, will help you see better. I often write about free tools, because they are available to everyone who tries to get them, and one need not worry about such "trivialities" as budgets. I want everyone who reads this column to benefit from it, so it is necessary to use examples from tools which everybody can potentially obtain for themselves. However, everything we say applies equally to vendor or IBM tools of equivalent function. Most of the free tools are found on the CBT MVS Utilities Tape, which can be ordered through NaSPA, and to which I will often refer. A good pair of eyes for beginning our discussion, is the SHOWMVS TSO command from Gilbert Saint-flour, which can be found on File 183 of the CBT MVS Tape. SHOWMVS has many of the features of a performance monitor, in that it will display dozens of aspects of the current state of your system. Last IPL date and time, your SYSRES and installed software release levels, CPUs and available storage, your IODF and SMS setups, all online peripheral devices, your available and active SMF datasets, and many many other things, are quickly available to be viewed. SHOWMVS gives you a very quick and comprehensive overview of many aspects of your system. Commercial performance monitors will show you many of the same types of things that SHOWMVS will, and it is not our place here to evaluate which is better. I only want to say that if you need an overview of how your system is set up, and what it is doing, something like SHOWMVS is a good place to start. If you want to browse things to look at them, the REVIEW command from File 134 of the CBT Tape is very helpful. REVIEW uses BSAM, so it can be used to look at keyed files, such as SYS1.BRODCAST. Among many other capabilities, REVIEW can browse VSAM files, very large sequential files, and it can interpret the records in SMF and LOGREC datasets. REVIEW is currently maintained by Greg Price of Melbourne, Australia. Another good browsing tool is the BR command (also from Gilbert Saint-flour), which extends the capabilities of ISPF Browse to be able to look at VSAM files and other weirder types of datasets. I think that REVIEW can do more than BR, but in many cases, the familiarity of the ISPF BRIF (Browse) service used by the BR command, such as its vertical HEX display, will come in handy. I use both of them. Now let's put the two types of eyes, SHOWMVS and REVIEW, together to solve a relatively simple problem (that for most people is rather difficult). Somebody deleted a dataset in the last hour, and we want to find out who did it. How do we start? First, we can understand that this type of information is kept in SMF records, and SMF records are located either in the SYS1.MANx datasets, or on backup tapes. To tell if the needed SMF records are still in an active SYS1.MANx dataset, we issue TSO SHOWMVS from any ISPF command line, and we are treated to an ISPF browse of much system information. After scrolling down about 40 lines or so, there is the SMF information, and the active and not-yet backed up SYS1.MANx datasets will be very obviously displayed. The REVIEW command can look at all of this information, either on disk or on tape, but to keep things simple, lets say that SHOWMVS tells us that SYS1.MAN2 is the active SMF dataset right now, and that it contains quite a bit of data. We can now issue the command TSO REVIEW 'SYS1.MAN2', and we will be given a glance at the raw SMF data in SYS1.MAN2. To format the data, we need only enter a REVIEW subcommand, SMF, which will interpret all of the SMF records and their timestamps. Repeated invocation of the SMF subcommand toggles the interpretation of the SMF records on and off. Now, we have several FIND commands at our disposal. If we enter, FIND our.dataset.name ALL, we'll get a display of all the SMF records in the dataset that contain our dataset's name. We then have to only look for the last Type 15 record for that dataset, together with the userid and the timestamp, to figure out who deleted the dataset, and when it was done. MORE EXAMPLES. We just saw how the careful use of several types of eyes, can neatly solve a somewhat tricky problem in a routine way. If you set your system of eyes up properly, you can tackle many difficult system problems equally well, and very routinely. Now let's look at another problem which is very different. A user can't get job output back. The job has completed, and has flown through the network to "we know not where". The user's JCL has an interesting mixture of /*ROUTE PRINT and OUTPUT cards specifying destinations. The whole picture is very confusing to us, and it is our job to untangle it. We want to make sure that this user, and all the other users connected with the job, get their proper job output. Our problem is made difficult by the fact that as soon as the user's job completes, it starts flying through the network nodes, and it is gone from our spool before we can get a proper look at it. The pain-in-the-neck solution is to start looking at SYSLOGs from the various systems, and tracing the job as it has flown. This is a very difficult and time-consuming task at its best. However, there is a better way, which involves a certain kind of eye. I strongly recommend that every systems programmer in a JES2 shop install the appropriate version of the QUEUE command. QUEUE is a TSO command which displays JES2 spool data of many kinds. Even though your shop may have commercial spool browsing products such as SDSF or IOF, I think you would profit by having an available copy of QUEUE, in addition to these. Versions of QUEUE are available on the CBT Tape for JES2 versions 1.3.6 through Version 5.2. These, and even earlier versions, can also be found on the JES2 SHARE Tape obtainable from Jack Schudel at 904-392-4601. A parallel product called SDF for all recent versions of JES3, can be found on the JES3 SHARE/GUIDE Tape, obtainable from Alan Field at 612-828-4979. QUEUE can be used to solve our job-in-the-network problem. Get a copy of the user's JCL and add TYPRUN=HOLD to the JOB card. One might question the use of TYPRUN=HOLD, because it will keep the job, unexecuted, in our own spool. But this is what we want. With the user's job HELD, but ready to go, we can use QUEUE to display the JES2 JCT control block for this job, while it is being held in our system. The JCT will show all the routing destinations in various fields, and will enable us to leisurely figure out where the job's output will eventually wind up. SDSF doesn't display JES2 control blocks. IOF does. It doesn't really matter which eyes you use. I just wrote about QUEUE because it is free and available to everyone. Let's conclude this month's discussion with one more example. I recently obtained a new copy of SHOWMVS from Gilbert Saint-flour and assembled it (with old JCL). The resulting load module, before displaying anything, went down with various system abends, such as S0C1, S0C2, and S0C4. My instinct pointed to an unresolved external reference, which was confirmed by mapping the SHOWMVS load module with a mapping tool. This is another kind of eye. IBM's AMBLIST could have done the job, but I used the PDSTOOLS vendor product (from Serena in Burlingame, California) with its MAP subcommand, directed at the module. A more primitive version of this product, called PDS, which is on the CBT Tape (File 182), could also have done this particular job, and revealed the unresolved external reference to the missing sub-program. Once I saw that the sub-program was missing, I re-examined Gilbert's source code. It turned out that he had added another CSECT, and there was an END card between the first CSECT and the second. In my assembly JCL, the BATCH option had been turned off, so that the assembly ended after the first END card, not including the new CSECT. I turned the BATCH option on in the assembly, assembled both CSECTs, included the sub-program, and resolved the external reference. The resulting load module has given me no problems. Space has come to an end, and today's session must close, but I hope to continue on this subject of eyes, next time. My personal systems programming motto is to learn as much about the system, from the system itself. To do so, you need to look, and to look, you must learn to use your eyes. Good luck and good searching.