MVS TOOLS AND TRICKS OF THE TRADE JANUARY 2004 Sam Golob MVS Systems Programmer P.O. Box 906 Tallman, New York 10982 Sam Golob is a Senior Systems Programmer. He also participates in library tours and book signings with his wife, author Courtney Taylor. Sam can be contacted at sbgolob@cbttape.org. Information about the CBT MVS Tapes can be found on the web, at http://www.cbttape.org. PLAYING I recently saw a job posting for an S/390 system administrator. The job description called for extensive experience in supervising and hands-on, covering almost all the areas of modern MVS work. I would have figured that the requirement would be for a 15 to 20-year person. Instead, at the top of the page, the posting said: "Candidates should have 3-7 years of IT experience, and should have at least 3 years of Mainframe Server Administration experience." This was an enormous shock to me--it most certainly did not sound realistic to get such a person with so few years, who would know so much. Perhaps they expected a genius--a wunderkind! I suspect that the posting was from a country that is halfway around the world from me, and I do know there are some pretty smart people there, but that requirement still is an awful lot to expect of even a smart human being. But they might be able to get away with it. Let me share my views with you. WHAT IS "EXPERIENCE"? As one of my bosses once put it, MVS "experience" is partially the following: "One needs to get a big general picture of all the components of the MVS operating system, and about how they fit together and work together." That's an "essential", but that's by no means all of it. It usually takes at least a couple of years to get this far. Of course, some coaching by a knowledgeable MVS "old hand" will help shorten this process. But there are a few more things. On my first day as an MVS systems programmer, I was told: "If there's a problem, always read the messages first, and look them up." Also, that's only part of it. Eventually, you get to hook up the message types (in your mind) with the MVS components that they belong to, and you can begin to zero in on the problems instinctively, through following that path. Next, there's detailed knowledge of the MVS components themselves, and about how each of them works. You get this knowledge by your work experience. The tasks each of us has to do, determines which areas of the MVS system will be covered more strongly. For example, if you have to code certain MVS exits, you'll acquire some knowledge about the system components that those exits add code to. In this category also, are the ISV (Independent Software Vendor) components that your system has. If you administer some of them, you'll acquire knowledge about the system areas which those products touch. Then, you always have to know about PARMLIB member setups, for both MVS itself and for the vendor products also. Besides all these things, there's SMP/E. Quite a few MVS sysprogs know the other categories pretty well. But when it comes to putting modifications by means of PTFs onto the system, they can run the jobs, but they don't really know what the jobs do. An essential piece of MVS experience consists of getting a feel for how SMP/E will insert a component into an existing system, and why SMP/E will put this piece here, that piece there, and get everything to run properly. To get SMP/E right, it either takes some good coaching from an expert, or several years of playing. I didn't get good coaching in the area of SMP/E. After four years of playing, I learned it well enough so I didn't want the next guy to have to go through the same thing. So I wrote a series of 3 articles (in 1988) about the concepts behind SMP/E, which were published in "Techinical Support" magazine. These articles can also be found on File 014 of the CBT Tape collection of MVS goodies. My articles get to the essence--they tell you what SMP/E is all about. But unfortunately, there are 100 other areas of MVS to learn about, as well. For all of those, you either need to hit the books (i.e. the IBM manuals), or you need "playing time" with each of them. WHAT IS "PLAYING"? To my mind, "playing" means trying various commands, programs, or other things on a system, to see what works, how it works, when it works, and when it doesn't work. The first thing that comes to mind about playing, is that it has to be done responsibly. Our obligation, when we work as systems programmers at an installation, is to make sure that the machine stays up and functioning properly, as long as our management and the programmers want it to. Our "playing", especially as systems programmers with "authority", must never damage the system or hinder system operation. So bearing in mind that we must never make a blunder while "playing", it is still my contention that a systems programmer must spend a good deal of time doing it. It helps to have a test MVS system, nowadays nicknamed a "sandbox system", to play on, so that the production system is not put at any excessive risk by your learning attempts. If you really really have to test something in a production environment (most of the time you never need to), I'd recommend trying to go to a disaster recovery test. You can try some stuff toward the end of a disaster recovery test, because even though that system is simulating a production environment, you also know that it will have disappeared tomorrow. I've learned a lot of good things by getting myself invited to disaster recovery tests. Just to illustrate the difference between a systems programmer and a computer operator in this regard, I remember a certain computer operator who was "too smart for the position", but the company wouldn't promote him. So he looked in the MVS System Commands book to see what he could learn. Every once in a while, we (the sysprogs) would find some setting on the system that was wrong, and when we traced it down, we realized that this operator was just trying some command he had learned in the book. I understood his dilemma and sympathized with him. Fortunately, I was the one elected to inform him of the consequences of his learning attempts. You can't do it when you're a computer operator, because you affect the entire MVS system. He later changed companies and got himself placed in a much more responsible job. TWO ADVENTURES I'll mention two of my big areas of discovery, which were largely accomplished through responsible playing. Although the second of these was the result of an initial blunder of mine, I have to say that I never once afterward, caused another disturbance of my company's production (in more than five more years there). My first adventure was about obtaining APF authorization for TSO. Nowadays, that's done globally through settings in a PARMLIB member, IKJTSOxx. But 20 or 25 years ago, it wasn't that easy. And it took some "playing" on my part, to discover a neat system programmer trick that I like to call "personal TSO APF authorization". Under TSO, there always have been several categories of program names which were slated for special treatment. One such category is: all TSO commands which are allowed to be run APF authorized under TSO. Another category is all batch programs which can run APF authorized when CALLed under TSO. A third is all TSO commands which are not to be allowed to run under TSO-in-Batch. A fourth is all programs which can run authorized when called through the TSO Service Facility. Before the PARMLIB settings were invented (I think that was some time during the ESA era), these four categories of program names were maintained in load module tables that resided in four CSECTs. APF authorized TSO commands were listed in CSECT IKJEFTE2. APF authorized programs callable under TSO were listed in CSECT IKJEFTE8. TSO commands that you can't run in batch, were listed in CSECT IKJEFTNS, and programs which can be authorized when called from the TSO Service Facility were listed in CSECT IKJEFTAP. The tricky part of all this was the question: "Where are these four CSECTs located?" In the old pre-TSO/E days, these CSECTs were part of the main TSO control load module IKJEFT02. If you wanted to zap new names into these CSECTs for your own use, you had to make another copy of the entire IKJEFT02 load module, put it into an authorized STEPLIB for your own TSO session in your private LOGON PROC, and re-linkedit into it, one or more of the four CSECTs with a copy of your own choosing. Problem was that if you did that, and later changed releases of TSO, you'd be running your TSO with a back-leveled release of the TSO control program, unless you remembered to do the whole process over for the IKJEFT02 load module in the new release. With TSO/E, IBM got a little bit smarter, and separated these four CSECTs out away from the main TSO control program. Instead, IBM created a separate load module called IKJTABLS which would contain only these four CSECTs of tables of program names. Then, since the IKJTABLS load module would not contain any executable code, it would thus be release independent, and you could carry a modified version of it over, from one TSO release to the next. After that, IBM created the global PARMLIB settings, and used its name entries to construct the four dynamic global TSO tables of all these lists of program names. That's what's always used by all the TSO users. Right? Not quite. I asked the question whether the old IKJTABLS load module is still honored in this "modern" environment. It's a legitimate question. I did some playing to find out the answer. First, I saw that there still exists a dummy copy of IKJTABLS in SYS1.LPALIB, even today. This copy is obviously not honored, because it has very few entries, way fewer than the PARMLIB member. And the PARMLIB member is what works. Then, I put a doctored-up copy of IKJTABLS into an ISPLLIB library, and I saw that it wasn't honored. After much playing, and having noted that I had to use an APF authorized STEPLIB in my TSO LOGON PROC before (during the IKJEFT02 era), I tried putting my own copy of IKJTABLS into an authorized STEPLIB in my personal TSO LOGON PROC, and I found that it worked. Not only did it add authorizations that weren't in PARMLIB, it lost authorizations that were still in PARMLIB and which I forgot to copy to my own personal IKJTABLS. I found that IKJTABLS in an authorized STEPLIB completely overrides the PARMLIB authorizations. This is useful if I myself have to run certain commands authorized, and I don't want anyone else to. I can accomplish that without disturbing the entire TSO environment. Score one discovery achieved by playing. Discovery number two centers around something that used to exist in an MVS PARMLIB, a member called the BLDL list. The BLDL list was addressed in a PARMLIB member called IEABLDxx. This member no longer exists in PARMLIB. Its functionality has been replaced, in MVS/XA systems and later, by LLA, which used to be called Link List Lookaside, and now is called Library Lookaside. The original problem that needed solving was the fact that if a system program was to be called many times over, the system would have to do a (very inefficient) search of pds directories, repeatedly, to find its actual disk location so Program Fetch would load it for execution. As a first attempt to help matters, old non-XA MVS releases had this PARMLIB member where you'd mention all the names of programs which you thought were to be called frequently. At IPL time, MVS would create a table in common storage, called the BLDL table, which would list the absolute actual disk locations of each load module, as it was at IPL time. This table could not be refreshed without another IPL. Nowadays, LLA maintains such a table in its own address space, and it can be changed dynamically with an operator command, but in the old days you were stuck. One fine day (in the old days), I accidently wiped out a library containing some BLDL-table programs, and refreshed it with a new set of programs, of course in different locations. It was painful to see the production system die a slow death, as each necessary program was being fetched from its old disk place, in vain. So afterwards I found a DYNAMIC BLDL program on the CBT tape instead, and did a lot of playing around to make sure it worked. This solved all our subsequent problems in the pre-XA days. The new program which was called DYNABLDL, interfaced with SVC 18 that does the program fetch. Every system program that was called once, would have its disk location dynamically entered into a new table, and on subsequent fetches, the new table with the exact disk location, was the one used for the fetch, and a second directory search was not necessary. Of course in order for DYNABLDL to be completely effective, you had to remove all the program entries from your IEABLDxx PARMLIB member. Somebody updated DYNABLDL for XA, which wasn't easy, because SVC 18 was moved into read-only common storage in XA, and in order to zap it dynamically, which DYNABLDL did, you had to turn off page protection, which was a trick. Once this was done, I benchmarked DYNABLDL under XA, comparing it with the new LLA. LLA was better, so DYNABLDL, despite its nice feature of allowing you to display the most-used programs, went away, unless I needed to fire it up to display the most-used programs. CONCLUSION It is my contention that an ordinary smart person can't get this kind of MVS experience in 3 to 7 years. We have to give the proper credit to the old MVS hands who have learned their trade through much trial and error, and through playing. However, I do think that an "MVS generalist" with a good head for thinking, can manage a crew of more experienced people, provided that he respects them, especially for their many years of playing. I wish all of you the best of everything, as this column goes into its sixteenth year. Please visit here again, next month.