MVS TOOLS AND TRICKS OF THE TRADE JUNE 2005 Sam Golob MVS Systems Programmer P.O. Box 906 Tallman, New York 10982 Sam Golob is a Senior Systems Programmer. He also participates in library tours and book signings with his wife, author Courtney Taylor. Sam can be contacted at sbgolob@cbttape.org. Information about the CBT MVS Tapes can be found on the web, at http://www.cbttape.org. FIGURING STUFF OUT After spending many years in this field, I feel that experience is the best teacher. We accumulate experience, a bit at a time, and hopefully store it up for future use. I like to archive my past work and keep it with me. I keep big personal JCL libraries from all my previous sites, so I know how to make a job work properly and I don't have to redo all the parameters and control statements each time. Most of this experience is hard-won, so I don't want to lose it, or leave it at a previous site. Early in my career, I developed a concept, or work ethic, which has helped me throughout. I call it my 50-50 100-100 rule. It goes like this. When I do my job, I split it 50-50 with my employer. The employer gets my work, and I get the experience. That's why I call it 50-50. But it's really 100-100. The employer gets 100 percent of my work, and I get 100 percent of the experience. Knowing that I am always going to come away with valuable experience from all the work I do, I try to do my job well, even if I'm not particularly thrilled with the attitudes and approaches of my current employer, but even more so if I am happy there. In any case, using the 50-50 100-100 principle and keeping it in mind, I always tend to do a good job wherever I am. Today's topic is about applying your experience, especially when it comes to finding out information about how components of the MVS system work. I'll show you an example from my own experience, and of course, you'll be able to make connections to your own experience and hopefully, profit from the effort. Some of the best systems programmers and MVS software developers we have today, came from difficult beginnings which we can't re-create now. And that's what made them good. For example, in Australia, there used to be a lot of Fujitsu FACOM installations, which are slightly different from IBM's MVS systems. Let me explain. These sites arose because IBM, in the 1970s, used to give out the OS/360 and MVS operating systems for free, since IBM (at that time) made the bulk of their money from selling hardware. Back then, MVS was practically public domain software. Fujitsu, a Japanese company, decided to expand on IBM's last free software level, after a lawsuit from other hardware manufacturers forced IBM to charge money for the MVS software (and IBM started charging a lot). The Fujitsu FACOM system cost very much less money than MVS did, and IBM in turn sued them. As one of the results of the Fujitsu settlement, Fujitsu was not allowed to sell their "derivative" MVS-like operating system in the United States. So the Fujitsu system caught on in Australia, and it proliferated until the XA and ESA times, when the Fujitsu developers in Japan found themselves unable to keep up with IBM. Until then, Australian sysprogs at Fujitsu sites often found themselves in the position of having to adapt IBM software tools to the Fujitsu system, which had slightly, but significantly different control blocks and structures than "real MVS" did. Since the MVS tools didn't work unless you "got down to the nuts and bolts" and completely reworked them for Fujitsu's FACOM, these Australian sysprogs got very unique experience, which was impossible to create almost anywhere in the United States (except at a few special Fujitsu software development sites that were allowed to exist in the U.S.) In this atmosphere, Greg Price and a whole bunch of other skilled Australian software developers cut their teeth. The Russian experience is another, even more extreme case. During the Cold War, ending around 1990, IBM did not sell their systems to Russia. So the Russians, who needed the IBM systems to keep the Moscow subway system running, and to maintain their other institutions, had to get the IBM hardware and software clandestinely, through other countries. Russian sysprogs, who later came to the U.S. (I'm friends with a bunch of them), could not call the IBM support centers when they had a problem; they weren't even supposed to have the computers at all! So the Russian sysprogs had to debug all the systems, and make their own PTFs and fixes without outside help. Most of them wrote their own debuggers and disassemblers. And today, they are so sharp in many aspects of this work, that their experience is unequalled. Since the political climate has now changed and many of the Russian programmers are in other places as well as Russia, it has turned out that we all are receiving benefit from these Russians' wonderful work, born of their unique experience under duress. SOME OF MY OWN EXPERIENCE For illustrative purposes, I'd like to share one of my own recent experiences, which in some sense is a small window into the "process" that I want to describe. I was tempted to call this column "Reverse Engineering" but that is not truly accurate. The idea is to figure out how the IBM programs must work, without having to know their code itself. To be able to do this, is an important systems programming skill. When you see the example, you'll get what I mean. A uniqueness in my own experience, is that I was once paid to develop utilities to manage the user messages in SYS1.BRODCAST. You can see many of the results of my work by looking at File 247 of the huge free CBT Tape collection of MVS software. (Just do a google search on the words "CBT Tape".) As most of us know, and all of us can find out by doing a TSO LISTD command against the SYS1.BRODCAST dataset, this dataset has DSORG=DA or "Direct Access" dataset organization, and it is "keyed", having a one-byte key at the beginning, with 129 bytes of data following the key in each record. And it is "fixed unblocked". Such datasets, which formerly were in wider use, are composed of "slots" of records, with one record in each slot, each record having a uniquely addressable position, marked by its "relative record address". In the case of SYS1.BRODCAST, the relative record address of each record is a three-byte hexadecimal number, starting with X'000000' for the first record, or "header record". (In the particular case of SYS1.BRODCAST, the header record has information to find all the other records.) Since these datasets are very uncommon today, but the SYS1.BRODCAST dataset itself is necessary for the correct functioning of MVS, experience in reading and writing to SYS1.BRODCAST directly, is very hard to come by. If you'll look at my "Broadcast Manager" software package on File 247 of the CBT Tape collection, you'll see that I have very complete coverage of the "user messages" portion of the SYS1.BRODCAST dataset, and I have several utilities which manage and can duplicate the dataset as a whole. But recently I wanted to complete the package with programs that handled the "global notices" portion of SYS1.BRODCAST, which works completely differently than the "user messages" portion. So I came up with two programs that read and write directly to this part of SYS1.BRODCAST. They are: BCMNLIST (to list active Notice message lines, like the SEND LIST subcommand of OPER does), and BCMNOTFY which can write a new message line to a message number, delete a message number line, or write a message of (30) blanks to a message number line. I obviously thought that by doing this, it would be enough. In my mind, I had it covered: BCMNLIST to list the "Notify" messages by number, and BCMNOTFY to write them, change them, delete them, and even to write blank lines. I can also write my Notify messages to a COPY of SYS1.BRODCAST as well as to the "real" one. All I have to do is to allocate the ddname of BRODCAST to either SYS1.BRODCAST or to the copy. But when I got finished writing BCMNOTFY, I was shocked to discover that LISTBC did not show my changes, even though I had indelibly written them to SYS1.BRODCAST. It was obvious to me that LISTBC was looking at an incore copy of the Notices, not at SYS1.BRODCAST itself. It should have occurred to me that this made sense. Suppose at a large site, that 150 people were trying to LOGON to TSO at one time. Since LISTBC is automatically invoked by every TSO LOGON, unless the site turns off the invocation of LISTBC (which is very rare), then if LISTBC had to read SYS1.BRODCAST directly for each LOGON, there would be a big performance bottleneck in these LOGONs which are occurring together. So to avoid unnecessary waits at LOGON time, IBM had created an incore copy of the "Notices" part of SYS1.BRODCAST, which LISTBC would read. Since (officially) the Notices part of SYS1.BRODCAST could only (using IBM's tools) be changed by the SEND subcommand of the OPER TSO command, it made sense to change the incore copy only when the OPER SEND command combination was invoked. So what was I to do? I had to make it possible for LISTBC to detect my changes, and force a re-read of the SYS1.BRODCAST dataset to create a new incore copy of the Notices. I had figured all of this out without asking one question of IBM, nor of any of my friends. But then I needed to ask something: "Where was the incore copy of the Notices kept?" So I put out the question on the IBM-Main newsgroup, and Jim Mulder, one of the IBM developers who monitors IBM-Main (voluntarily, I might add), told me to look at the IKJTSVT macro, which maps the "TSO Vector Table", and to look at anything with the string TSVTNCT in it. There were two occurrences of TSVTNCT in the assembler part of IKJTSVT. One was TSVTNCT itself, that is an address pointer to the actual incore Notices table, and the other is TSVTNCTU, a flag byte which indicates that the incore Notices table needs to be updated. Now I'd gotten somewhere. The TSVT is pointed to from the CVT (Communication Vector Table) at X'9C' off the beginning of the CVT, and the data is pointed to at +8 off the TSVT. The "need to update" flag is X'80' at +5 off the TSVT. I used the free LOOK TSO command from File 264 of the CBT Tape collection to browse the core. The J10 subcommand of LOOK (look at storage pointed to by virtual location X'10') got me to the CVT, which LOOK formats. Then the LTVT command (link to the TVT field of the CVT formatted control block) got me to the TSO Vector Table, which LOOK also formats. Then LNCT got me to browse the incore Notices table itself. Pressing PF9 to back up one screen and look at the formatted TSVT again, I entered the ONULL LOOK subcommand, to unformat the storage so I could browse it directly. I found out everything I wanted. Now I just had to write a program to throw the "table needs to be updated bit", and see which IBM program would read that bit and update the table. The program to throw the bit is now called BCMNUPD, and it is a TSO command which has to be run APF authorized. But the other programs, BCMNLIST and BCMNOTFY, don't have to run authorized. When I flipped the TSVTNCTU bit on, I found, on recent systems, that LISTBC itself will inquire about that bit, and it itself will update the incore Notices table. But on older TSO systems (it turned out on z/OS 1.2 and older), you needed to run an OPER SEND to force an inquiry of this bit and update the incore Notices table. So I solved my problem. Please notice that this entire process did not require any real "reverse engineering" or disassembly of IBM code. It only required some knowledge of IBM control blocks, which is public knowledge on SYS1.MACLIB. I didn't need to do any reverse engineering in my actual coding efforts, at all. IBM gives you the information publicly, if you have the experience to know how to look for it. USING OTHER PEOPLE'S EXPERIENCE It is obvious that very very few people have done what I have, in the previous example. But MVS is a very huge system, and there's lots to learn about, and lots of other stuff to be done. Each person does the job in those areas which he or she has to support. And for the other stuff that needs doing, we all have to rely on tapping into other people's experiences. We all use the RESULTS of my experiences. But almost nobody else has to duplicate my work. That's the beauty of it. MVS is a large system, but a lot of coverage is obtained, when each different person takes his own piece of it to study. With a lot of people working, there is then a lot of coverage. And you can see the result of the other people's work by looking at the CBT Tape collection (most of which comes with source code), and at www.xephon.com's older stuff, which is free, as well as in other scattered places on the net. Another good place for finding facts about MVS internals is the source code for the SHOWMVS program, on CBT Tape File 492, where a lot of this control block information has already been figured out. Some of the tools that I used in my particular endeavors are very interesting and useful, such as the LOOK program from File 264 of the CBT Tape, which I used for browsing virtual storage. LOOK can potentially format any storage which has a macro definition to describe it. The version of LOOK which I used, has a nice selection of macros for formatting, and you can add many more, if you want to. If a control block is formatted by field names, you can use an L command (or Link) to follow the control block chains by field name, as long as all the control blocks involved are formatted. If not, you just point to the next field by displacement, such as J+8, which would mean to show the storage that is pointed to by the current location +8. Other programs on the CBT Tape can show storage, such as MXI (Files 409 and 410) from Rob Scott and XMDSMAIN on File 690 from Martin Kline. The TASID program from Doug Nadel at www.sillysot.com/mvs is another free program that can be used to browse storage too, among its "many other talents". At this point (right before I have to close) I want to say that many facts about MVS internals can be learned by "figuring them out" and by experimenting. Each person can't do all of it alone. And each separate investigation takes time. But each person can profit by looking at the other people's work as it relates to theirs, learning piece by piece, one fact at a time. That's how you accumulate knowledge and eventually you can do more and more, to everyone's gain. I wish you all the best of everything, and I'm hoping to see you here again next month.