MVS TOOLS AND TRICKS OF THE TRADE February 1999 Sam Golob MVS Systems Programmer P.O. Box 906 Tallman, New York 10982 Sam Golob is a Senior Systems Programmer. He also participates in library tours and book signings with his wife, author Courtney Taylor. Sam can be contacted at sbgolob@cbttape.org. Information about the CBT MVS Tapes can be found on the web, at http://www.cbttape.org. Resource Affinity Implementation Efforts Nowadays, as before, MVS shops come in all sizes. But it's even more common nowadays than before, with parallel processing and sysplexes, to have multiple MVS images working together at one installation. Where it would formerly suffice to have one big MVS machine, nowadays it's more likely that there are two or more MVS images running as a sysplex. As we all know, there are advantages in reliability and scalability when we have more than one machine running. But there could also be management and licensing headaches, and those have to be handled too. Under these conditons, it sometimes happens that a certain resource, for example a database, is attached to one MVS image in a sysplex, but not to another. Any job that uses this resource must run on the MVS image that the resource is attached to. If it runs on another MVS by mistake, it will have to fail. Since the application programmers aren't normally responsible for the hardware and software setup at an installation, the principal task falls on systems programming and operations to make sure that each job will run on the correct machine. The traditional and ancient method of controlling the affinity of a batch job to a given machine, relies on the Job Entry Subsystem (JES2 or JES3). In the JCL on the JOB card, the user can specify initiator types, say CLASS=I initiators, which have been started on System A, but which haven't been started on System B. As long as there are no CLASS I initiators started on System B, CLASS I work will not run there. The method is simple and it usually works, but it's static. In a more complicated resource scenario, we'd want to have a better way. There are other ways of controlling which machine a job will execute on, such as a JES "/*ROUTE XEQ" card in the JCL, that goes below the JOB card. The "native JES2" ROUTE XEQ card (as opposed to the ROUTE XEQ card when the "Mellon Mods" or other user modifications have been applied to JES2) is quite limited, and is not granular. It just specifies a JES2 node the job should run on. But that method of control helps too. Every additional mechanism will help in some way. But what if tech support management has determined that a resource must be moved from System A to System B? All of the programmer JCL will have to be changed, on the "big day" when the move will take place. A commotion will take place among the programmers. It's generally quite painful to force programmers to globally change a lot of JCL. Changing "application programmer JCL" is a practice which has to be avoided, if possible. What would be a more ideal situation? It would be nice if programmers would be given a name of an "environment" that's "healthy" for a job to run in. They'd code this environment name in their JCL. Then it would be transparent to them, on which system these jobs actually ran. The programmers wouldn't have to think about it any more. JCL would never have to be changed. Tech support would get this responsibility, which is really in their domain anyway. Tech support would set up the "scheduling environments" properly. And each job that needed special resources, would run correctly in the proper place and the proper time, with each resource being "up" or "down", as determined by the scheduling environment. If a resource were moved from System A to System B, the programmers would never hear about it. Only the scheduling environment definitions, which are done by tech support, would have to be altered. This may sound like a more ideal world, but IBM has actually begun implementing it through the workload manager component, WLM. A new keyword, SCHENV=, which supports a "scheduling environment name" in the JOB card, has been added, provided that the installation will implement the new system. Support for the SCHENV= keyword has been introduced into JES2, JES3, and the MVS converter, to consider this keyword as valid parameter. However, at this point in time, supporting WLM Resource Affinity Scheduling is not simple. It's IBM's "first cut" in developing the facility. The individual installation has to do a lot of work to get the WLM resource scheduling mechanisms to function as they want. It is necessary to have an automated operations package running, as well as having a WLM policy in effect (not necessarily in goal mode). And all MVS images in a sysplex where WLM Resource Affinity Scheduling is to be implemented, have to be running OS/390 Release 2.4 or higher. Nevertheless, many of us feel that we are moving in the right direction. Most shops which need such services, are running home-grown packages, or are doing makeshift procedures to work around the fact that a supported Resource Affinity controlling mechanism for jobs, started tasks, and CICS transactions hasn't been there. The Mellon Mods to JES2, which are available on the JES2 SHARE Tape, have also provided a lot of flexibility in this direction for those shops which run them. But the Mellon Mods highly modify JES2, and as of this writing, they are only supported up until OS/390 Release 1.3. It is a great relief that IBM itself is finally addressing the problem of Resource Affinity control, and that there finally exists an IBM-supported mechanism which we can convert to. CONVERSION TO WLM RESOURCE AFFINITY SCHEDULING My friend Bob Break, of St. Louis, Missouri, is highly involved in Resource Affinity Scheduling issues. Bob works for a very large shop, and he is an expert in JES2 internals. (Several months ago, we mentioned in this column that Bob wrote a dynamic JES2 exit loader program which can be obtained online at http://members.aol.com/cbttape, and which can be found on File 198 of the newer CBT Tapes - Version 418 or later.) Bob's shop is so large, that resource affinity (which job will run on which machine) has to be automated. To accomplish this, Bob wrote a home-grown package, called PRO. PRO allows the detailed specification of each job's resource requirements in the JOB card, through the use of "symbolic job classes", which Bob invented. Bob's "PRO" package is complicated, and it only runs at his shop--nowhere else. Nevertheless, I'll describe the externals of PRO here, so we can better understand the problem of conversion to WLM Resource Affinity Scheduling. The main feature of PRO is support of "symbolic job classes". The user at Bob's shop will code CLASS=xxxxxxxx (from 2 to 8 letters) in the JOB card, if the job has special execution requirements. That's different from the normal job classes, which are limited to only one character. The ordinary jobs, which can run anywhere, are submitted with a normal, one-character job class coded. This is what PRO does: PRO includes a JES2 job card exit (Exit 2), which does two things. First, the exit parses the CLASS= keyword parameter to see if it has more than one character. If the user has coded one character, that's regarded as a normal job class. If the CLASS keyword has 2 or more characters, the exit validates it against an installation-defined table, and converts it to a one-character job class under the covers, so the MVS converter is satisfied. Also, the exit saves the symbolic jobclass name, which really signifies an execution environment, in a job-related JES2 control block for later retrieval (I think it gets saved in the JQE). All initiators at Bob's shop are drained by default. The shop's automation package (OPS/MVS) is signalled by PRO that the job wants to execute. OPS/MVS sets up the proper execution environment for the particular job, using the requirements specified by the symbolic jobclass name, which PRO has retrieved from the JES2 control block. Then, OPS/MVS makes sure the proper resources are available, and starts up an appropriate initiator on the proper system. When the job has finished running, its initiator is drained again. All of this is done automatically, and transparently to the user. That's roughly how the system works. To convert this system to WLM Resource Affinity Scheduling, Bob's shop set up WLM scheduling environment names, to match the existing symbolic jobclass names. The WLM scheduling environment was installed in parallel to the existing scheduling environment. Since WLM knows nothing about PRO's symbolic job classes, the WLM setup had to be done from scratch. OPS/MVS had to be programmed to handle the WLM information, as well as PRO's information. The two systems were thus set up to run in parallel. In other words, once WLM Resource Affinity Scheduling was set up, the SCHENV= keywords, using the same names as the CLASS= keywords (if they are symbolic jobclasses), should have the same effect. Bob wrote a JES2 converter exit (Exit 6), which automatically converts a symbolic job class name, if one exists, to a WLM scheduling environment name. Exit 6 fills in the value in the SCHENV= keyword, turns off the PRO controls, and sends the job on. With this in place, the old JCL will still work. But WLM Resource Affinity Scheduling will now supervise where and how the job will run, instead of PRO. CONVERSION FROM THE MELLON MODS Shops that run the Mellon Mods to JES2 will use extensions of the normal /*ROUTE XEQ card to support (it's brand of) resource affinity scheduling. The Mellon Mods allow the shop to specify an execution environment name in the ROUTE XEQ card, instead of merely a node name. These special execution environment names are stored in an internal table, and are used to set up the job properly, to run on the appropriate MVS image where the resources are. In other words, the Mellon Mods allow the coding of installation-specified names similar to PRO's symbolic jobclass names. So the conversion to WLM management should be similar in method, but different in detail. If a shop running the Mellon Mods is to be converted to WLM Resource Affinity Scheduling, an automated operations package has to be in place, and the WLM scheduling environment names have to be set up, so they match the Mellon execution environment names. The WLM Resource Affinity Scheduling environment has to be set up in parallel to the old environment. Finally, JES2 exits have to be written to transparently convert the /*ROUTE XEQ environment names to SCHENV= keywords. Bob Break has done this JES2 exit writing for you. Bob has coded a JES2 JECL exit (Exit 4) and a JES2 converter exit (Exit 6), to transparently convert ROUTE XEQ names to SCHENV= keywords. Bob's exits will soon be available on the CBT Tape (Version 420), but if you can't wait that long, they'll be posted on http://members.aol.com/cbttape, so you can download them. WHAT'S IN IT FOR THE REST OF US? To those of you who aren't running the Mellon mods (which is nearly all of us), what is the value of this discussion? First, it informs us that IBM is actually starting to develop a supported way of controlling when and where a job will run, so it gets the required resources it needs, in the proper state (up or down), at the proper time. Second, if your shop already has such a system, albeit home-grown, we've discussed possible conversion paths. If your own resource affinity job scheduling system is driven by installation-defined names, JES exits can be written to transparently change these names to SCHENV= keywords, so the global JCL change impact will be softened. Your old JCL will actually feed the work to the new WLM controls. Third, the new WLM-controlled system promises to allow resource scheduling of CICS transactions and other system work, as well as batch jobs. If you're interested in finding out more, you should contact your IBM representatives for the latest available literature, as WLM Resource Affinity Scheduling is improved and developed further. Bob Break can be reached at bbreak@swbell.net . I hope this discussion has opened your eyes to a new IBM direction that'll help your shop somewhere down the road. Good luck. See you next month.