MVS TOOLS AND TRICKS OF THE TRADE
                         October 1998

                                      Sam Golob
                                      MVS Systems Programmer
                                      P.O. Box 906
                                      Tallman, New York 10982


Sam Golob is a Senior Systems Programmer.  He also participates in
library tours and book signings with his wife, author Courtney Taylor.
Sam can be contacted at sbgolob@cbttape.org.  Information about the CBT
MVS Tapes can be found on the web, at http://www.cbttape.org.


AN IMPORTANT Y2K TECHNIQUE


      As the "magic deadline" of January 1, 2000 approaches, it may be
necessary for us systems programmers to make some sweeping program
changes, very quickly, on that morning.  There is one technique that
once literally "saved my company" in such a situation, and I feel
morally bound to teach it to every systems programmer in the world,
if I can get to them, and if they'll listen.  It is absolutely
essential that each and every one of you knows about this technique.
If you don't have it (or an equivalent) in place and ready to use, you
could be lucky, but you may be very sorry.  It could cost your company
many hours or days of production time, and possibly, for some people,
their jobs.  This is serious business, and it's our business!

      As far as I know, IBM doesn't have an easy way to do this job
either, so I don't think you'll get much help from them, unless they
change things very much between now and that big January 1st.  I do have
an easy and quite painless way to do this almost impossible-looking task
in a very short time.  Here it is.  Please listen well, for your own
good.


THE SUBROUTINE REPLACEMENT PROBLEM.

      I faced this problem on January 1, 1991.  A time-dependent
subroutine which did date conversion for our company's production
processing, failed on that January 1st.  The consequences of the
failure were catastrophic--a user abend was produced by the
subroutine when it didn't work.  Batch programs which used this
subroutine, failed with the user abend.  CICS and other online
transactions which used this subroutine, did worse.  At that time,
each CICS region used only a single TCB--remember?  In other words, a
failure of the subroutine in any transaction, would bring the entire
CICS region down.  Moreover (and this is the kicker), this subroutine
was directly linkedited into many of the company's batch and online
production programs.  We didn't know which programs contained it.
There were upwards of 3300 load modules in the batch load library.
And there were more than 1100 load modules in the online load library.
Using "standard IBM methods", this problem would take months to fix!
Can you imagine the production impact?

      It didn't take long for a programmer to fix the actual
subroutine, which was a small assembler program, and relatively
simple.  The big question was, how could we find and replace the bad
version of that subroutine in all the load modules which contained it?

      IBM doesn't help.  Load module names don't indicate any of the
subroutine names, and ISPF member lists only list load module names,
not any subroutine names.  The standard "brute force" IBM solution
would be to take an AMBLIST listing of all the load modules to find
the presence of the bad subroutine.  (4500 load modules?)  Then you'd
have to write re-linkedit JCL to include the new subroutine that would
replace the bad one--by hand!  How could you make sure the linkedit
JCL was right?  You could cause far worse production problems, if it
was wrong.  You'd have to carefully examine the linkage editor report
for each reworked load module.  This sticky situation would take a
long time to fix, and you'd never be sure you didn't miss a load
module somewhere.

      I fixed the problem, with complete accuracy, in two hours flat,
on January 2, 1991.  And I didn't miss one load module.  Do you want
to read on, to see how to do it?


SOLVING THE PROBLEM.

      My quick solution to replacing a bad subroutine in many places,
requires the installation of a product.  The up-side, though, is that
the product is free.  Many of you have this product already installed
at your shops, however.  I've also posted the latest source code for
it on a web site, so you can download and install a new version for
yourselves.  With this product in place, you can solve the "global
subroutine replacement problem", and many others as well.

      The product we need, is the PDS program package, currently at
version 8.5, which resides on File 182 of the CBT MVS Utilities Tape,
or which can be downloaded from a web site at:

         http://members.aol.com/freepds

The CBT Tape may be obtained from the NaSPA office, and a (somewhat
backleveled) version of it can be found on the NaSPA cd-rom.  The
CBT Tape is a huge conglomeration of system programmer goodies, which
no MVS or OS/390 systems programmer should be without.  There is a
vendor-supported version of PDS, called STARTOOL, from Serena
International in Burlingame, California.  STARTOOL contains all the
functionality of "free PDS", and much more, so if your installation is
licensed for STARTOOL, you don't have to install anything else.

      The PDS package works as follows:  PDS is a TSO command, that
can be used with ISPF or without ISPF.  You point PDS at a dataset,
and then you have a very large number of choices what to do afterward.
PDS, even the free version, has perhaps 50 separate subcommands, that
allow more than 1000 separate utility functions to be done to the pds,
or to some of its members.  PDS automatically treats "load module"
datasets (RECFM=U) differently from "source type" datasets (RECFM=FB
or VB).

      The PDS package (and of course, STARTOOL) has a powerful
facility for keeping track of "member subgroups" of the current
partitioned dataset it is pointing at.  One of the main purposes of
having member subgroups, is to be able to do an operation to all the
members of the subgroup, as if they were one member.  In other words,
PDS has the capability of dealing with all the members from the
current subgroup as one unit.  You just refer to an asterisk '*'
instead of the member name, and a selected operation will be performed
on ALL members of the current "member subgroup", instead of on one
individual member.  This capability will help us very much.

      Choosing the proper member subgroup in each load library, is an
important key to solving our sticky problem.  To be specific, we'd
like to isolate a subgroup of load module members which all contain a
CSECT having a given name (the name of our subroutine).  Then we can
re-linkedit all those load modules, and those only, to fix them.  In
other words, we've got to find them, then fix them.

      Sometimes even that is not enough.  Suppose there are two
different versions of the subroutine floating around in our libraries,
with the same CSECT name.  Or perhaps (in a tougher case) there are
two entirely different programs with the same CSECT name.  We might
want to further refine our member subgroup to contain only one of
those versions or programs, but not both.  Using the PDS package, that
can be done also, without much additional effort.  We're getting an
indication at this point, that with the PDS command, we have a lot of
power.

      It remains, once we have a proper and complete list of load
modules to rework, that we need to re-linkedit them and replace the
version of the subroutine which is wrong.

      The PDS package comes up strong in that department also.  PDS
has a facility, where it can look at a load module, and generate
accurate JCL and linkage editor control statements, to re-linkedit the
load module exactly as it was before.  All the proper attributes are
generated, such as reentrancy, refreshability, APF authorization, and
so forth.  Also, ORDER statements are generated by PDS in the linkage
editor control statements, so that the re-linkedited module has all
its CSECTs in the same order that the original module had them.

      It is now obvious that we have all the ingredients in place, to
solve our problem.  Besides that, there are a few other considerations.
If your load library is a PDSE, copy it with IEBCOPY to a pds, and
then you can use the free PDS package on it.  Free PDS doesn't have
PDSE support, but STARTOOL does.  If you're using STARTOOL for this,
it doesn't matter if the library is a pds or a PDSE.  The PDS package
is written in assembler language.  It runs at assembler speed (i.e.
"quick"), and has optimized I/O routines to fetch the members of any
partitioned dataset with the greatest speed.

      For most of this, we have to thank Bruce Leland and Steve Smith.
They are the principal "enhancers" and builders of this product, and
they maintain the vendor version, STARTOOL.  John Kalinich is
maintaining the free version of PDS at this time, and he's doing a
great job, too.  So far, you've seen that PDS can do this formerly
tedious job very quickly, and now we'll look at the specifics.


CORRECTLY FIXING ONLY THE BROKEN LOAD MODULES

      In fixing our "globally scattered subroutine" problem, we must
first determine which libraries are affected.  These, most likely, will
be application load module libraries.  First, you must determine the
production libraries involved, and then, you should probably make the
same changes to the programmer libraries and development libraries, so
that the old error will never work its way back into production again.
If you have a change control system, you'll have to work within its
restrictions and parameters.  In any case, if you've anticipated the
problem, you'll be able to be orderly.  If not, you'll have to be just
as orderly, but also fast.

      Once you've got the list of libraries, you'll deal with them
one at a time.  For simplicity from now on, we'll pretend that only
one load library is affected.  But it usually isn't the actual case,
and you'll just repeat the same procedure for all the libraries that
you have to rework.

      The first task is to determine all the load modules which
contain the affected CSECT name.  This will probably be the name of
the subroutine you want to replace.  PDS and STARTOOL both have a
powerful method of finding only those load modules which contain a
CSECT with a certain name.  This is the "IF" subcommand of PDS, a
little-known but powerful tool.  The purpose of the IF subcommand is
to create a subgroup of members, based on a certain criterion.  The IF
subcommand will select a subgroup of members, starting from all
members, or it will further refine a subgroup that was previously
selected.  Let's invoke PDS, and point it to the proper library.
Under TSO, you'd say:    PDS 'library.name'    to invoke the PDS
command, and at the same time, to point it to the proper library.
Then you deploy the IF subcommand, and say:

IF : MODULE(csectnam) THEN(SUBLIST)

The IF subcommand will now look at the starting member group, which
is the colon ":".  The colon means "all members".  From this starting
member group, IF examines the structure of every load module, and if
it finds a CSECT matching the given (partial or complete) name you
said in the MODULE() keyword, it selects that load module for inclusion
in the new "member subgroup".  Otherwise IF excludes the load module
from the subgroup.  So we now know, that after this operation, every
member of the current member subgroup contains a CSECT with the given
module name.  We've done the "find" part, of the "find and fix" process.

      Well, not always.  Sometimes, there might be two or more
versions of that CSECT floating around in our libraries.  To further
select one of the two, we could do any of a number of things.  One
thing we could do, is to find a character or hex string which exists
in one of the versions, but not in the other.  This string should be
so unique, that it's not likely to be found in a different CSECT than
the one you're dealing with.  Under the PDS package, the FIND
subcommand not only finds strings, but it can select members for
inclusion into a member subgroup, based on whether a certain string is
found in the member.  FIND, when used in this way, works like IF.
Remember that the asterisk "*" refers to all members of the current
member subgroup.  Then, you can say:

       FIND * /charstring/ MODULE(csectnam)  THEN(SUBLIST)  or
       FIND * xhexstringx  MODULE(csectnam)  THEN(SUBLIST)

This operation will further refine the member subgroup, picking only
from members of the former member subgroup, which contain the string
somewhere within the named CSECT.  There are other ways you can pick
and choose members.  The ultimate aim is to make sure that you have a
member subgroup which contains every member you want, and no others.

      Now the re-linkedit part gets deployed.  The PDS subcommand
which looks at a load module and generates re-linkedit JCL
automatically is the "MAP" subcommand with its RELINK keyword.  So,
once you've determined a proper member subgroup, you use the MAP
subcommand as follows:

         MAP * RELINK

This produces an output listing to the PDS LOG (under ISPF mode) or to
the terminal (under line mode).  That output may be written to a file,
using one of the keywords of the CONTROL subcommand, so it can be
edited.  This brings us to the last step.

      The re-linkedit JCL produced by the PDS program, assumes that
the SYSLMOD library and the original library are the same partitioned
dataset--the load library that PDS was pointing at.  After PDS
generates re-linkedit JCL for all the load modules in the member
subgroup, you must edit each SYSLMOD DD statement to point to a
different, newly allocated output library, so that the re-linkedited
load modules do not replace the original members immediately, during
this re-linkedit operation.  Only when these new load modules are
tested and checked out, can you copy them over the old members in the
original library, or you should use whatever "safe" production control
procedures you have.

      The resulting JCL usually needs only a JOB card to run it,
once the SYSLMOD DD cards are fixed.  But in our case, where we need
to replace the bad CSECT, we need two more things.  A repaired
version of the subroutine, as a load module or as an object deck,
must be placed in a library, and a DD card must be inserted into
each step of the re-linkedit JCL, to point to that library where the
repaired module resides.  Then, an INCLUDE linkage editor control
card, pointing to the name of the module, and the DDNAME referring
to its library, must be inserted before the main INCLUDE cards, so
the new CSECT will replace the old one having the same name.  This
must be done for each module you want to re-linkedit.

      That being done for all the generated JCL, for all the load
modules in the member subgroup, the JCL is run, and all the necessary
load modules will have been found and fixed.  It only remains to test
the new load modules, and (once tested), to copy them back to the
original libraries.  Or you can run production from new libraries
containing the newly fixed modules, in addition to all the other ones.


DETERMINING YOUR VULNERABILITY IN ADVANCE

      It is important to look at all your production programs, and to
determine if any subroutines are used often, during the processing.
Once it is found that there are frequently used subroutines in
production processing, you can do all of the IF processing in advance,
to determine where these subroutines are.  Then, you can generate
the re-linkedit JCL in advance, so that it will be ready in case of
emergency, on January 1, 2000.  As you become more familiar with the
process, it should be redone whenever there are significant changes to
the production load libraries.  In case any of these subroutines has
to be fixed, you've got the mechanisms already there, so the fixed
versions can be put into place quickly and accurately, wherever they
belong.

      Just for reference, the SMP/E language for this situation, in a
system module context, is having multiple LMOD entries for a single
MOD.  But we're dealing with application load modules here, and we have
to do this job without the help of SMP/E and its record-keeping.

      I sincerely hope that every one of you learns this technique.
With the Y2K situation looming over us, this technique is so
important, that I cannot, in good conscience, neglect to teach it.
You might be able to easily fix a program, but replacing it wherever
it is found in production processing, may be a far harder matter.
Now, it doesn't have to be so hard.  You've just got to know how to do
it right.  Good luck.  Please learn this technique well!  We'll see
you next month.