Hercules Diagnostic Tools


Back in the "good old days" before Hercules, systems programmers that were lucky enough to get standalone machine time at all usually had to drag themselves into the datacenter at odd hours. With the advent of the Hercules "personal mainframe" that is no longer necessary.

This section of the tutorial will introduce you to several of the Hercules diagnostic tools you can use just as if you had standalone time on a mainframe.

Standalone environment

Standalone time might be something with which you are not familiar. Generally speaking, it roughly equates to "single user mode." If you are familiar with the TSO TEST command, you can roughly equate standalone testing with debugging under TSO TEST except that your efforts affect the entire S/370 machine.

The normal niceties of bringing up all the production facilities aren't necessary unless you want/need them. Likewise, shutting down the standalone system can be as brutal as you wish. Frequently I don't even bother bringing down JES2 on my test system; I just bring down Hercules and walk away. Of course, you have to know what you're doing to avoid causing later problems, but with most of the MVS38j software we'll be using it's fairly difficult to break something.

Hercules console

Hercules offers two main user interfaces that act as the Hareware Management Console (HMC): the line-mode oriented console and the GUI (currently only available for the Windows implementation). We will be using the line-mode oriented console function.

Here's what I see on the Hercules console when I bring up my (somewhat modified) Turnkey3 MVS system:

Hercules Version 3.00a.2
(c)Copyright 1999-2003 by Roger Bowler, Jan Jaeger, and others
Built on Nov 12 2003 at 20:16:23
Build information:
  Modes: S/370 ESA/390 ESAME
  Using setresuid() for setting privileges
  With Dynamic loading support
  Loadable module default base directory is /usr/local/lib/hercules
  Using shared libraries
  No external GUI support
  HTTP Server support
  HTTP document default root directory is /usr/local/share/hercules
  National Language Support

Running on Linux i686 2.4.20-4GB-athlon #1 Mon Mar 17 17:56:47 UTC 2003
HHCCF065I Hercules: tid=00004000, pid=3699, pgid=3695, priority=0
HHCTE001I Console connection thread started: tid=00008003, pid=3702
HHCTE003I Waiting for console connection on port 3270
HHCSD020I Socketdevice listener thread started: tid=0000C004, pid=3703
HHCSD004I Device 000C bound to socket localhost:3505
HHCDA020I /herc2/turnkey3/dasd/mvsres.148 cyls=560 heads=30 tracks=16800 trklen=19456


HHCDA020I /herc2/turnkey3/dasd/smp001.149 cyls=560 heads=30 tracks=16800 trklen=19456
HHCDA020I /herc2/turnkey3/dasd/smp002.14a cyls=560 heads=30 tracks=16800 trklen=19456
HHCDA020I /herc2/turnkey3/dasd/smp003.14b cyls=560 heads=30 tracks=16800 trklen=19456
HHCDA020I /herc2/turnkey3/dasd/smp004.14c cyls=560 heads=30 tracks=16800 trklen=19456
HHCCP002I CPU0000 thread started: tid=0001C008, pid=3759, priority=15
HHCCP003I CPU0000 architecture mode S/370
HHCTT002I Timer thread started: tid=00020009, pid=3760, priority=0
HHCPN008I Script file processing started using file hercules.rc
> * Running /herc2/turnkey3/hercules.rcHHCPN011I Pausing SCRIPT file processing for 2 seconds...
HHCPN001I Control panel thread started: tid=00004000, pid=3699
HHCPN012I Resuming SCRIPT file processing...
sh /herc2/turnkey3/termup-turnkey3
starting konsole telnet for hardcopy console
HHCTE009I Client connected to 3215 device 0009
HHCTE009I Client connected to 3270 device 0010
HHCTE009I Client connected to 3270 device 00C0
HHCTE009I Client connected to 3270 device 00C1
HHCTE009I Client connected to 3270 device 00C2
HHCPN011I Pausing SCRIPT file processing for 2 seconds...
HHCPN012I Resuming SCRIPT file processing...
ipl 148
HHCPN013I EOF reached on SCRIPT file. Processing complete.
CPU0000: SIGP CPU0001 Initial program reset PARM 00000000


HHCCD001I Readahead thread 1 started: tid=0004800D, pid=3805
HHCCD001I Readahead thread 2 started: tid=0004C00E, pid=3806
HHCCD002I Writer thread 1 started: tid=0005000F, pid=3807
HHCCD003I Garbage collector thread started: tid=00054010, pid=3808
HHCCD002I Writer thread 2 started: tid=00060013, pid=3811

At this point, I have the Hercules console session, four 3270 (TN3270) sessions (one master console and three TSO terminals), and a telnet session to the 3215 console.

The default Turnkey install should provide you with something similar; if not, please feel free to ask for help in the hercules-s370asm or turnkey-mvs Yahoo! groups.

Hercules commands

Hercules provides a wealth of commands that you can issue from the HMC (also called the "panel"), many of which mirror the operations you can perform on real S/370 hardware. There are several HTML and man pages included in recent Hercules distributions, and you should make good use of them as they are the official description of how Hercules should operate. Hercules also provides an online help facility via the "?" command:

HHCPN140I Valid panel commands are...

  Command      Description...
  -------      -----------------------------------------------
  ?            list all commands
  help         command specific help

  hst          history of commands

  quit         terminate the emulator
  exit         (synonym for 'quit')

  cpu          define target cpu for panel display and commands

  start        start CPU (or printer device if argument given)
  stop         stop CPU (or printer device if argument given)

  startall     start all CPU's
  stopall      stop all CPU's

  .reply       scp command
  !message     scp priority messsage

  ptt          display pthread trace
  i            generate I/O attention interrupt for device
  ext          generate external interrupt
  restart      generate restart interrupt

  store        store CPU status at absolute zero
  archmode     set architecture mode
  loadparm     set IPL parameter
  ipl          IPL from device xxxx

  psw          display program status word
  gpr          display general purpose registers
  fpr          display floating point registers 
  cr           display control registers
  ar           display access registers
  pr           display prefix register 
  clocks       display tod clkc and cpu timer
  ipending     display pending interrupts 
  ds           display subchannel
  r            display or alter real storage 
  v            display or alter virtual storage
  u            disassemble storage 
  devtmax      display or set max device threads
  k            display cckd internal trace

  attach       configure device 
  detach       remove device
  define       rename device 
  devinit      reinitialize device
  devlist      list all devices

  sh           shell command 
  cache        cache command
  cckd         cckd command 
  shrd         shrd command
  quiet        toggle automatic refresh of panel display data

  b            set breakpoint 
  b-           delete breakpoint
  g            turn off instruction stepping and start CPU

  pgmtrace     trace program interrupts 
  savecore     save a core image to file
  loadcore     load a core image file 
  loadtext     load a text deck file
  ldmod        load a module
  rmmod        delete a module 
  lsmod        list dynamic modules
  lsdep        list module dependencies
  iodelay      display or set I/O delay value
  toddrag      display or set TOD clock drag factor 
  panrate      display or set rate at which console refreshes
  syncio       display syncio devices statistics 
  script       Run a sequence of panel commands contained in a file
  cscript      Cancels a running script thread
  evm          ECPS:VM Commands (Deprecated) 
  ecpsvm       ECPS:VM Commands
  aea          Display AEA tables 
  sf+          add shadow file
  sf-          delete shadow file 
  sf=          rename shadow file
  sfc          compress shadow files 
  sfd          display shadow file stats
  t{+/-}       turn instruction tracing on/off 
  s{+/-}       turn instruction stepping on/off
  t{+/-}dev    turn CCW tracing on/off 
  s{+/-}dev    turn CCW stepping on/off
  t{+/-}CKD    turn CKD_KEY tracing on/off 
  f{+/-}adr    mark frames unusable/usable

Some Hercules commands have specific help, which you can display with "help command".

Your help display may be slightly different than what appears above, depending on what version of Hercules you are using (and the fact that my version of Hercules is mildly downlevel).

Hercules breakpoint command

Just as in a real S/370, Hercules lets you set hardware breakpoints. While the operator procedure for setting breakpoints on real S/370s varied somewhat depending on CPU model, it is always the same under Hercules:

b address       # set breakpoint
b-              # turn off breakpoint

Note Hercules only provides for one breakpoint at a time to be active.

Before we set the breakpoint and find out what it does, we need to know where to set the breakpoint.

Fortunately there is a relatively easy way that should work just fine for our simple programs, as long as we follow one simple rule when linkediting our program: always specify PAGE alignment.

We use a program designed to "blow up" to produce an abnormal termination dump which we examine to determine the typical address at which batch programs are loaded by MVS.

Using JCL similar to the "Assembling, Linking, and Executing IEFBR14" tutorial page, the jobstream appear as below:

//SYSIN     DD *
         DC    H'0'
         END   ,
//             COND=(5,LT,ASMF),
//          DD *

In particular, note the "PAGE ABEND0C1" statement in the LKED step. The ABEND0C1 matches the CSECT name ABEND0C1, and when MVS loads the ABEND0C1 load module this CSECT will be placed on a page boundary.

For various reasons (such as different PARM= values on the JCL EXEC card) MVS sometimes loads the first load module in a batch address space at different addresses. Specifying PAGE alignment will usually stabilize such "address jitter," thus providing a consistent beginning address for your programs.

Unfortunately, it is quite possible to modify your MVS system in such a manner that that "consistent" address will move around. What we are accomplishing by running the ABEND0C1 program is determining as best we can what that "consistent" address is in your system as currently configured.

Before running the ABEND0C1 job, issue the following Hercules command in order to separate the job's output from anything else you might have placed on the printer:

devinit e prt/abend0c1.prt

OK, run the ABEND0C1 job shown above (the RPF "sub" command is a handy way to accomplish this), and retrieve the printer output (there will be quite a bit of it, on the order of several hundred lines).

Assuming everything worked as planned, when we scroll down past the linkage editor output, should we see something like:

JOB ABEND0C1         STEP ABEND0C1        TIME 203136   DATE 76320    ID = 000


PSW AT ENTRY TO ABEND  078D0000 000A5002        ILC 2   INTC 0001

While there is a lot more to dump reading than we'll cover here, the important thing to notice is the PSW AT ENTRY TO ABEND 078D0000 000A5002. Subtracting the opcode length (ILC 2) from A5002 produces the address at which our ABEND0C1 program was loaded, which is A5000 (at least on my Turnkey system).

Take a moment to browse through the remainder of the SYSUDUMP output (the "dump") if it interests you, but don't be concerned if little of it makes sense.

We have now located where normal batch programs are loaded in our MVS system. Happily, similar (but more condensed) versions of this information are also present on the Hercules console:

HHCCP014I CPU0000: Operation exception CODE=0001 ILC=2
PSW=078D0000 000A5002 INST=0000         ?????
GR00=009ECD00	GR01=000A4FF8	GR02=00000040	GR03=009E1634
GR04=009E1610	GR05=009EC8A8	GR06=009C3018	GR07=FD000000
GR08=009EC108	GR09=809EC2B8	GR10=00000000	GR11=009EE080
GR12=40EB7B9A	GR13=000A4FB0	GR14=00019DC0	GR15=000A5000
CR00=C080EC40	CR01=0FC6BC00	CR02=FFFFFFFF	CR03=00000000
CR04=00000000	CR05=00000000	CR06=00000000	CR07=00000000
CR08=00000000	CR09=00000000	CR10=00000000	CR11=00000000
CR12=00000000	CR13=00000000	CR14=EFC00000	CR15=00FDD368

A quick breakdown of what Hercules has shown us about the Operation Exception (produced by our attempt to execute an invalid X'00' opcode) shows a description of the exception, the PSW (Program Status Word), the GPRs (General Purpose Registers), and the CRs (Control Registers).

Note that if you don't see the Operation Exception display on your Hercules console you may have specified an OSTAILOR value (such as QUIET) which is preventing you from seeing the display. I normally comment out the OSTAILOR QUIET command in my Hercules configuration file and accept the default, which will provide some "extraneous" things that look like ABENDs but at least you can see what happens when your program "blows up".

If you want to change the OSTAILOR setting, you might be able to do it on the Hercules console; if not, you'll have to modify your Hercules configuration file and restart Hercules. Whether you'll succeed at changing OSTAILOR from the Hercules console will depend on what version of Hercules you're running.

While we've got the GPRs so handily displayed for us, let's revisit a topic we previously mentioned: standard MVS linkage conventions.

Standard MVS Linkage Conventions

The contents of the GPRs when our program receives control from MVS look like this:

R1   address of standard MVS parmlist (JCL PARM=)
R13  address of standard MVS savearea provided by MVS
R14  return address to MVS for when our program completes
R15  entry point address of our program

Other register's contents are considered "undefined"

Now that we have a bit more insight into the register contents when our program receives control, let's stop the virtual CPU just before it reaches our invalid X'00' opcode by issuing the Hercules breakpoint command before we run the ABEND0C1 program again:

b a5000
HHCPN040I Setting breakpoint at 00000000000A5000

The "b a5000" command produces the HHCPN040I message which Hercules issues to acknowledge acceptance of the breakpoint command.

OK, resubmit the ABEND0C1 job.

You'll probably notice that your TSO terminal that you used to submit the job is now "dead"; likewise the master console. Indeed, the whole S/370 CPU is "hung" on the breakpoint at A5000 and the Hercules console has displayed the same message as before (abbreviated here to ignore the Control Registers about which we don't really care):

HHCCP014I CPU0000: Operation exception CODE=0001 ILC=2
PSW=078D0000 000A5002 INST=0000         ?????
GR00=009ECD00	GR01=000A4FF8	GR02=00000040	GR03=009E1634
GR04=009E1610	GR05=009EC8A8	GR06=009C3018	GR07=FD000000
GR08=009EC108	GR09=809EC2B8	GR10=00000000	GR11=009EE080
GR12=40EB7B9A	GR13=000A4FB0	GR14=00019DC0	GR15=000A5000

Eerie, isn't it? Hercules isn't running any S/370 instructions.

Ah, but Hercules commands are still working. Let's use this instruction interlude to poke around in main storage and see what we can learn.

Displaying Virtual Storage

First, let's check out where R15 points. As you'll recall, R15 should point at our program's entry point.

To display virtual storage, we use the Hercules "v" command:

v a5000
V:000A5000 (primary) R:00BD7000
V:000A5000:K:8E=0000C1C2 C5D5C4F0 C3F140F1 F161F1F5  ..ABEND0C1 11/15
V:000A5010:K:8E=61F7F640 F2F14BF2 F5000000 00000000  /76 21.25.......
V:000A5020:K:8E=00000000 00000000 00000000 00000000  ................
V:000A5030:K:8E=00000000 00000000 00000000 00000000  ................

Sure enough, there's our program with the halfword of X'0000' and the eyecatcher consisting of our program name, the date (adjust by the SYSEPOCH 1928 Hercules configuration command), and the time our ABEND0C1 program was assembled.

Next, let's look at where R1 points, which you will recall should point at the standard MVS parameter list:

v a4ff8
V:000A4FF8 (primary) R:00BBBFF8
V:000A4FF8:K:8E=800A4FFE 00000000                    ....
V:000A5008 (primary) R:00BD7008
V:000A5008:K:8E=C3F140F1 F161F1F5 61F7F640 F2F14BF2  C1 11/15/76 21.2
V:000A5018:K:8E=F5000000 00000000 00000000 00000000  5...............
V:000A5028:K:8E=00000000 00000000 00000000 00000000  ................

The format of the MVS parameter list is based on the following layout:

R1  address of fullword containing address of halfword containing
    length of the JCL PARM= string, immediately followed by the
    actual PARM string

The high order bit of the address pointed to by R1 will be B'1'
to indicate the end of the address portion of the parmlist.
This B'1' is frequently called the "VL" (variable length parmlist) bit.

So for the fullword containing an address, we see X'800A4FFE'. This is the address of the halfword describing the PARM string's length. Let's be lazy and display the parm string rather than digging it out of the previous display:

v a4ffe
V:000A4FFE (primary) R:00BBBFFE
V:000A4FFE:K:8E=0000                                 ..

And there we have it! A halfword of X'0000', indicating that the PARM string's length is zero (meaning no JCL PARM= value was provided for the ABEND0C1 execution step).

Let's backtrack and look more carefully at the data Hercules shows for the "v" command, using the ABEND0C1 program storage as an example.

V:000A5000 (primary) R:00BD7000
V:000A5000:K:8E=0000C1C2 C5D5C4F0 C3F140F1 F161F1F5  ..ABEND0C1 11/15
V:000A5010:K:8E=61F7F640 F2F14BF2 F5000000 00000000  /76 21.25.......
V:000A5020:K:8E=00000000 00000000 00000000 00000000  ................
V:000A5030:K:8E=00000000 00000000 00000000 00000000  ................

First, we see a header line describing a section of storage (presumably

   V:000A5000 (primary) R:00BD7000

      The "V:000A5000" means this is a virtual address, and the address of
      that virtual storage.

      Ignore the "(primary)" designator, it has to do with whether the
      related segment table represents the primary, secondary, or home
      address space.

      Finally, we see the real address which maps to the virtual address,
      indicated here by "R:00BD7000".

Following the storage header, we see the virtual storage contents, with
some additional information:

   V:000A5000:K:8E=0000C1C2 C5D5C4F0 C3F140F1 F161F1F5  ..ABEND0C1 11/15

      The "V:000A5000" means this is a virtual address, and the address of
      that virtual storage.

      The "K:8E" represents the storage key and related bits.

      Following the "=" equals sign, we see the hex values of storage,
      and the character representation (if any) of the storage.

We now have some pretty decent tools at our disposal to diagnose our programs: the breakpoint, and virtual storage display.

Before we introduce the next Hercules diagnostic facility, let's switch back to the IEFBR14 program so we have something more interesting to work with.