Downloads
CBT
Top 10
Files
Updates
Overflow
JES3
JES2
CBT249 & Older
UCLAmail
Xmit Manager
XMIT Viewers
MVS 3.8J
VM/370 R6
OS/360
OS/VS1
PDS 8.6
Linux/390
MVT Mods
Fish's Herc Tools
PC Tools
AWSTAPE
DSS-FDR Format
AWS Format
ZIP Format Data
Information
Order CDROM
Newsletter
FAQ
Links
Mail & Newsgroups
Fun
Features
Steve Samson
Contribute files
Freebies
Linking to CBT site
Advertising here
Complaints
Workbench
Cross Links
AWSTAPE Utilities and information
-
What are .AWS nee AWSTAPE files?
An AWS file is a portable virtual tape. The format was developed and documented, although not officially, by IBM. It is used in the IBM P/390 machines, the freeware Hercules emulator, and most of the commercial emulator products provide support for AWS format, as well. An AWS tape is a single binary file, which when mounted using the correct commands for your emulation platform, appears as a real tape to the guest operating system. Tools also exist to import/export real tapes to virtual, and to directly create an AWS file or extract from an existing one, without using a real tape drive. Currently the most robust tools are all on MVS.The CBT BASE and OVERFLOW tapes are available to download in AWS format.
MVS Tools
File # 477 AWSUTIL to create "virtual tape" disk files
File # 478 RAWSTAPE from Jan Jaeger
File # 533 AWS Tape <---> Real Tape Conversion Utilities on MVS
File # 585 AWSSL -
Dataset to SL AWS Tape Utility - Reed Petty
VM Tools
- will be added soon
VSE Tools
- will be added soon
Windows Tools
Rob Story AWS File Browser is a windows tool to browse the contents of an AWSTAPE file
Sam Golob's article on AWSTAPE
MVS TOOLS AND TRICKS OF THE TRADE
DECEMBER 2001
Sam Golob
MVS Systems Programmer
P.O. Box 906
Tallman, New York 10982
Sam Golob is a Senior Systems Programmer. He also participates in library tours and book signings with his wife, author Courtney Taylor. Sam can be contacted at sbgolob@cbttape.org. The Online CBT Tape web site has its own URL, which is www.cbttape.org.
VIRTUAL TAPES
Today I'm going to talk about a subject that many of us currently aren't closely connected to. But I can almost guarantee that in the foreseeable future, even those of us who never heard of this subject, will have to deal with it on a daily basis.
One often does not foresee the future. You never know when a "minor development on the outskirts of computing" will make a deep, lasting impression and will have a permanent effect. One of these "minor developments" is the use of "virtual peripheral devices" that are emulated by disk storage. For example, if you are running MVS on a P390 or a system that is similar in concept (such as Hercules), you have defined "tape drives" and "3380 or 3390 disk packs" that are really PC or OS/2 disk files. Someone who has, for years, run MVS on large mainframe machines, has no exposure to these virtual devices. However, tape files that have been created as "emulated tapes", and which are really disk files, will soon be sitting at your doorstep, knocking on your door.
Please allow me to mention something else which relates to this subject. For years, I have seen misconceptions which people have had about copying tape data. Many (otherwise knowledgeable) folks feel that IEBGENER is about the only tool you need, to copy tape files. This (in my opinion, erroneous) idea, is based on the concept of a "tape file" as the principal entity you're handling, and not on the more realistic idea of handling "the entire tape" as the principal entity. A program such as COPYMODS (from File 229 of the CBT Tape collection of free MVS utilities) copies, and handles, an entire tape, as though it were ONE object. It doesn't matter if there are many hundreds of separate files on that tape--COPYMODS will deal with the entire tape as a unit, even while copying each file of it. The newer versions of COPYMODS (currently at Level 088) can CHANGE the copied tapes in various ways, such as altering the Volume Label, but in concept, a program like COPYMODS will handle an entire tape as though it were one object, even with its new bells and whistles that can alter the tape label contents.
Why am I bringing up this apparent digression? Because it pictures the idea of an entire tape, with all its files, as ONE object, and not merely as a COLLECTION of the possibly large number of FILES that are contained in that tape.
The idea of a "virtual tape" as a disk file whose scope encompasses an entire tape from end to end, as ONE object, is about to burst on the MVS scene.
Current Implementations of Tapes as Disk Files - the VTS
At this time, many installations are installing Virtual Tape Systems (VTS boxes). These are really separate computers, to be attached to an MVS system, and they act as though they are large libraries of tape volumes. The devices employ large amounts of real memory and disk storage, and they contain backup tape robots, to be used when the memory and disk storage overflow. The very idea of a Virtual Tape System is to simulate tape mounts with disk files. If a Virtual Tape System is hooked up to MVS, and a job calls for a tape mount on it, the "tape" is really a disk file, unless it has been archived.
The way Virtual Tape Systems are constructed, we, as MVS users, have no access to the actual file structure of the "tape volume" as a disk file. We only care about that VTS "tape" as though it were really a tape. However, the P390 "emulated tapes" are an entirely different story. It is possible, through an understanding of the file structure of the disk file which emulates the tape, to be able to transport that file to other MVS systems, and even to create a "real tape" on a tape drive, from it.
The rest of this column will describe the "AWS" disk file format which represents an entire tape on disk.
The AWS Disk Format of an Entire Tape
Tapes are by nature sequential. They contain blocks of data, which are organized into files. To represent a tape on disk, you must know when a file begins, when each block of data begins and ends, and when each file ends. Suppose you picture the disk representation of a tape. It has to be a sequential file. But the beginning and end of the tape files, as well as the blocks of data which comprise them, must be clearly marked. In the AWS Tape format on disk, the marking is done by means of 6-byte headers, that are placed in between the blocks and files of "tape" data.
The concept of the AWS Tape header format on disk is very easy to grasp. Every header contains the number of bytes in the data block to follow (it might be zero), the number of bytes in the previous block (which also might be zero), and an indicator whether this header marks the end of a file. If that indicator is not on, then we are at the beginning, or in the middle, of a file. If it is on, then we are at the end of a file. Tape Labels are themselves only files on a tape, which "sandwich" the data files. Therefore, Standard-Labeled tapes, and Non-Labeled tapes can both be represented in this AWS format. Conceptually, that's really all there is to it.
But there is one more small idea which contributes to the picture. On a PC disk, or as an OS/2 file, the AWS-format "tape" is conceptually one continuous stream of data, without blocks. But on an MVS system, every file has to be logically broken into blocks. In normal MVS data management, there's no such thing as a logically continuous stream of data which is arbitrarily long. But on a PC or OS/2 disk file, even though the data is broken up into hardware sectors, the file is logically one big continuous piece.
So a small barrier has to be crossed, when you are thinking of "porting" AWS Tape data on disk, from a PC or OS/2 file to an MVS system. You have to arbitrarily break the data into blocks.
Actually, when you upload any PC data to an MVS system, you have the same situation. Usually, the FTP or TSO administrator (when setting up default IND$FILE uploads) makes a decision as to whether the default MVS file after an upload, will be FB or VB, and how big the LRECL and BLKSIZE will be. You can override their decisions, but some such decision will have to be made. The "blocked" nature of MVS data, forces you to make some decision as to how to re-block any uploaded data.
Therefore, an AWS "tape" file on disk, when uploaded to an MVS system, has to be artificially re-blocked. Some people (like Brandon Hill in his AWSUTIL program - File 477 of the CBT Tape) have decided to block the data with RECFM=VB. I have decided, with my new set of AWS utilities (File 533 of the CBT Tape) to block the data as card-image files: RECFM=FB and LRECL=80, with any convenient blocksize. If you FTP the MVS data back down to a PC file, you again can get a usable and "continuous" PC file. This MVS re-blocking only "means something" if the data is to be further manipulated on an MVS system.
Specifics of the AWS Tape format
Now we'll talk about the specific structure of the AWS headers, which delimit and describe the tape "blocks" and "files" in the AWS disk representation of a tape. Each six-byte AWS header is divided into 3 two-byte sections. And the first six-bytes of any AWS "tape" file, is always the first header, which contains enough information to find all the rest of the AWS headers in the file.
The first two-byte section of the AWS header contains the number of bytes to be found in the next block of tape data. Since this is a two-byte field, the number contained in it, can be as large as 65535, or 64K bytes. But there is one hitch. The number is not a normal halfword quantity. Instead, it is backwards, in the famous "Little- Endian Format".
I'll explain. Suppose we want this field to represent the number 32720, to tell us that the next block of "tape data" will contain 32720 bytes. The hexadecimal representation of 32720 is X'7FD0'. But the first halfword in the AWS header will not contain X'7FD0'. Instead, the two bytes will be backwards, and the field will contain X'D07F'. This is confusing to us MVS'ers who have never dealt with Intel or Microsoft programming. Therefore, any MVS-based program which deals with the numeric quantity contained in this two-byte header field, must first reverse the two bytes. Little-endian is ubiquitous here.
The second two-byte field in the AWS header, contains the number of bytes of tape data contained in the previous "block". Of course, if our header is at the beginning of a file, there are no previous blocks in this file, and the second field will contain X'0000'. Remember that this second two-byte field is also in little-endian format, and any MVS program dealing with the quantity, will have to reverse the two bytes.
The third two-byte field is the simplest of all, in the current implementation of AWS tape format. The last 3 hex numbers have to be zeros, and the first number is either a X'A' or a X'4'. A X'A' means that this header does not mark the end of a file. A X'4' means that is does mark the end of a file. Officially, in the description of the AWS tape format, there's a bit more detail, to cover data chunks that are smaller than the size of the tape blocks. But practically, that's all there is to it!
So How Does It Work?
Let's look at the first few headers of a Standard-Labeled (SL) AWS tape. We'll start with the representation of the first header labels at the beginning of the tape, including the VOL1 label, the HDR1 label, and the HDR2 label, each of which is an 80-byte record. These three 80-byte records constitute the first actual "file" of an SL tape.
The sequence of AWS header records describing this first label file is:
X'50000000A000' , then X'50005000A000' , then X'50005000A000' , then X'000050004000' .
Let's interpret: The first field of the first header contains X'5000'. This is little-endian format, so the bytes have to be reversed, to make X'0050', which is decimal 80. That tells us that the following block (really the VOL1 label) will contain 80 records. The second field of the first header contains X'0000', which reversed, is still X'0000', so we know that no file data has preceded this header. The third field is X'A000', so we know we're not at the end of the file yet. After advancing 80 bytes past the end of the first header, we come to the second header.
The second header (really before the HDR1 label) contains the quantity X'50005000A000'. First field X'5000', when reversed, gives X'0050', telling us that another 80 byte block of data will follow this header. The second field X'5000', which when reversed is X'0050', lets us know that the previous block of data contained 80 bytes also. Finally, the X'A000' in the third field, tells us that we're not at end-of-file yet. The first field has told us to advance 80 bytes past the end of this header, to come to the next header.
The third header in our sequence of four, contains the same quantity as the second header, namely X'50005000A000'. This tells us that the block which follows (the HDR2 label) will contain 80 bytes, the previous block (the HDR1 label) contained 80 bytes too, and the X'A000' in the third field tells us that we are still not at end-of-file.
Finally, after advancing another 80 bytes, which the first field in the third header has ordered us to do, we come to the fourth header. This fourth header contains X'000050004000'. Its first field, X'0000', when reversed, still X'0000', tells us that zero bytes of data will follow this header. The second field, X'5000', which reversed, is X'0050' or decimal 80, tells us that an 80-byte block has preceded this header. Finally, the X'4000' in the third field, informs us that we have reached the end of this file. Since the first field of this header contained X'0000', we expect another header to immediately follow this header, in the AWS tape file.
Some Practical Correspondences
Now that we've seen an entire file in AWS format, we can figure out what the headers signify to us, in conventional tape terms. Let's first consider that we're writing this tape, which the AWS headers are describing, to a real tape drive.
First, whenever there's a X'4000' in the third field of a header, that means that we write out the last block of tape data (on a real tape) and then we write a tape mark. If the third field is X'A000', but the second field is not zeros, then we write out the number of bytes in the second field, without writing a tape mark. Finally, if the second field is zeros, and the first field is not, we advance the tape by the (reversed) quantity in the first field, reading in that number of bytes of data. No writing is done yet.
Now, let's imagine that we're reading this tape. The first header tells us that we must look into the first field. If that is not zero, then we reverse the two bytes, and read forward, by the number of bytes thus described, coming to the second header. If the third field in the second header is X'4000', we are at end-of-file. Otherwise, we look at the second field in the second header. That should correspond with the number of bytes we just read. Finally the first field in the second header tells us, that we must advance that number of additional bytes of data, to get to the third header, and so on.
I hope that this month's topic has opened your eyes to the beginning of a new era in "tape" processing. Files describing an entire tape, can now be put on disk. On an MVS system, whether these files be "folded over" in FB-80 format (as in CBT File 533), or VB format (as in CBT File 477), the data can still be dealt with, as tape data. I have written a program to convert the FB-80 format data into a real tape on an MVS system (see File 533 - program VTT2TAPE). I sincerely hope that you will all expand your minds to include these new concepts in your thinking. Best of everything! See you next month.