DB2 DIAGNOSE DISPLAY MEPL

DIAGNOSE is a DB2 for z/OS utility that, as the name suggests, is used in diagnosing problems. It generates information that is useful in diagnosing problems. But you should use this utility only under the direction of IBM Software Support staff, except for DIAGNOSE DISPLAY MEPL.

You can use the DISPLAY MEPL even without contacting IBM in the following scenario.

First of all, MEPL is an acronym for Module Entry Point List (MEPL). DIAGNOSE DISPLAY MEPL dumps module entry point lists to SYSPRINT.

Recently, we had an issue with a query after applying the maintenance RSU1304. I opened a SR (Service Request) with IBM and the support staff found a PTF in PE status. It took about 2 months to find a fix and eventually when I applied a PTF to fix the PTF in PE, it didn't work. The query still had the same problem.

IBM asked me to run this utility to find if the PTF is in fact applied to the subsystem that we were testing the query. The SYSPRINT from this utility didn't show the PTF. That's when I realized that the PTF was applied to the maintenance library, but the load module was not copied to the production run time library. After copying the load module from maintenance library to run time library, the query ran good.

Usually I do search the SMPPTS library to see if PTF that was applied recently is found, but that's not foolproof. This method of running the DIAGNOSE DISPLAY MEPL utility looks foolproof. Lesson learnt !!

Here is a sample job to run this utility:

//jobcard
//JOBLIB   DD  DISP=SHR,DSN=SYS2.DB2.DBxx.SDSNLOAD
//STEP0010 EXEC PGM=DSNUTILB,PARM='DBxx,MEPL',REGION=4M
//SYSPRINT DD SYSOUT=*
//SYSUDUMP DD SYSOUT=*
//SYSIN    DD  *
  DIAGNOSE DISPLAY MEPL
/*

Thanks for stopping by.

How to load LOB tablespaces using DSN1COPY?

In the past, I've tried to load tablespaces with LOB columns using the regular IBM Load, BMC Load Plus & CA Fast Load. None of these tools support LOB tablespaces. You get into a series of errors.

The only utility that has ever worked for me is DSN1COPY. It's a pain to use DSN1COPY though. You need several things to go right for this to work. Let me try to explain as simple as possible.

In this example, I'm copying data from an imagecopy dataset into a table in a TEST subsystem. The first thing you need to remember is that, you need to know the DBID of the database, PSID of the tablespace and OBID of the table for both the source and target subsystems.

These are the steps:

1. STOP the base and LOB tablespaces and indexes for the LOB table on the target subsystem

2. Run DSN1COPY with the right parameters. I use PARM='PAGESIZE(32K),LOB,FULLCOPY,OBIDXLAT,RESET'. Supply the imagecopy dataset of the table on the source subsystem for SYSUT1 and supply the DSNDBC linear dataset of the table on the target subsystem for SYSUT2.

3. START the base and LOB tablespaces and indexes for the LOB table on the target subsystem

4. Rebuild the indexes

5. Run runstats

If the DSN1COPY step abends with "DSN1992I VSAM PUT ERROR, RPLERREG = 008, RPLERRCD = 028", check JESMSGLG. If you see a message like this "IEC070I 204-220,jobname,stepname,SYSUT2,6BF7,DBD115& IEC070I DB2CATx.DSNDBC.dbname.tsname.I0001.A001", it means you do not have enough space in that dataset. Make sure, you assign multiple datasets with suffix A001, A002, etc. in the target subsystem, it needs to accommodate all data from the source.

If you see a message like this "IEC070I 209-220,jobname,stepname,SYSUT2,89B5,DBD235,& IEC070I DB2CATx.DSNDBC.dbname.tsname.I0001.A001,", that means no space is available in the DASD volume. To fix this problem, make the dataset multi volume so that it can extend space into multiple volumes.

HTH !!

DB2 for zOS Index on Expression

DB2 for z/OS version 9 brought in a new feature called "Index on Expression". Yes, you can create indexes on frequently used expressions in your queries. For example, if you often do a SELECT FIRST_NAME WHERE UPPER(LAST_NAME) = 'SMITH', then creating an index on UPPER(LAST_NAME) will be useful.

Until V8, you can create index on just columns, not on expressions/functions etc. So, you would have created an index on LAST_NAME, but DB2 would have ignored it because when you use a function on a column, DB2 ignores index access. But starting in V9, you can create an index on UPPER(LAST_NAME) and DB2 will use this index.

Now, to the tricky part ...

I tried to create an index on UPPER(LAST_NAME), DB2 complained that I need to specify 'locale' name. After reading a little bit about 'locale names', I specified UPPER(LAST_NAME, 'En_US') and DB2 happily created the index. I also ran runstats on the table. However, when I did an explain on the query SELECT FIRST_NAME WHERE UPPER(LAST_NAME) = 'SMITH', DB2 didn't seem to pick up the newly created index. That puzzled me.

After reading a little further, I found that I need to specify the same 'locale' in the query too. After specifying the 'locale', I did an explain and DB2 picked up the index. This is the modified query:

SELECT FIRST_NAME WHERE UPPER(LAST_NAME, 'EN_US') = 'SMITH'

Keywords: DB2, z/OS, Index, Expression, Upper, Locale, function

DB2 V9 for z/OS - New ZPARM EDM_SKELETON_POOL

I'm working on upgrading DB2 for zOS from V8 to V9. I completed upgrading it to V9 CM (Compatibility/Conversion Mode) in one of our PROD subsystems about 2 weeks ago and the next business day was horrible. The performance was bad for the entire subsystem. The LPAR CPU peaked to 98% of the total MIPS capacity.

The activity rate against the SPT01 tablespace jumped several times and the CICS regions started sucking 20% more CPU. We noticed a few bad queries through Omegamon and we fixed them by either tweaking the stats or putting some new indexes. But that didn't bring down the CPU consumption much. The interesting part was, our zIIP engine started to peak at 100%, which usually stays at 40%. We found one query that had done a bunch of updates and got stuck - it was using a lot of CPU but was not doing any productive work. We cancelled this thread, but it wouldn't go away and the zIIP engine was still at 100%. Many more transactions/threads started to back log because of lack of CPU availability. The cancelled thread didn't die and still was burning CPU. We ended up cycling DB2 with a MODE(FORCE) command. Still, when DB2 came back, it started recovery of the cancelled thread, so we had to force kill it.

When that thread was at last gone, we looked at the CPU rate, and it was right back at the 98% with the only exception that zIIP processor was at 75% level.

We opened a Sev 1 PMR with IBM. As usual, they asked for dumps, SMF, RMF records etc. [Rant On] I would have thought IBM provided a solution to try for a problem with such a mature product. But instead they were waiting on us to provide docs. [Rant Off]


Anyway, we did some more research and found that, in DB2 V9, copies of packages and plans have been moved from below the 2GB bar to above the bar. This is done through a new zparm called EDM_SKELETON_POOL. The default value for this zparm is too low. So we increased it by 20 times. The system is in a much better shape now.

This is from Omegamon for DB2:
Package Table (PT) Reqs         1005483K 
PT Loads                        80073250 
% of PT Loads from DASD             7.96%


The "% of PT Loads from DASD" should be less than 10% for a healthy system. It was about 55% in our system before increasing the value of EDM_SKELETON_POOL. With any other virtual memory, make sure that there is enough physical memory to support this increase.


After this fix, the CPU looks much better but we are still seeing some runaway transactions or changed accesspath (which are not good in many cases). We're fixing them by rewriting the queries in most cases.

SQLCODE -331

Recently I altered a tablespace CCSID from EBCDIC to UNICODE. This tablespace had a PLAN_TABLE in it. After altering the tablespace, I couldn't retrieve certain rows from this PLAN_TABLE. I started getting SQLCODE -331. The message appeared like this:

"CHARACTER CONVERSION CANNOT BE PERFORMED BECAUSE A STRING, POSITION43 CANNNOT BE CONVERTED FROM 367 TO 37, REASON 12"

Position 43 refers to column IBM_REQUIRED_DATA in the PLAN_TABLE. As a short term solution, I spaced out the value in this column and it fixed it. However, it kept appearing for different programs/queries. So, I altered the table column attribute for IBM_REQUIRED_DATA from "FOR SBCS DATA" to "FOR BIT DATA". It fixed the problem.

SQLCODE -518

The reason mentioned in the DB2 manual for SQLCODE -518 is "THE EXECUTE STATEMENT DOES NOT IDENTIFY A VALID PREPARED STATEMENT"

You may get this SQLCODE for many reasons. I don't know them all. For me this is what happened. When I tried to connect to a remote DB2 database (on LUW) from a COBOL program on the mainframe, the CONNECT was successful. However when I tried to run an SQL on that database I got this SQLCODE.

After researching for a while, I found that I had an existing SQL statement within EXEC SQL and END-EXEC. That worked good. But when I commented out one of WHERE clause statements (using * at column 7), it gave me this error. I moved the commented statement to the line after END-EXEC and it started working again.

Bottom line is DB2 doesn't like COBOL-like commented code inside EXEC SQL and END-EXEC when you are trying to run the SQL at a remote DB2 database.

SQLCODE -804

SQLCODE -804 means something messed up with the SQLDA. Since SQLDA is internally created by DB2 for static COBOL programs, it is possible for a COBOL subscript to overwrite SQLDA statements which are inserted by DB2 at the end of the Working Storage section.

If your programmer complains about this sqlcode, have him/her increase the table size (increase the 'TIMES' parameter value) and recompile the program.

UNIQUE WHERE NOT NULL - Explained

I always had confusion over the meaning of indexes created with UNIQUE WHERE NOT NULL clause on DB2 for z/OS. The manuals don't seem to do a good job explaining it. I understand it now and here is how it works.

It means its ok to have multiple rows with null values in columns included in the index, but its not ok to have rows where the combination of the columns included in the index is not unique and all the columns are not null. I tested this on a three column index. I was able to insert rows rows shown below just fine, the only duplicate exception I got was where column a, b, c were the same value for two different rows and none of the 3 columns contained a null value.

  COL1  COL2  COL3
  ----  ----  ----
   A     B     C
   A     B     -
   A     B     -
   A     -     -
   A     -     -
   -     -     -
   -     -     -

CREATE UNIQUE WHERE NOT NULL INDEX PRE.IDX1
       ON PRE.TEST
       (COL1   ASC
       ,COL2   ASC
       ,COL3   ASC)
       USING STOGROUP ASTERISK
             PRIQTY 12
             SECQTY 10800
             ERASE NO
       FREEPAGE 0
       PCTFREE 0
       BUFFERPOOL BP1
       CLOSE YES
       PIECESIZE 2097152  K;

CREATE TABLE PRE.TEST
       (COL1     CHARACTER(1)   FOR SBCS DATA WITH DEFAULT NULL
       ,COL2     CHARACTER(1)   FOR SBCS DATA WITH DEFAULT NULL
       ,COL3     CHARACTER(1)   FOR SBCS DATA WITH DEFAULT NULL
       )
       IN DB.TS;

Copy Data from DB2 for z/OS to DB2 for LUW (UDB)

One of the applications developer in our shop wanted to know if he can copy data from a mainframe DB2 table to DB2 UDB on LUW. Though there are several ways of doing it, this is the one suggested him assuming the table structure in z/OS and LUW are the same.

1. Some of the tools that can be used to achieve this: Quest Central for DB2, Toad for DB2 or DB2CMD interface

2. Connect to the mainframe database (CONNECT TO ssid) and issue the EXPORT command "EXPORT TO d:\file_name.ixf OF IXF MESSAGES d:\msg_export.txt SELECT *FROM prefix.table_name"
Note: Don't forget to check the message file before moving onto the next step

3. Connect to the LUW database and issue the IMPORT command
"IMPORT FROM d:\file_name.ixf OF IXF MESSAGES d:\msg_import.txt INSERT INTO prefix.table_name"
Note: Don't forget to check the message file

It's that simple. But remember that this works well for relatively smaller tables. For larger tables, you may have to use the LOAD command with proper COMMIT COUNT.

DB2 CREATE INDEX abended with S04E 00E70005

There is a couple of news items attached to this post:
1. Today is the first anniversary of this blog
2. This is the 50th post

Well, now into the actual blog item ...

My colleague was trying to drop an existing UNIQUE & CLUSTER index and recreate it after adding 3 new columns at the end of the table, with one of the new columns added to the index. However, the DDL failed with S04E reason code 00E70005.

DB2 messages and codes manual had this info on this reason code: "A relational data system (RDS) subcomponent internal inconsistency was detected". It didn't make any sense. We suspected data issue first, so we REORGed the tablespace, and tried to create the index, but that didn't fix the problem. Next, we tried to repair the DBD using this JCL:

//*===============================================
//* Start the database in UT mode
//*===============================================
//STEP0010   EXEC PGM=IKJEFT01,DYNAMNBR=20
//STEPLIB    DD DSN=SYS2.DB2x.DSNLOAD,DISP=SHR
//SYSTSPRT   DD SYSOUT=*
//SYSPRINT   DD SYSOUT=*
//LISTING    DD SYSOUT=*
//SYSUDUMP   DD DUMMY
//SYSTSIN    DD *
  DSN SYSTEM(DB2x)
  -START DB(database) ACCESS(UT)
  END
/*
//*===============================================
//* Run REPAIR utility on the DBD
//*===============================================
//STEP020 EXEC PGM=DSNUTILB,PARM='DB2x,utilid'
//SYSPRINT DD SYSOUT=*
//SYSUDUMP DD SYSOUT=*
//SYSIN    DD  *
  REPAIR DBD REBUILD DATABASE database
/*
//*===============================================
//* Start the database in RW mode
//*===============================================
//STEP0030   EXEC PGM=IKJEFT01,DYNAMNBR=20
//STEPLIB    DD DSN=SYS2.DB2.DB2x.DSNLOAD,DISP=SHR
//SYSTSPRT   DD SYSOUT=*
//SYSPRINT   DD SYSOUT=*
//LISTING    DD SYSOUT=*
//SYSUDUMP   DD DUMMY
//SYSTSIN    DD *
  DSN SYSTEM(DB2x)
  -START DB(database) ACCESS(RW)
  END
/*
 

Even that didn't fix the problem. We were not very keen to open a ticket with IBM, since we wanted to fix the problem fast rather than debugging it. So as a last resort we dropped and recreated the tablespace (off course, we had saved the data). That fixed the problem. We don't know what caused it, but the problem is fixed !! I really hate when I don't know what caused the problem, and the fix is based trial-and-error method.

"DB2LOOK" to extract DDL from DB2 for z/OS

Many of you may be aware that "db2look" command can be used to extract DDL from DB2 UDB for LUW database. But do you know that you can use this command to extract DDL from a DB2 database for z/OS? Here is how you do it.

1. Bring up a db2cmd prompt (press Start button on your PC, click Run, type db2cmd and hit enter)
2. Connect to your mainframe DB2 subsystem using the command "db2 connectto SSID user USERID". It'll prompt for the password. Enter the password and hit enter
3. Issue the following command:
db2look -d SSID -i USERID -w PWD -e -a -l -x -p -o FILENAME.TXT

The options specified extracts the following info:

-e ==> Extract DDL statements for database objects. DDL for the followingdatabase objects are extracted when using the -e option: Tables, Views, Automatic summary tables (AST), Aliases, Indexes, Triggers, Sequences, User-defined distinct types, Primary key, referential integrity, and check constraints, User-defined structured types, User-defined functions, User-defined methods, User-defined transforms, Wrappers, Servers, User mappings, Nicknames, Type mappings, Function templates, Function mappings, Index specifications, Stored procedures

-a ==> When this option is specified the output is not limited to the objects created under a particular creator ID. All objects created by all users are considered. For example, if this option is specified with the -e option, DDL statements are extracted for all objects in the database. If this option is specified with the -m option, UPDATE statistics statements are extracted for all user created tables and indexes in the database.
Note: If neither -u nor -a is specified, the environment variable USER isused.

-l ==> If this option is specified, then the db2look utility will generate DDL for user defined table spaces, database partition groups and bufferpools.

-x ==> If this option is specified, the db2look utility will generate authorization DDL (GRANT statement, for example).

More info is available in the DB2 V8 LUW Command Reference: http://publib.boulder.ibm.com/infocenter/db2luw/v8/index.jsptopic=/com.ibm.db2.udb.doc/core/r0002051.htm

SYSIBM.SQT* tables and SQLCODE -526

This is going to be a small post ...

If you plan to use the IBM Data Server Driver for JDBC and SQLJ or the DB2 ODBC driver, you must create the SYSIBM.SQT* Tables & SYSIBM.SQL* Stored Procs that provide support for those drivers.

GRANT ALL access to PUBLIC to these tables. Trying to GRANT individual access like GRANT SELECT won't work (you'll get SQLCODE -526).

SQLCODE -904 with Reason Code "00D70024"

When you try to access or drop a table, you may get this error if the underlying VSAM data set for the table does not exist. If you are dropping the table, try dropping the tablespace instead (if there is just one table in the tablespace). If you are trying to browse the data, you are out of luck. Try recreating the table from another subsystem and load the data from there. Or, do a LOAD REPLACE with DUMMY SYSREC (it will create the underlying datasets). This will allow you access the table, but the table will be empty.

NULL VALUES and NULL INDICATORS in DB2

I was looking for storage space required for a column defined as WITH NULL. Got the answer here:

http://mainframe-faqs.blogspot.com/2008/02/null-values-and-null-indicators-in-db2.html

Excellent information here. Just wanted to share it with my readers here.

Thanks Arunam !!

Disclaimer:
THIS IS NOT MY OWN BLOG ENTRY. I'm just linking it to Arunam's blog entry.

DB2 crash with S04F (DSNJ113E)

This happened a couple of days ago. One of our CICS transactions inserted more than 18 million rows (probably loop) in a DB2 table before someone noticed and cancelled it. DB2 started rolling back the data but was slow, so the DBAs tried to issue the CANCEL THREAD command with NOBACKOUT option (this was in a TEST region) but that didn't work. So they just let it roll back. When there was less than 2 million rows to be rolled back DB2 crashed with the following message (S04F):

DSNJ113E +DB2T DSNJR003 RBA 'C35EF8BC2000' NOT IN
ANY ACTIVE OR ARCHIVE LOG DATA SET. CONNECTION-ID=DB2T,
CORRELATION-ID=003.RCRSC 02, MEMBER-ID=0

The operators restarted DB2, but it went down again after some time. DB2 manual suggested that the log record might be missing from the active/archive log. DBAs ran DSNJU004 utility and found that this RBA was available in a archive log data set which was not in the BSDS. DB2 manual suggested to add the dataset to the BSDS using DSNJU003 utility. But before doing that, the DBAs stopped DB2 and ran DSNJU003 using the parameters

CRESTART CREATE,FORWARD=YES,BACKOUT=NO

to reset checkpoint. Then they stopped DB2 with MODE(FORCE) since it didn't stop after issuing a regular stop DB2 command. But that failed. So they issued the following command to stop IRLM:

F DB2TIRLM,ABEND,NODUMP

After stopping DB2, they ran the DSNJU003 utility again and then started DB2.

Most of these are possible because it was a TEST system. Not sure what would we have done if this had happened in a PROD system.

DB2 "FIRST_DAY" function

DB2 comes with a lot of in-built functions. One of them is called LAST_DAY which returns the Last Day of a month, but there is no FIRST_DAY of the month function. Some of my colleagues asked about it. Here is an UDF (User Defined Function) that'll return FIRST DAY of the month:

CREATE FUNCTION schema.FIRST_DAY (IN_DATE DATE)
RETURNS DATE
LANGUAGE SQL
CONTAINS SQL
NO EXTERNAL ACTION
DETERMINISTIC
RETURN DATE(LAST_DAY(IN_DATE - 1 MONTH) + 1 DAY);

SQL ERROR "00D31034"

You have a web application connecting to DB2 for z/OS and it times out with the message:

SQL30081N A communication error has been detected. 
Communication protocol being used: "TCP/IP". Communication
 API being used: "SOCKETS". Location where the error was
detected: ip-address", Communication function detecting
the error: "recv". Protocol specific error code(s):
 "*", "10054", "*". SQLSTATE=08001

There could be many reasons for this error. One that I found recently is here.

Look in your z/OS DB2 MSTR log for a message like this:

DSNL030I  +DB2x DSNLILNR DDF PROCESSING FAILURE FOR
LUWID=xxxxxxxx.GE21.000000000000
AUTHID=-N/A-, REASON=00D31034

As per the DB2 manual, "00D31034" means

The MAX REMOTE CONNECTED limit was reached and the
conversation was abnormally deallocated. The thread
was not allowed to connect to DB2.

Now, as the message says, there may be too many remote database connections and the number of remote connections exceeded the ZPARM CONDBAT(MAX REMOTE CONNECTED) limit.

But, before jumping to the conclusion that all the connections are genuine and asking your DBA to bump this number, look at your connection threads closely.

The web application might be using an old version of the arch.jar (arch-2_2_3.jar). This version of arch.jar does not close the database connection successfully on a few occasions.

The resolution is a simple upgrade to the latest arch.jar (arch-3_0_1.jar) which has the fix to close the connection.

DB2 SQLCODE -741

I was installing a DB2 vendor tool yesterday. Part of the install was to create a new temporary database. When I ran the following DDL,

CREATE DATABASE database
              STOGROUP ASTERISK
              BUFFERPOOL BP2
              INDEXBP BP2
              AS TEMP

I got this message:

DSNT408I SQLCODE = -741, ERROR:  A TEMP DATABASE IS ALREADY DEFINED FOR
       MEMBER ssid

The problem here is, I was trying to define a temporary database whereas a temporary database already exists for that subsystem. ONLY ONE TEMPORARY DATABASE is allowed per subsystem or data sharing member.

I learnt a lesson !!

BMC Unload PLUS - BMC50033E

We're moving away from CA/Platinum, yes, our shop is converting from CA/Platinum tools to BMC tools for DB2.

I ran my first UNLOAD PLUS job today and it went down with this message:

BMC50033E CPU NOT AUTHORIZED TO RUN UNLOAD PLUS FOR DB2 RETURN CODE = 0004 REASON CODE = P1

Found out that, in the JCL, I need to include the Password dataset - either in STEPLIB or JOBLIB:

// DD DISP=SHR,DSN=hlq.BMCPSWD

DB2 Subsystem Size

How do you find the size of your entire DB2 subsystem on z/OS?

Try this ...

List the base datasets in ISPF option 3.4, e.g., DB2CAT*.DSNDBD.**, and then do a SAVE LIST. It'll save all the Tracks Used info. Add them up to come up with the total tracks used. Remember, for compressed tablespaces, this information shows size for the compressed data. Uncompressed data might occupy more space.

If you want to know the relation between tracks and Kilo Bytes, use this formula (this example is for a 3390 DASD - your model may be different, check with your systems programmer):
1 Track = 48 KB
1 Cylinder = 720 KB
1 Cylinder = 15 Tracks