datastage4you QAnT - April2013 - 1

 #1. Use the Data Set Management utility, which is available in the Tools menu of the DataStage Designer or the DataStage Manager, to examine the schema, look at row counts, and delete a Parallel Data Set. You can also view the data itself.

#2. Use Sort stages instead of Remove duplicate stages. Sort stage has got more grouping options and sort indicator options.

#3. for quick checking if DS job is running on Server or not from UNIX
ps -ef | grep 'DSD.RUN'

Optimzie your DataStage Job Performance with relevant Environment Variables

DataStage has many parameters which can be tweaked and used to optimize the performance of various DataStage Jobs. Even many available to collect more information during the event of crash to get more traces.

For any DataStage Job if you run into problem or want to get more details need to check following variables.

Local and Shared Containers

As the name suggest local container (LC) contains the stages locally in the job and shared container (SC) contains stage in the job as well as allow its usage to other jobs. Differences or qualities between these two types of containers can be listed as follows:

 
Local Containers

1) Mainly used to ease the job design.
2) Piece of logic can be combined together and used inside a job.
3) Can only be used in the job after its creation.
4) Number of input and output links supported by local containers.
5) Availability throughout the job.

Useful Perl Scripts : Part-4

For more please visit :  Perl Series

 Array referencing

 

 

Read/Write/Load Message with MQ : Part-1

 Previous Post :            What is MQ Stage ???

Stage MQ: The 3 parameters are: Queue manager, User and Password.

What is MQ Stage ???

 

The WebSphere MQ stage is a passive stage that offers a message based solution to customers where messaging represents another form of source and target data. The WebSphere MQ stage lets WebSphere DataStage read from and write to WebSphere MQ message queues. You can use this stage as:

• An intermediary between applications, transforming messages as they are sent between programs
• A conduit for the transmission of legacy data to a message queue
• A message queue reader for transmission to a non-messaging target

More Warnings causes Low Performance in DataStage. WHY ???

 
Warning messages are bad in any programming language; more so, in the case of DataStage. 

This is because DataStage is typically used for large scale processing of data. Every time a warning message is encountered, the engine code has to encounter an exception handler, throw the warning message and resume its task. 

DS Director Error (Unable to open Specific folder)

Reason -

Unable to open a specific folder in the director, it prompts the following error message. when I say, 'OK' , Director gets close. But there is no problem with the other folders. The Error message is "Run-time error '457'; This key is already associated with an element of this collection"

Start/Stop/Delete a Queue Manager

Starting a Queue Manager

Before we can use a Queue Manager, we need to start it, using the STRMQM command. The command to start a Queue Manager called QMA is:

$ strmqm QMA

You should see output similar to the following on your screen:

WebSphere MQ queue manager 'QMA' starting.

2108 log records accessed on queue manager 'QMA' during the log replay phase.

Log replay for queue manager 'QMA' complete.

Transaction manager state recovered for queue manager 'QMA'.

WebSphere MQ queue manager 'QMA' started.

Colourizing Unix Programs & Shell Scripts

 

In an ANSI compatible terminal (like xterm, rxvt, konsole ...) text may be shown in colors different from black/white. Here first the colour coding is shown for shell programming .

Each color code is present after an escape sequence. This can be given by

echo  –e "\033[1m Demo Test \033[0m\n";

Something about DataStage Phantom(&PH& - Universe DB file) directory

&PH& directory which is a UniData/UniVerse/U2 file structure in the DataStage contain the files which get created during a job's execution. 
The two letters in the name of the directory hint to the word "phantom" which is the naming convention in DataStage for background processes. Every time a job runs, it generates these logs which contain the status of the different phantom processes generated during the run. 
There are two types of the files that can be seen in &PH& directory one which starts from DSD.RUN and another one which starts from DSD.OshMonitor. 

DataStage client receives error when compile jobs: Cannot get exclusive access to executable file for job sampleJob - job may be being monitored

Problem :

DataStage user sees the following error in Designer client when attempting to compile a specific job:
Cannot get exclusive access to executable file for job sampleJob - job may be being monitored.

Similar problems may occur in other releases.

Create/start/stop a Queue Manager

To create a Queue Manager, our userID needs to be part of the mqm group. If we do not have the appropriate authority, then we need to ask our MQ administrator to issue this command.

The command to create a Queue Manager is CRTMQM and there are various parameters available with this command:

crtmqm [-z] [-q] [-c Text] [-d DefXmitQ] [-h MaxHandles]

[-g ApplicationGroup] [-t TrigInt]

[-u DeadQ] [-x MaxUMsgs] [-lp LogPri] [-ls LogSec]

[-lc | -ll] [-lf LogFileSize] [-ld LogPath] QMgrName