DataStage Warning - dsjob Command Returns Status code = 80011

Running dsjob utility with or without -domain option results in an error message: Status Code = 80011

Resolution -

The dsjob utility is unable to authenticate and validate against the DataStage Engine.

Use -domain NONE option.

DataStage Warning - When deleting data set previous delete attempt not complete

main_program: When deleting data set /full_Path_of_dataset.ds, previous delete attempt not complete; removing /full_Path_of_dataset.ds.being_deleted in order to proceed.


Solution -

This error usually comes when a previous run of respective job got aborted and the auto cleanup process hasn't been completed properly.

DataStage Error - irreconcilable constraints on the number of partitions

Fatal Error: There are irreconcilable constraints on the number of partitions of an operator: parallel <stage name>. The number of partitions is already constrained to 1, but an eSame partitioned input virtual dataset produced by parallel {natural=MOD_Nulls, synthetic="modify(0)"} has 2. 


This Error is simply Data Partitioning Error, The Stage mentioned in Error is not getting the Partition as defined for it. This is usually happen in SAME partitioning.

DataStage job fails with code=-14 [Timed out while waiting for an event]

DataStage job fails with code=-14 [Timed out while waiting for an event] - this generally happens when jobs are called from sequencers 

 

Solution :

Follow steps 1-3 in resolving this error.
1) DSWaitStartup - environment variable -
When multiple instances of a job are run from a sequence, and one or more of the job instances are set to reset, the sequence might report a status=99 or -14. This can occur because the controlling sequence did not give the job instances enough time to reset before polling its status. The start up time for a job reset must be increased. The environment variable DSWaitResetStartup can be used for this purpose.

(The maximum value that can be set for DSWaitResetStartup is the value of DSWaitStartup (default is 60). For example, if a value of 120 is required for DSWaitResetStartup, then ensure that DSWaitStartup is also set to a minimum of 120.)

DataStage Warning : When checking operator: Operator of type "APT_TSortOperator": will partition despite the preserve-partitioning flag on the data set on input port 0

DataStage Warning : When checking operator: Operator of type "APT_TSortOperator": will partition despite the preserve-partitioning flag on the data set on input port 0.

These warnings are obtained when we are using some specific partitioning method in a stage in the job e.g hash/same etc but in the preceding stage , preserve portioning(also default) is set in the stage>advanced tab.

DataStage Warning - APT_CombinedOperatorController

While we are running DataStage job, we got this warning like..

APT_CombinedOperatorController(1),0:  XXXXXXXXXXXXXXXXXXXXXXX
APT_CombinedOperatorController(2),1:  XXXXXXXXXXXXXXXXXXXXXXX

We don't get the stage name which is causing this issue ? How we will get the Culprit Stage :-) ?


Solution :

DataStage Warning - Agg,1: Hash table has grown to 16384 entries

Sometime when we are running Datastage job in which we are using Aggregator stage, we get below warning -

Agg,1: Hash table has grown to 16384 entries.

Agg,0: Hash table has grown to 16384 entries. 


Solution :

Performance Tunings in DataStage

JOB LEVEL

• Parametrize all the inputs needed for the jobs; avoid hard coding of parameters like username, password, directory paths etc.
• By using the environment variable ‘APT_CONFIG_FILE’ as a job parameter, user can dynamically change the number of nodes being used for processing a particular job.
• For reading/writing data from large tables/files, make use of the environment variable ‘APT_BUFFER_MAXIMUM_MEMORY’. It can be used to change the memory buffer size being used for each stage.
• It is recommended to set the environment variable $APT_DUMP_SCORE to a value of 1. When this environment variable is set, an entry is placed in the WebSphere DataStage job log showing the actual runtime structure (processes, their associated internal operators, datasets, nodes, etc) used to execute the job flow.

Datastage Common Errors and Solutions

1.     While running ./NodeAgents.sh start command... getting the following error: “LoggingAgent.sh process stopped unexpectedly”

SOL:   needs to kill LoggingAgentSocketImpl

              Ps –ef |  grep  LoggingAgentSocketImpl   (OR)

              PS –ef |               grep Agent  (to check the process id of the above)

2.     Warning: A sequential operator cannot preserve the partitioning of input data set on input port 0

SOL:    Clear the preserve partition flag before Sequential file stages.

DataStage BASIC functions

These functions can be used in a job control routine, which is defined as part of a jobs properties and allows other jobs to be run and controlled from the first job. Some of the functions can also be used for getting status information on the current job; these are useful in active stage expressions and before- and after-stage subroutines. 


Specify the job you want to control
DSAttachJob

Optimzie your DataStage Job Performance with relevant Environment Variables

DataStage has many parameters which can be tweaked and used to optimize the performance of various DataStage Jobs. Even many available to collect more information during the event of crash to get more traces.

For any DataStage Job if you run into problem or want to get more details need to check following variables.

More Warnings causes Low Performance in DataStage. WHY ???

 
Warning messages are bad in any programming language; more so, in the case of DataStage. 

This is because DataStage is typically used for large scale processing of data. Every time a warning message is encountered, the engine code has to encounter an exception handler, throw the warning message and resume its task. 

Tips & Tricks for debugging a DataStage job

The article talks about DataStage debugging techniques. This can be applied to a job which

• is not producing proper output data or
• to a job that is aborting or generating warnings

1. Use the Data Set Management utility, which is available in the Tools menu of the DataStage Designer or the DataStage Manager, to examine the schema, look at row counts, and delete a Parallel Data Set. You can also view the data itself.