Datastage Common Errors and Solutions

1.     While running ./NodeAgents.sh start command... getting the following error: “LoggingAgent.sh process stopped unexpectedly”

SOL:   needs to kill LoggingAgentSocketImpl

              Ps –ef |  grep  LoggingAgentSocketImpl   (OR)

              PS –ef |               grep Agent  (to check the process id of the above)

2.     Warning: A sequential operator cannot preserve the partitioning of input data set on input port 0

SOL:    Clear the preserve partition flag before Sequential file stages.

Buffering in DataStage

Buffering is a technique used in the Datastage jobs to ensure a constant and uninterrupted flow of data to and from stages in such a way that there is no potential dead lock or any fork join problems. It is been implemented in Datastage keeping in mind the fact that the data has to keep moving in the process with an optimized use of the memory in the server. As mentioned by IBM the ideal scenario is when the data flows through the stages without being written on the disk. As in the case of buffering in any system, the upstream operators should to wait for the downstream operators to consume their input before starting to create their records. This is the intention in Datastage too.

14 design tips for better performance in Datastage

1) Avoid unnecessary type conversions: set the OSH_PRINT_SCHEMAS environment variable to verify that run time schemas match the job design column definitions. If you are using stage variables on a Transformer stage, ensure that their data types match the expected result types.

2) Use Transformer stages sparingly and wisely. Transformer stages can slow down your job. Do not have multiple stages where the functionality could be incorporated into a single stage, and use other stage types to perform simple transformation operations

Stage to OPerator in DataStage - the Real Driver

The parallel job stages are built on operators. Or Operators are individual parallel engine stages where the data actually flows. In a typical job flow, operators are end-points, and data sets are the links between the operators.

Each operator listed in the DUMP SCORE is a number of processes that are dependent on:

• the job's established configuration file (APT_CONFIG_FILE)
• constrained by the node pool settings
• the operator configuration in the parallel engine code
• Several environment variables, such as APT_DISABLE_COMBINATION, being set/unset.