Performance Tunings in DataStage

JOB LEVEL

• Parametrize all the inputs needed for the jobs; avoid hard coding of parameters like username, password, directory paths etc.
• By using the environment variable ‘APT_CONFIG_FILE’ as a job parameter, user can dynamically change the number of nodes being used for processing a particular job.
• For reading/writing data from large tables/files, make use of the environment variable ‘APT_BUFFER_MAXIMUM_MEMORY’. It can be used to change the memory buffer size being used for each stage.
• It is recommended to set the environment variable $APT_DUMP_SCORE to a value of 1. When this environment variable is set, an entry is placed in the WebSphere DataStage job log showing the actual runtime structure (processes, their associated internal operators, datasets, nodes, etc) used to execute the job flow.

14 design tips for better performance in Datastage

1) Avoid unnecessary type conversions: set the OSH_PRINT_SCHEMAS environment variable to verify that run time schemas match the job design column definitions. If you are using stage variables on a Transformer stage, ensure that their data types match the expected result types.

2) Use Transformer stages sparingly and wisely. Transformer stages can slow down your job. Do not have multiple stages where the functionality could be incorporated into a single stage, and use other stage types to perform simple transformation operations

Converting .rpm Packages To Debian/Ubuntu .deb Format With Alien

This article shows how you can convert .rpm packages to .deb packages with a tool called alien so that you can easily install them on Debian and Ubuntu systems. Sometimes this is quite convenient as not all software projects release their software as Debian/Ubuntu packages.

However, you should keep in mind what the alien man page says:

Convert a single row into multiple rows ( horizontally pivoting ) with Pivot stage ?

In this example, the Pivot Enterprise stage is set up to horizontally pivot some data.

You can generate a pivot index that will assign an index number to each row within sets of horizontally pivoted data. The following tables provide examples of data before and after a horizontal pivot operation.


Input Data
REPID,last_name,Jan_sales,Feb_sales,Mar_sales
100,Smith,1234.08,1456.80,1578.00
101,Yamada,1245.20,1765.00,1934.22
102,Xing,2190.89,1287.98,2054.55
103,Anderson,1498.09,1287.23,3298.76