Things need to consider while developing a Datastage job

Datasets
Datasets are the best when storing the results intermediately. Datasets will keep the partitions and sort order if set. This will save re-partitioning, sorting and would make the job more robust.

Performance of the job can be improved if:
1) Unnecessary column are removed from the up and down stream links.
2) Removing these unnecessary columns will help reducing the memory consumption.
3) Always specify the list of columns in the select statement when reading from database. This will not bring unnecessary column data in the job which will save memory and network consumption.
4) Use RCP very carefully.
5) Understand the data-type before using them in the job. Do the data profiling before bringing data in the job.

Data Warehouse Testing Checklist

Unit testing checklist

 A checklist will aid database programmers to systematically test their code before formal QA testing.

• Check the mapping of fields that support data staging and in data marts.
• Check for duplication of values generated using sequence generators.
• Check the correctness of surrogate keys that uniquely identify rows of data.
• Check for data-type constraints of the fields present in staging and core levels.
• Check the data loading status and error messages after ETLs (extracts, transformations, loads).
• Look for string columns that are incorrectly left or right trimmed.
• Make sure all tables and specified fields were loaded from source to staging.
• Verify that not-null fields were populated.
• Verify that no data truncation occurred in each field.
• Make sure data types and formats are as specified during database design.
• Make sure there are no duplicate records in target tables.
• Make sure data transformations are correctly based on business rules.
• Verify that numeric fields are populated precisely.
• Make sure every ETL session completed with only planned exceptions.
• Verify all data cleansing, transformation, and error and exception handling.
• Verify stored procedure calculations and data mappings.

Performance Tunings in DataStage

JOB LEVEL

• Parametrize all the inputs needed for the jobs; avoid hard coding of parameters like username, password, directory paths etc.
• By using the environment variable ‘APT_CONFIG_FILE’ as a job parameter, user can dynamically change the number of nodes being used for processing a particular job.
• For reading/writing data from large tables/files, make use of the environment variable ‘APT_BUFFER_MAXIMUM_MEMORY’. It can be used to change the memory buffer size being used for each stage.
• It is recommended to set the environment variable $APT_DUMP_SCORE to a value of 1. When this environment variable is set, an entry is placed in the WebSphere DataStage job log showing the actual runtime structure (processes, their associated internal operators, datasets, nodes, etc) used to execute the job flow.

Datastage Coding Checklist

1. Ensure that the null handling properties are taken care for all the nullable fields. Do not set the null field value to some value which may be present in the source.
2. Ensure that all the character fields are trimmed before any processing. Normally extra spaces in the data may lead to some errors like lookup mismatch which are hard to detect.
3. Always save the metadata (for source, target or lookup definitions) in the repository to ensure re usability and consistency.

ETL Job Design Standards - 2

Part 1 --> ETL Job Design Standards - 1

Parameter Management Standards

This section defines standards to manage job parameters across environments. Jobs should use parameters liberally to avoid hard coding as much as possible. Some categories of parameters include: 

• Environmental parameters, such as directory names, file names, etc.
• Database connection parameters
• Notification email addresses
• Processing options, such as degree of parallelism

DataStage Server Hang Issues & Resolution

Server hang issue can occurred when

1) Metadata repository database detects a deadlock condition and choose failing job as the victim of the deadlock.
2) Log maintenance is ignored.
3) Temp folders are not maintained periodically.

I will try to explain above three points in detail below:

Why Entire partition is used in LOOKUP stage ?

Entire partition has all data across the nodes So while matching(in lookup) the records all data should be present across all nodes.

How can I improve performance during an import into Datastage?

One way to improve performance of an import is to exclude the executables.
Here are three different ways to exclude executables from the import. 

• Exclude the executables from the export.
  In DataStage Designer export using the option export Jobs without executables.
  

• Create an xml file instead of a dsx file.
  Executables are not included in xml files.
  (The xml option is available with dscmdexport but only with /JOB option)
  

What can you delete to free up disk space in IBM InfoSphere Information Server

What can you delete to free up disk space in IBM InfoSphere Information Server when disks are becoming full?

What to do ????

Here are some things you can do to clean up space:

• Clear the &PH& file in the Project directory. There is a &PH& directory in each DataStage project directory, which contains information about active stages that is used for diagnostic purposes. The &PH& directory is added to every time a job is run, and needs periodic cleaning out.
  

14 design tips for better performance in Datastage

1) Avoid unnecessary type conversions: set the OSH_PRINT_SCHEMAS environment variable to verify that run time schemas match the job design column definitions. If you are using stage variables on a Transformer stage, ensure that their data types match the expected result types.

2) Use Transformer stages sparingly and wisely. Transformer stages can slow down your job. Do not have multiple stages where the functionality could be incorporated into a single stage, and use other stage types to perform simple transformation operations

DataStage Jobs hang or have slow performance when temp directories have many files

Datastage jobs write multiple files to temporary directories which are not automatically cleaned up. When the number of files grows large, DataStage jobs will have slower performance or can appear to hang.

DataStage jobs write multiple files to temporary directories which are not automatically cleaned up. When the number of files grows large, DataStage jobs will have slower performance or can appear to hang. For sites who have been running DataStage for a year or more and have not previously performed cleanup of these directories, they could contain 100,000 or more files.

Perform periodic cleanup of old DataStage files in these directories to prevent performance problems relating to the number of files in the directories.

datastage4you QAnT - April2013 - 2

Click here ---> datastage4you QAnT - April2013 - 1

#14. DataStage Server – starting the server
a) DB2
su - db2inst1 -c "db2start“
b) WAS Server
/opt/IBM/WebSphere/AppServer/bin/startServer.sh server1 -user wasadmin -password PASSWORD
c) NODE
/opt/IBM/InformationServer/ASBNode/bin/NodeAgents.sh start
d) Engine
/opt/IBM/InformationServer/Server/DSEngine/bin/uv -admin -start

Optimzie your DataStage Job Performance with relevant Environment Variables

DataStage has many parameters which can be tweaked and used to optimize the performance of various DataStage Jobs. Even many available to collect more information during the event of crash to get more traces.

For any DataStage Job if you run into problem or want to get more details need to check following variables.

More Warnings causes Low Performance in DataStage. WHY ???

 
Warning messages are bad in any programming language; more so, in the case of DataStage. 

This is because DataStage is typically used for large scale processing of data. Every time a warning message is encountered, the engine code has to encounter an exception handler, throw the warning message and resume its task. 

14 Good design tips in Datastage

1) When you need to run the same sequence of jobs again and again, better create a sequencer with all the jobs that you need to run. Running this sequencer will run all the jobs. You can provide the sequence as per your requirement.

2) If you are using a copy or a filter stage either immediately after or immediately before a transformer stage, you are reducing the efficiency by using more stages because a transformer does the job of both copy stage as well as a filter stage

Speed up your Lotus Notes performance by tuning the JVM

Lotus Notes uses Java Virtual Machine (JVM) for processing of Lotus Notes applications and plugins. 

IBMs Lotues Notes client configuration has a standard value that defines how many megabytes of RAM Lotus Notes can maximally use for JVM. This value is for PCs with above 1 GB RAM initially set too low, as it should be approx. 1/3 of the RAM of the PC (default value = 256 MB, but should be 768 MB).