Some DataStage Job design

Updating the Master Reports List (MRL) in Datastage

Create a Job which has a Row Generator stage linking to an ODBC or DB2 stage. Change the parameter in the Row Generator stage to output zero records. Write the SQL to update the MRL table as 'After SQL' in the output ODBC or DB2 stage.

Surrogate Key Generator - Generate Surrogate Key for Data

In this post, We will see how to generate surrogate key for data, where we have to use surrogate key stage.

A) Design :  
Below design is a demo design of job. Here our data source is a row generator which is generating rows. In real time scenario, Source can be a flat file, DB stages, Passive Stage or can be a Active stage also.
In Row Generator Stage, we are generating a col "Name".

Surrogate Key Generator - Create/Update State File

In last post ( Surrogate Key Generator - Create State File ) we have seen how to generate the surrogate key file, In this post we will discuss about what are the other ways to generate it and how to update state file

Update Surrogate State File (Flat File) :-
To update the state file, add a Surrogate Key Generator stage to a job with a single input link from another stage.

 

Dummy Data Generation using Row Generator in DataStage - 2

By default the Row Generator stage runs sequentially, generating data in a single partition. You can, however, configure it to run in parallel and meaningful data.

We are using the same job design as in Dummy Data Generation using Row Generator in DataStage - 1


a) Job Design :

Dummy Data Generation using Row Generator in DataStage - 1

How to Generate Input Data for your dummy jobs n practice ??

In DataStage, There is a Stage called "Row Generator" under "Devlopment/Debug Stages" category. For Generating the dummy data,  we will use this stage.

So, We are going to create a job which generates dummy data.

Development/Debug Stage in DataStage

Head stage

The Head Stage is a Development/Debug stage. It can have a single input link and a single output link.

It is one of a number of stages that InfoSphere DataStage provides to help you sample data

The Head Stage selects the first N rows from each partition of an input data set and copies the selected

rows to an output data set. You determine which rows are copied by setting properties which allow you

to specify:

·         The number of rows to copy

·         The partition from which the rows are copied

·         The location of the rows to copy

·         The number of rows to skip before the copying operation begins

This stage is helpful in testing and debugging applications with large data sets. For example, the Partition

property lets you see data from a single partition to determine if the data is being partitioned as you

want it to be. The Skip property lets you access a certain portion of a data set.