We have moved to www.dataGenX.net, Keep Learning with us.
Showing posts with label counter. Show all posts
Showing posts with label counter. Show all posts

Saturday, July 05, 2014

Count Rows and Calculate Sum in same Aggregator


Generally, we are using Fork n Join method ( divide the data into 2 links which goes to 2 Aggregator ) to calculate Sum and Count Rows of Data because Aggregator Stage is providing one type of aggregation at a time, You can Count the rows or can do some other aggregation.
            Here, I have describe a way in which we can Count Rows and Calculate Sum in a Single Aggregator Stage with some limitations :-)


Design :

As per design, we used Column Generator and Aggregator Stage to calculate and count.

Wednesday, April 30, 2014

Performance Tunings in DataStage


JOB LEVEL

  • Parametrize all the inputs needed for the jobs; avoid hard coding of parameters like username, password, directory paths etc.
  • By using the environment variable ‘APT_CONFIG_FILE’ as a job parameter, user can dynamically change the number of nodes being used for processing a particular job.
  • For reading/writing data from large tables/files, make use of the environment variable ‘APT_BUFFER_MAXIMUM_MEMORY’. It can be used to change the memory buffer size being used for each stage.
  • It is recommended to set the environment variable $APT_DUMP_SCORE to a value of 1. When this environment variable is set, an entry is placed in the WebSphere DataStage job log showing the actual runtime structure (processes, their associated internal operators, datasets, nodes, etc) used to execute the job flow.

Monday, March 10, 2014

DataStage Scenario - Problem17




Goal : Count the data in each column


Input :
col1 col2 col3
a {NULL} b
f k {NULL}
h {NULL} n
i d {NULL}
{NULL} s {NULL}
g u m
l x o
m {NULL} {NULL}
c d z

Wednesday, December 11, 2013

DataStage Scenario - Problem6


Goal : Get the count of Vowels in Columns

Input :

Akash Aggrawal
Priya Awasthi  
Anil chahal    
Diya Singh    
Kashish Patel 
Sunil Verma    
Rashid Patel    
Rashmi Arya   
Gopal Joshi     
Neha Tomar    

Monday, December 09, 2013

Monday, October 07, 2013

Create a unique counter in datastage


This entry describes various ways of creating a unique counter in DataStage jobs.
A parallel job has a surrogate key stage that creates unique IDs, however it is limited in that it does not support conditional code and it may be more efficient to add a counter to an existing transformer rather than add a new stage.

In a server job there are a set of key increment routines installed in the routine SDK samples that offer a more complex counter that remembers values between job executions.
The following section outlines a transformer only technique.