We have moved to www.dataGenX.net, Keep Learning with us.

Tuesday, June 30, 2015

Maximum number of DataStage Jobs in a project


Filesystems can have a hard limit on number of subdirectories that can be contained in a directory, for example 32766 (32K -2) for Linux

For every job, DataStage will create hash files as subdirectories in the project directory. Here is a list of the subdirectories created for every job. In this list "nn" represents the number (or id) of the job:

DS_TEMPnn
RT_CONFIGnn
RT_STATUSnn
RT_LOGnn
RT_BPnn
RT_BPnn.O
RT_SCnn (only for parallel jobs)
The folders have been created for jobs -- 
Parallel Job - 07
Server Job - 06


Our assumptions are below  --

Total sub directories can be created is - 32766
Project threshold - 80   
No of Parallel job can be creates is -   32766/7 - 80  ~ 4600
No of Server job can be creates is -   32766/6 - 80  ~ 5380

where 80 is for overhead of project default directory. In practice, the number might be smaller due to subdirectories not counted above, such as directories created by QualityStage jobs, some project folders etc.

This is just approximated calculation, variables can vary system to system.



Like the Facebook Page & join Group
https://www.facebook.com/DataStage4you
https://www.facebook.com/groups/DataStage4you

https://twitter.com/datastage4you
https://groups.google.com/d/forum/datastage4you
For WHATSAPP group , drop a msg to 91-88-00-906098






No comments :

Post a Comment