We have moved to www.dataGenX.net, Keep Learning with us.

Tuesday, August 27, 2013

Stage to OPerator in DataStage - the Real Driver


The parallel job stages are built on operators. Or Operators are individual parallel engine stages where the data actually flows. In a typical job flow, operators are end-points, and data sets are the links between the operators.

Each operator listed in the DUMP SCORE is a number of processes that are dependent on:
  • the job's established configuration file (APT_CONFIG_FILE)
  • constrained by the node pool settings
  • the operator configuration in the parallel engine code
  • Several environment variables, such as APT_DISABLE_COMBINATION, being set/unset.
 This is not one to one mapping between stage and operators. Some depends on the options which is used with the stage and can be used in different stages.



InfoSphere DataStage Stage Operator Options (where applicable) Comment
File Set - - Represents a permanent data set
Sequential File Import Operator - file
-filepattern
Export Operator -file
File Set Import Operator -fileset
Export Operator -fileset
Lookup File Set Lookup Operator -createOnly
External Source Import Operator -source
-sourcelist
External Target Export Operator -destination
-destinationlist
Complex Flat File Import Operator
Transformer Transform Operator
BASIC Transformer Represents server job transformer stage (gives access to BASIC transforms)
Aggregator Group Operator
Join fullouterjoin Operator
innerjoin Operator
leftouterjoin Operator
rightouterjoin Operator
Merge Merge Operator
Lookup Lookup Operator
The oralookup Operator for direct lookup in Oracle table (`sparse' mode)
The db2lookup Operator for direct lookup in DB2® table (`sparse' mode)
The sybaselookup Operator for direct lookup in table accessed via iWay (`sparse' mode)
The sybaselookup Operator for direct lookup in Sybase table (`sparse' mode)
The sqlsrvrlookup Operator
Funnel Funnel Operators
Sortfunnel Operator
Sequence Operator
Sort The psort Operator
The tsort Operator
Remove Duplicates Remdup Operator
Compress Pcompress Operator -compress
Expand Pcompress Operator -expand
Copy Generator Operator
Modify Modify Operator
Filter Filter Operator
External Filter - - Any executable command line that acts as a filter
Change Capture Changecapture Operator
Change Apply Changeapply Operator
Difference Diff Operator
Compare Compare Operator
Encode Encode Operator -encode
Decode Encode Operator -decode
Switch Switch Operator
Generic - - Any operator
Surrogate Key Surrogate key operator
Column Import The field_import Operator
Column Export The field_export Operator
Make Subrecord The makesubrec Operator
Split Subrecord The splitsubrec Operator
Combine records The aggtorec Operator
Promote subrecord The makesubrec Operator
Make vector The makevect Operator
Split vector The splitvect Operator
Head Head Operator
Tail Tail Operator
Sample Sample Operator
Peek Peek Operator
Row Generator Generator Operator
Column generator Generator Operator
Write Range Map Writerangemap Operator
SAS Parallel Data Set - - Represents Orchestrate® parallel SAS data set.
SAS The sas Operator
The sasout Operator
The sasin Operator
The sascontents Operator
DB2/UDB Enterprise The db2read Operator
The db2write and db2load Operators
The db2upsert Operator
The db2lookup Operator For in-memory (`normal') lookups (
Oracle Enterprise The oraread Operator
The oraupsert Operator
The orawrite Operator
The oralookup Operator For in-memory (`normal') lookups
Informix® Enterprise The hplread operator
The hplwrite Operator
The infxread Operator
The infxwrite Operator
The xpsread Operator
The xpswrite Operator
Teradata Teraread Operator
Terawrite Operator
Sybase The sybasereade Operator
The sybasewrite Operator
The sybaseupsert Operator
The sybaselookup Operator For in-memory (`normal') lookups
SQL Server The sqlsrvrread Operator
The sqlsrvrwrite Operator
The sqlsrvrupsert Operator
The sybaselookup Operator For in-memory (`normal') lookups
iWay The iwayread Operator
The iwaylookup Operator