Input/Output data buffering (on Link) in DataStage

To improve performance and resolve bottlenecks, you can specify how input and output data is buffered. Although the size and operation of the buffer are usually the same for all links on all stages, you can modify the settings for specific links.

By default, data is buffered so that no deadlocks occur. Be careful when changing data buffering settings because specifying inappropriate values might create a deadlock.

Any changes that you make to the properties on the Advanced tab are automatically reflected on the Advanced tab of the stage at the other end of the link.

To change the settings for data buffering:

1.    double-click the Stage.

2.    In the Input/Output/stage editor, select the link that you want to edit.

3.    On the Advanced tab, select one of the following options for the Buffering mode:

(Default)              Select this option to use the setting that is specified by the environment
                                 variable APT_BUFFERING _POLICY. The default setting for this environment
                                 variable is AUTOMATIC_BUFFERING.

Auto buffer          Select this option to buffer data only when buffering is required to prevent
                                 at a flow deadlock.

            Buffer                  Select this option to buffer all data for this link.

No buffer             Select this option to perform no buffering. Selecting this option might result
                          in deadlocks.

4.    If you selected Auto buffer or Buffer, specify the following values:

a.    Maximum memory buffer size (bytes): Specify the maximum amount of virtual memory, in bytes, to use for each buffer. The default size is 3145728 (3 MB).

b.    Buffer free run (percent): Specify how much of the available in-memory buffer to consume before the buffer writes to disk. The value ofBuffer free run is expressed as a percentage of Maximum memory buffer size. When the amount of data in the buffer consumes less than the percentage that is specified for Buffer free run, new data is accepted automatically. When the data exceeds this percentage, the buffer tries to write some of the data that it contains before accepting more data. The default value is 50%. You can set this value to greater than 100%, in which case the buffer continues to store data up to the specified percentage before writing to disk.

c.     Queue upper bound size (bytes): Specify the maximum amount of data, in bytes, that can be buffered, including data that is buffered in memory and on disk. The default, 0, means that the buffer size is limited only by the value that is specified for resource scratchdiskin the WebSphere® DataStage configuration file. If you specify a value for Queue upper bound size, the amount of data in the buffer cannot exceed this value (in bytes) plus the size of one block (32 KB).

d.    Disk write increment (bytes): Specify the size, in bytes, of blocks of data that are moved to and from disk by the buffering operator. The default value is 1048576 (1 MB). When you specify a value for this parameter, consider your needs for decreasing disk access and for increasing throughput. For example, specifying a larger block size reduces disk activity, but might decrease performance when data is written in smaller units. Likewise, specifying a smaller block size increases data throughput, but might increase disk activity.

5.    Click OK to save your changes and close the stage editor.

njoy the simplicity.......
Atul Singh