We have moved to www.dataGenX.net, Keep Learning with us.

Friday, July 19, 2013

DataStage Jobs hang or have slow performance when temp directories have many files


Datastage jobs write multiple files to temporary directories which are not automatically cleaned up. When the number of files grows large, DataStage jobs will have slower performance or can appear to hang.

DataStage jobs write multiple files to temporary directories which are not automatically cleaned up. When the number of files grows large, DataStage jobs will have slower performance or can appear to hang. For sites who have been running DataStage for a year or more and have not previously performed cleanup of these directories, they could contain 100,000 or more files.
Perform periodic cleanup of old DataStage files in these directories to prevent performance problems relating to the number of files in the directories.


DataStage writes temporary files to the following locations. Normally these files are cleaned up when the job ends, though jobs which aborted may leave behind files. Check these temporary directories for old DataStage files which can be removed, or a large number of files in general (since other programs also write to these directories).

The directories are pointed to by the following environment variables (you can refer to the job log of any DataStage job to determine the value of these variables at job run time):
  • TEMP
  • TMPDIR
  • TMP

On Unix systems, also check location /tmp

Additionally, DataStage writes non-temporary files to the following locations which will not automatically be cleaned up. You can remove DataStage files from these directories that are older than longest job run time or older than the last DataStage engine restart. Unless you have very long running jobs or "always-on" web service jobs, it is generally safe to remove files more than 2 days old from these directories:
  • The &PH& directory within each project directory.
  • The directory pointed to by UVTEMP variable in the uvconfig file in the DSEngine directory. The primary files to cleanup are the capture files which capture the system output of external commands called from DataStage jobs, for example: capture0001360098aa1312969887

    On UNIX platforms, UVTEMP setting often points to /tmp directory.

    On Windows platform, if UVTEMP variable is undefined, it will default to the UVTEMP directory under DSEngine, for example: /opt/IBM/InformationServer/Server/DSEngine/UVTEMP
  • The directory pointed to by UVSPOOL variable in the uvconfig file in the DSEngine directory on Unix systems. On Windows, this variable may not exist in uvconfig.