We have moved to www.dataGenX.net, Keep Learning with us.

Monday, September 09, 2013

Deleting temporary lookuptable files in IBM InfoSphere DataStage

When a DataStage job with a lookup stage aborts, there may be lookuptable files left in the resource directories and they will consume space. The filenames are similar to "lookuptable.20091210.513biba"

When a job aborts it leaves the temporary files for postmortem review in the resource directories. Usually that is done in scratch, however, for lookup files, they are created in resource. Lookup filesets will not go away, just like regular datasets.

A lookup fileset looks like:

A lookup file looks like:

Look for files with filenames similar to "lookuptable.yyyymmdd.nnnnnnn" left on disk when no jobs are running.

Solution :

All files with lookuptable at the beginning of the filename can be removed as long as there are no running jobs. These files get recreated with every new run of the job and are never reused. If the job runs successfully, then only the lookuptable file created during that job run is removed.

It is recommended that you create a maintenance script that cleans up the lookuptable* files on a regular basis when jobs are not executing.