What is EPOCH ?

EPOCH is time in seconds since 00:00:00 Coordinated Universal Time (UTC), Thursday, 1 January 1970 ( 1970/01/01 00:00:00 UTC). Or the epoch is Unix time 0 (midnight 1/1/1970), but 'epoch' is often used as a synonym for 'Unix time'. EPOCH time is widely used in every field of IT sector (specially in unix like OS ) where we have to keep difference is generation of something based on time.

Maximum number of DataStage Jobs in a project

Filesystems can have a hard limit on number of subdirectories that can be contained in a directory, for example 32766 (32K -2) for Linux

For every job, DataStage will create hash files as subdirectories in the project directory. Here is a list of the subdirectories created for every job. In this list "nn" represents the number (or id) of the job:

DS_TEMPnn

RT_CONFIGnn

RT_STATUSnn

RT_LOGnn

RT_BPnn

RT_BPnn.O

RT_SCnn (only for parallel jobs)

Windows Command Prompt setup for DataStage Command Line

As we know that for using Datastage Command Line in Linux system, we need to setup linux shell session likewise we need to do same for Windows if server is installed on Windows server.

Follow below steps to setup your Datastage command Line ---

Get list of locked item on DataStage Server with Script

This script can be run with or without any argument and provide the list of Job, item locked in DataStage environment by any user. This script will help release the locks if user doesn't locked the item.

Script Name : DSgetJobLocks.sh
Usage : ./DSgetJobLocks.sh [one or more argument]

DOS Batch Script to Export DataStage Jobs Automatically from a Project

Copy the following script into a text file and rename into ".bat" file.

 This batch script is used to Export all jobs from a Projects
 This script must be run from a DataStage client machine and the parameters below should be given

 Host is server name
 User is DataStage username
 Password is DataStage password
 Imp Location is the directory where the datastage dsx's are stored

How to Schedule Jobs and Sequencers via Datastage Scheduler

Datastage Native Job Scheduler

Datastage includes a scheduling option, it does not have its own. DataStage doesn't include a "scheduler" so leverages the underlying O/S. For UNIX that means cron and a check of the crontab entries for the scheduling user will have what you need. DataStage leverages cron for recurring schedules and at for 'one off' schedules. For Windows, it uses scheduled tasks of Windows. 

From the operating system command line, logged in as the scheduling user, a "crontab -l" will list the scheduled jobs.

A SQL Client Tool - TeraData Studio Express

Teradata Studio Express is a graphical Java program, developed on the Eclipse Rich Client Platform (RCP),  that will allow you to view the structure of a JDBC compliant database, browse the data in tables, issue SQL commands etc.
This is a nice tool which can connect many DBs like Aster database, DB2 for LUW, DB2 for i5/OS, DB2 for z/OS, Oracle, SQL Server, Generic JDBC connection and of course TeraData DB.

Some Windows Shortcuts

1. CTRL+C (Copy)
2. CTRL+X (Cut)
3. CTRL+V (Paste)
4. CTRL+Z (Undo)
5. DELETE (Delete)
6. SHIFT+DELETE (Delete the selected item permanently without placing the item in the Recycle Bin)
7. CTRL while dragging an item (Copy the selected item)
8. CTRL+SHIFT while dragging an item (Create a shortcut to the selected item)
9. F2 key (Rename the selected item)

How can I run the osh command of the InfoSphere Parallel Engine?

The osh command is the main program of the InfoSphere Parallel Engine. This command is used by DataStage to perform several different tasks including parallel job execution and dataset management. Normally, there is no need to run this command directly but sometimes it is useful to use it for troubleshooting purposes.

To run this command there are 3 environment variables that must be set. These are:

1. APT_ORCHHOME should point to Parallel Engine location
2. APT_CONFIG_FILE should point to a configuration file
3. LD_LIBRARY_PATH should include the path to the parallel engine libraries. Please note that the name of this environment variable may take a different name (such as LIBPATH in AIX or SLIB_PATH in HP-UX) depending on your Operating System. Note: This variable does not need to be set in Windows environments.

Commands to delete files in &PH& directories in IBM InfoSphere DataStage

How should we manage cleaning up the &PH& directories for my DataStage projects? Can the process be automated?

Files in the &PH& directories under DataStage project directories store runtime information when jobs are run and need to be cleared out periodically.

 

Steps :

To clear the &PH& directory from within DataStage:

Datastage Job Scheduler

Datastage includes a scheduling option, it does not have its own. DataStage doesn't include a "scheduler" so leverages the underlying O/S. For UNIX that means cron and a check of the crontab entries for the scheduling user will have what you need. DataStage leverages cron for recurring schedules and at for 'one off' schedules. For Windows, it uses scheduled tasks of Windows.

From the operating system command line, logged in as the scheduling user, a "crontab -l" will list the scheduled jobs.

WinSCP - Save all configurations in an INI file

You can configure WinSCP to save configurations to an INI file (instead of Windows Registry):

• Open WinSCP and go to the Preferences section.
• Next to "Other general options" click on the "Preferences" button.

Setting Up and Using PuTTY

•  
• Run the installer package and extract the PuTTY components. PuTTY may be downloaded from http://www.chiark.greenend.org.uk/~sgtatham/putty/
• Start the PuTTYGen application. This application is used to create your authentication keys. Select either "SSH2 RSA" or "SSH2 DSA" for the "Type of key to generate", with 1024 bit key size or greater. Press the "Generate" button and follow the instructions on the screen.
  

How To Hide Text In Notepad

Here is a small trick to hide text inside your windows default text editor i.e Notepad usingcommand prompt,  but this trick works only on NTFS file system.

Steps to hide text in a Notepad
1. Open your command prompt Start-->Run and Type cmd
2. Type cd.. to move to C:\> drive or Type cd desktop to move to your desktop.
3. Type the below code in your 

First time IIS User Setup - DataStage 8.7

Typical users list:

• wasadmin
• isadmin
• db2inst1,db2iadm1
• db2fenc1,db2fadm1
• dasusr1,dasadm1
• xmeta,xmeta (db owner)
• xmetasr (staging area)
• iauser,iadb (db user)
• dsadm,dstage

What Service is Running on which Port in Windows

Finding out what application is running on a specific port in Windows is easy by using two command line utilities. By using netstat with the proper switches you can list out all of the ports that are being used and then use Tasklist to list all of the applications that are running. The combination of the two utilities will allow you to figure out which application us using which port.

As an example you can use the below directions to figure out At which port Oracle is running....

Stopping and Starting Oracle 11g Database Services in Windows

There are lot of services consuming a lot of CPU when an Oracle DB is running on machine. If you are using Oracle for learning/testing purpose only then you do not want to keep it up all the time.

So, here I am sharing 2 batch files for Stopping & Starting the Oracle Services on Windows which can do your work in one click only...

DataStage - Problem when running large parallel jobs on Windows

If you have large parallel jobs (8.1) running in Windows environment then it is worth checking the servers event log. You could find an error related to Nutcracker.exe. An administrator with access right on the server can see the events logs.

Nutcracker error could come up while running very heavy jobs where total cpu utilization is upto or above 90% and server starts using virtual memory. This is just an example of the impact of running large parallel jobs in Windows server. Hence to make the system somewhat full proof following changes can be done in Windows registry. The idea is to tune the TCP/IP buffer, increase the heap size and set the virtual memory to the recommended amount.

DataStage Error: Job xxx is being accessed by another user

DataStage Error:  Job xxx is being accessed by another user  

Take below steps to get rid of locks in datastage --

• In web console check active sessions and if job is listed disconnect session.
  
  To view all active sessions:
  
  In the Web console, click the Administration tab.
  In the Navigation pane, select Session Management > Active Sessions
  

Clean ReStart of the IBM InfoSphere DataStage Engine

If you suspect some Client sessions did not cleanly end or disconnect, you can perform a clean restart to make sure the DataStage Engine is starting from scratch. Please make sure all clients are logged off and no jobs are running before performing the below steps.

Here are the instructions for a clean restart of DataStage Engine: