What is EPOCH ?

EPOCH is time in seconds since 00:00:00 Coordinated Universal Time (UTC), Thursday, 1 January 1970 ( 1970/01/01 00:00:00 UTC). Or the epoch is Unix time 0 (midnight 1/1/1970), but 'epoch' is often used as a synonym for 'Unix time'. EPOCH time is widely used in every field of IT sector (specially in unix like OS ) where we have to keep difference is generation of something based on time.

Get nth Highest or Lowest data from Table

Here, we are considering a table student and we need to find out Nth highest and lowest marks of a student.

Nth Highest

select * from student s1 where N-1 = ( select count(distinct(marks)) from student s2 where s1.marks > s2.marks)

Order of temporary disk space uses by Sort stage

The Sort stage uses temporary disk space when performing a sort. It looks in the following locations, in the
following order, for this temporary space.

1. Scratch disks in the disk pool sort (you can create these pools in the configuration file).
2. Scratch disks in the default disk pool (scratch disks are included here by default).

Encrypt Password for DataStage Command Line

As datastage user,  we need lots of scripts help our daily  jobs,  most of these scripts need authenticated by datastage engine,  that means we need to pass our Datastage user password in script or command line. 
                     But there is a problem if we are using "Password" here, It can be view by checking the content of script or command history. To secure password, there is a way provided by datastage and that is ENCRYPT the password before using. Let's see how we can do that ---

Get nth character in a String in Linux

This is a  very popular Unix question, How to get 3rd char from a 3rd column of a tab delimited file ? We are usually know how to get the nth column by using "CUT" or "AWK" command. But not aware with this "CUT" options which can give the particular column. Lets see how to use this -

Get nth character in a file in Linux

Oracle SQL Tuning Tips - 3

• Oracle automatically performs simple column type conversions(or casting) when it compares columns of different types. Depending on the type of conversion, indexes may not be used. Make sure you declare your program variables as the same type as your Oracle columns, if the type is supported in the programming language you are using.Use:
  SELECT emp_no, emp_name, sal FROM emp WHERE emp_no = ’123′;
  HERE if emp_no indexed numeric, then after implicit conversion query will be:
  SELECT emp_no, emp_name, sal FROM emp WHERE emp_no = TO_NUMBER(’123′);
  Thus, index is used in this case.
  Don’t use:
  SELECT emp_no, emp_name, sal FROM emp WHERE emp_type = 123;
  HERE if emp_type is indexed varchar2, then after implicit conversion query will be:
  SELECT emp_no, emp_name, sal FROM emp WHERE TO_NUMBER(emp_type) = 123;
  Thus, index will not be used in this case.

Oracle SQL Tuning Tips - 2

• Don’t forget to tune views. Views are SELECT statements and can be tuned in just the same way as any other type of SLECT statement can be. All tuning applicable to any SQL statement are equally applicable to views.
• Avoid including a HAVING clause in SELECT statements. The HAVING clause filters selected rows only after all rows have been fetched. Using a WHERE clause helps reduce overheads in sorting, summing, etc. HAVING clauses should only be used when columns with summary operations applied to them are restricted by the clause.Use:
  SELECT city FROM country WHERE city!= ‘Vancouver’ AND city!= ‘Toronto’; GROUP BY city;
  Rather than:
  SELECT city FROM country GROUP BY city HAVING city!= ‘Vancouver’ AND city!= ‘Toronto’;
  

Oracle SQL Tuning Tips - 1

Consideration when writing an SQL statement is that it returns a correct result. The second is that it be the most efficient for a given situation.  You can use many different SQL statements to achieve the same result. It is often the case that only one statement will be the most efficient choice in a given situation.

Remember that processing SQL is a sequence of Parse (syntax check and object resolution), Execution (required reads and writes), and Fetch (row results retrieved, listed, sorted, and returned). SQL “tuning” consists, quite simply, of reducing one or more of them.

Note: generally Parse is the greatest time and resource hog. Parse overhead can be minimized by the use of Procedures, Functions, Packages, Views, etc.

Some more design tips for DataStage Job Development

• Common information like home directory, system date, username, password should be initialized in a global variable and then variable should be referred everywhere.

• Stage Variables allow you to hold data from a previous record when the next record, allowing you to compare between previous and current records. Stage variables also allow you return multiple errors for a record of information. By being able to evaluate all data in a record and not just error on the first exception that is found, the cleanup of data is more efficient and requires less iteration.

Setting up "CRON" Jobs in Nix

If you want to use the emacs editor for editing cron jobs, then, set the following in your "/home/user/.bash_profile"

EDITOR=emacs

Then, to edit cron jobs

$ crontab -e

You may want to put in the following header

DataStage Naming Standard

For maintaining the design and easy understanding of what happening inside the job, we are giving meaningful name to stages, links, jobs and sequencer. So that other developer/designer can have a idea about purpose of job/seq by seeing the design.

Here I have shared some DataStage Naming Standard we usually follow.

Check whether DataStage Job is Multi-Instance or not with Sctipt

The following script can be used to check if the given datastage job is multi instance or not.

Arguments to the Scripts:

Arg1:Datastage Project Name
Arg2:JobName

DataWareHouse (ETL) Designing Steps

A) Planning and Designing Steps

i) Requirement and Realities Gathering

                       1. Business Needs

                       2. Data Profiling and other Data source realities

3. Compliance Requirement

4. Security Requirement

5. Data Integration

6. Data Latency

7. Archiving and Lineage

8. End user delivery interfaces

9. Available Development Skills

10. Available management Skills

11. Legacy licenses

Datastage Coding Checklist

1. Ensure that the null handling properties are taken care for all the nullable fields. Do not set the null field value to some value which may be present in the source.
2. Ensure that all the character fields are trimmed before any processing. Normally extra spaces in the data may lead to some errors like lookup mismatch which are hard to detect.
3. Always save the metadata (for source, target or lookup definitions) in the repository to ensure re usability and consistency.

List of strong points for InterView :-)

Most of time we stuck when Interviewer ask about our Strong Point, there are lot of reasons behind this ;-) anyways, Here sharing some words which can help you to decide. Pick which suits you but know them ;-) before using.


• A activating, adapting, administering, analyzing information, arranging, advising

• B budgeting, building teams, briefing, balancing,

• C communicating, controlling, co-ordinating, creating, checking, counseling, compiling, coaching

• D deciding, detailing, developing people, directing, devising, discovering, data input

Removing last or more character of string

Want to remove last character from a string in Linux, We can do this with help of SED commands.

Use this

a) Remove last char
sed 's/.$//' input
or
sed 's/.\{1\}$//' input

i.e. -
echo 123456 | sed 's/.$//'
12345

echo 123456 | sed  's/.\{1\}$//'
12345

ETL Job Design Standards - 2

Part 1 --> ETL Job Design Standards - 1

Parameter Management Standards

This section defines standards to manage job parameters across environments. Jobs should use parameters liberally to avoid hard coding as much as possible. Some categories of parameters include: 

• Environmental parameters, such as directory names, file names, etc.
• Database connection parameters
• Notification email addresses
• Processing options, such as degree of parallelism

Adding background colors or background images to emails and other Notes documents - Part 2

Adding background colors or background images to Lotus Notes mail - Part 1

 

To add a reusable graphic from an application

1. While creating or editing a document, click Text > Text Properties.
2. In the properties box, click the arrow next to Text and select Document.
3. Click the Background tab.
4. Enter the name of the graphic in the text field in the Graphic or Resource section, or use the folder button to select a graphic.
   

14 design tips for better performance in Datastage

1) Avoid unnecessary type conversions: set the OSH_PRINT_SCHEMAS environment variable to verify that run time schemas match the job design column definitions. If you are using stage variables on a Transformer stage, ensure that their data types match the expected result types.

2) Use Transformer stages sparingly and wisely. Transformer stages can slow down your job. Do not have multiple stages where the functionality could be incorporated into a single stage, and use other stage types to perform simple transformation operations