My Blog Report - till now

First of all, I would like to say thank you all for loving this blog so much. As you are aware we have moved to "www.DataGenX.net" now. Keep Learning, Keep Sharing n Keep loving us :-)

Today I am sharing Blog Report for "Nuts & Bolts of DataStage".

Like the Facebook Page & join Group
https://www.facebook.com/DataStage4you
https://www.facebook.com/groups/DataStage4you
https://twitter.com/datagenx
https://groups.google.com/d/forum/datagenx
For WHATSAPP group , drop a msg to 91-88-00-906098

DataGenX

Hi Guys,

We have moved to new Site now , Kindly Visit

http://www.datagenx.net/

Twitter -- https://twitter.com/datagenx
Reddit --  https://www.reddit.com/user/datagenx
Pinterest -- https://www.pinterest.com/atulsingh0/datagenx/

Regular Expressions in Linux

What Are Regular Expressions?

A regular expression is a pattern template you define that a Linux utility Uses to filter text. A Linux utility (such as the sed editor or the awk program) matches the regular expression pattern against data as that data flows Into the utility. If the data matches the pattern, it's accepted for processing.

              If the data doesn't match the pattern, it's rejected. The regular expression pattern makes use of wildcard characters to represent one or more characters in the data stream.

Types of regular expressions:

There are two popular regular expression engines:

The POSIX Basic Regular Expression (BRE) engine

The POSIX Extended Regular Expression (ERE) engine

A.     Defining BRE Patterns:

The most basic BRE pattern is matching text characters in a data stream.

Error while creating new jobs in DataStage

If you face the error error similar to below can occur when trying to save a newly created job:

Error On CREATE.FILE command
Creating file "RT_CONFIG4817 as type 30, mkdbfile: connot create file
RT_CONFIG4817. Unable to create operating system file "RT_CONFIG4817"

Types of Jobs in DataStage

Server jobs

• Executed by the DataStage server engine
• Compiled into Basic
• Runtime monitoring in DataStage Director
• No parallel capability

What is EPOCH ?

EPOCH is time in seconds since 00:00:00 Coordinated Universal Time (UTC), Thursday, 1 January 1970 ( 1970/01/01 00:00:00 UTC). Or the epoch is Unix time 0 (midnight 1/1/1970), but 'epoch' is often used as a synonym for 'Unix time'. EPOCH time is widely used in every field of IT sector (specially in unix like OS ) where we have to keep difference is generation of something based on time.

Get list of activated tools in Datastage Installation

If we want to check what features have been installed with datastage, we can check it by using LicensingServiceAdmin  tool. This will list down all the tool packs or features including Qualitystage as it comes as a part of datastage.


Tool Location :  {IS_INSTALL_DIR}/ASBServer/bin

Maximum number of DataStage Jobs in a project

Filesystems can have a hard limit on number of subdirectories that can be contained in a directory, for example 32766 (32K -2) for Linux

For every job, DataStage will create hash files as subdirectories in the project directory. Here is a list of the subdirectories created for every job. In this list "nn" represents the number (or id) of the job:

DS_TEMPnn

RT_CONFIGnn

RT_STATUSnn

RT_LOGnn

RT_BPnn

RT_BPnn.O

RT_SCnn (only for parallel jobs)

Get nth Highest or Lowest data from Table

Here, we are considering a table student and we need to find out Nth highest and lowest marks of a student.

Nth Highest

select * from student s1 where N-1 = ( select count(distinct(marks)) from student s2 where s1.marks > s2.marks)

SQL Scenario - 1

You have a table like below :

Name Age Gender
Atul         11 F
Ram 10 F
Disha  6 M
Pavan  8 F
Mani         10 M
Lovely 10 M
Rani    9 M

Unlocking XMETA locks from jobs

How can I unlock an XMETA locked job?

The job cannot be exported and when attempting to open the job, the following error appears:

Unable to open the Job
The Job 'TestJob' is locked by user 'dsadm'
(SessionID '8F1B0326-4F7E-45A8-B8EC-EAA8C4780528')

Do you want to continue and open a read-only view of the Job?

This issue arises when a lock still remains in XMETA for the corresponding job. Information Server will lock the job to ensure that there is no indeterminancy regarding the editing and saving of a job.
The most common causes of this is when a user has closed a client window unexpectedly (e.g. Force Terminate).

What happen when Import Sequential File with Extra Column

Job reads sequential file and load into RDBMS table.

What happen if sequential file has extra column in the back of each record?

It depends on data type of last column defined by DS and RDBMS table.

• If data type of last column = varchar, extra column will be included as last column.

DataStage Installation Checklist

• Review the system requirements.
• learn about tiers and components.
• Design the topology of your installation.
• Determine whether to pre-install your database system.
• Determine whether to pre-install IBM WebSphere Application Server or pre-configure an existing installation of WebSphere Application Server.

trim leading 0's from decimal in DataStage

1. Convert the decimal (or whatever it was) to String.
2. then trim '0's from left.

Installing PyMongo with MongoDB in Linux

1. Download Pymongo Source from HERE.
2. Then follow below steps --

Error - mongod shell doesn't start, /data/db doesn't exsist

Error - mongod shell doesn't start, /data/db doesn't exsist


If you face this error while starting MongoDB (mongod), most probably there are 2 reasons for this -

a.  /data/db is not exist at all
b. mongod process don't have access on /data/db

Solution :-

The default db path for mongodb is /data/db/.
So when you run:

MongoDB Configuration in Linux

DOWNLOAD the stable version of MongoDB. It will a tar file.

1. Create a folder named 'learn' ( or whatever you want to give )

     $ mkdir -p /learn/mongodb  /app/dbMongo

      # /learn/mongodb  = Holds the MongoDB source

      # /app/dbMongo = Holds the MongoDB database

2. Extract the Mongo DB zip file in /learn folder

  $ tar -xvf MongoDB.tar  /learn/mongodb

3.  change the permission of folder to user who run the db here-  In my case User - atul and Group - atul
  $ chown -R atul:atul  /learn/mongodb  /app/dbMongo

Trigger types in Sequencer Job

While designing Sequencer jobs when we are using any Sequencer Stage ( Job activity, Command activity or others ), it has a tab named 'trigger' which actually determine the actions that occur when the activity runs. Each activity can output different trigger types. 

Three types of triggers are available, with some types having subtypes.

Conditional

A conditional trigger runs the target activity if the source activity fulfills the specified condition. The condition is defined by an expression, and can be one of the following types:

Data Warehouse Load Methods

A data warehouse is a warehouse that contains data. Sounds funny doesn't it? Imagine a warehouse as a storage structure that may be physical or logical. In the DW it's both...logical as in the data model representation and physical as to the hard disks and other computer equipment that support the data warehouse.

The first question should be "How does the data get there?" Here are three key areas to keep in mind....

Availability - will someone create a report with transactions completed 20 minutes ago or will it be with yesterday's data?

Data sources in DataStage

IIS Datastage connectivity options give us a wide scope to connect with different source or targets. It's support RDBMS, ERP, z/OS DB, OLAP system and many more.

Below listed Data Sources are available in IIS v11.3