Regular Expressions in Linux

What Are Regular Expressions?

A regular expression is a pattern template you define that a Linux utility Uses to filter text. A Linux utility (such as the sed editor or the awk program) matches the regular expression pattern against data as that data flows Into the utility. If the data matches the pattern, it's accepted for processing.

              If the data doesn't match the pattern, it's rejected. The regular expression pattern makes use of wildcard characters to represent one or more characters in the data stream.

Types of regular expressions:

There are two popular regular expression engines:

The POSIX Basic Regular Expression (BRE) engine

The POSIX Extended Regular Expression (ERE) engine

A.     Defining BRE Patterns:

The most basic BRE pattern is matching text characters in a data stream.

What is EPOCH ?

EPOCH is time in seconds since 00:00:00 Coordinated Universal Time (UTC), Thursday, 1 January 1970 ( 1970/01/01 00:00:00 UTC). Or the epoch is Unix time 0 (midnight 1/1/1970), but 'epoch' is often used as a synonym for 'Unix time'. EPOCH time is widely used in every field of IT sector (specially in unix like OS ) where we have to keep difference is generation of something based on time.

Maximum number of DataStage Jobs in a project

Filesystems can have a hard limit on number of subdirectories that can be contained in a directory, for example 32766 (32K -2) for Linux

For every job, DataStage will create hash files as subdirectories in the project directory. Here is a list of the subdirectories created for every job. In this list "nn" represents the number (or id) of the job:

DS_TEMPnn

RT_CONFIGnn

RT_STATUSnn

RT_LOGnn

RT_BPnn

RT_BPnn.O

RT_SCnn (only for parallel jobs)

Installing PyMongo with MongoDB in Linux

1. Download Pymongo Source from HERE.
2. Then follow below steps --

MongoDB Configuration in Linux

DOWNLOAD the stable version of MongoDB. It will a tar file.

1. Create a folder named 'learn' ( or whatever you want to give )

     $ mkdir -p /learn/mongodb  /app/dbMongo

      # /learn/mongodb  = Holds the MongoDB source

      # /app/dbMongo = Holds the MongoDB database

2. Extract the Mongo DB zip file in /learn folder

  $ tar -xvf MongoDB.tar  /learn/mongodb

3.  change the permission of folder to user who run the db here-  In my case User - atul and Group - atul
  $ chown -R atul:atul  /learn/mongodb  /app/dbMongo

DataStage UNIX Commands

I've been asked so many times for Unix/Linux command which we usually used in DataStage. Here, I have tried to compile them together, please let me know if I missed something here. Njoy ....... :)

•Check the permissions
1.Log on to server as dsadm or root
2.cd $DSHOME/bin
3.ls -l | grep rws

The output should be (6) files owned by root with the setuid bit set -- especially the uv file.

Get nth character in a String in Linux

This is a  very popular Unix question, How to get 3rd char from a 3rd column of a tab delimited file ? We are usually know how to get the nth column by using "CUT" or "AWK" command. But not aware with this "CUT" options which can give the particular column. Lets see how to use this -

Rescuer 'xxd' command in Linux

"xxd" - a Rescuer , you guys are thinking why I am saying it a Rescuer so this is why so -

In our day to day work in Datastage/Linux, sometimes we have to deal with "Non Printable Character" ( bad characters ;). To confirm these we have many commands but for me this one is more reliable as this is displaying Hexadecimal value for each char from input file.

Every 2 no is representing 1 char from a file, here I have represent a char "a" which hexadecimal value is "61".

\n not creating new line in Linux

When we are using Linux, Some Shell are quite good to understand the Escape characters we want to print but some are not.

This is how we have to make them understood these characters -


$ echo "Atul \n Singh"
Atul \n Singh

Cut, Copy n Paste in VI editor

We Engineers are well trained in Cut, Copy and Paste things :-) and who the hell don't know the Ctrl-X, C n V. But when we come to same but in Linux Platform we are behaving like dumb ;P. Don't feel offended...I'll help u here

So Lets learn some command which can do the same in Linux VI editor --

Cut - Ctrl-X

The command 'dd' is used for cutting text. Cutting current word, lines or many more summarized below -

dd Delete current line D Delete from cursor to end of line
d$ Delete from cursor to end of line
d0 Delete from cursor to beginning of line
dw Delete from cursor to end of current word
db Delete from cursor to beginning of current word
Ndd Delete N no of lines including current line

File descripters in Unix

A file descriptor is an abstract value of type Unix.file_descr, containing information necessary to use a file: a pointer to the file, the access rights, the access modes (read or write), the current position in the file
or we can say
Under the covers of the Unix operating system, files are referenced, copied, and moved by unique numbers known as file descriptors.

Three descriptors are predefined. They correspond to standard input, standard output, and standard error.

0 - stdin (Standard Input)  "<"
1 - stdout ( Standard Output) "1>" or ">"
2 - stderr ( Standard Error) "2>"

Show Hidden Character in VI Unix

If you need to show hidden characters such as tab, return carriage, you can use the following command in vi

In vi (command mode -> Escape) , type


:set list
to show all the hidden characters.

You can also type


:set nolist 
to turn it off



Like the Facebook Page & join Group
https://www.facebook.com/DataStage4you
https://www.facebook.com/groups/DataStage4you
https://twitter.com/datastage4you
For WHATSAPP group , drop a msg to 91-88-00-906098

Connect to Oracle DB with Linux Shell Script

In this post, we will see how to connect to Oracle DB in Linux Shell Script. There are some prerequisite for a Linux server to connect to Oracle DB. There should be oracle db driver installed on the Linux server from which you want to connect to Oracle DB. Check whether you can call/connect SQLPLUS from command line.
$ sqlplus
if you are not able to, check whether oracle path is set in .profile file. Lets assume, Oracle DB driver is installed on Server and Sqlplus path is set.

Now, there are some basic steps to connect to Oracle DB-
a) Connect to Oracle DB with UserID and Password
b) Execute the SQLs
c) Store the output in a file or send to standard output (terminal)

List all RPMs in Linux

Use this command

$ rpm -qa --qf '%11{SIZE} %{NAME}\n' | sort -k1nr

Interview Questions : Unix/Linux : Part-8

1. Display all the files in current directory sorted by size?
ls -l | grep '^-' | awk '{print $5,$9}' |sort -n|awk '{print $2}'


2. Write a command to search for the file 'map' in the current directory?
find -name map -type f

3. How to display the first 10 characters from each line of a file?
cut -c -10 filename

4. Write a command to remove the first number on all lines that start with "@"?
sed '\,^@, s/[0-9][0-9]*//' < filename

ETL Testing : Approach

Testing is undoubtedly an essential part of DW life-cycle but it received a few attention with respect to other design phases.

DW testing specification:
 – Software testing is predominantly focused on program code, while Software testing is predominantly focused on program code, while DW testing is directed at data and information DW testing is directed at data and information.  .
 – DW testing focuses on the correctness and usefulness of the information delivered to users information delivered to users
 – Differently from generic software systems, DW testing involves a huge data volume huge data volume, which significantly impacts performance and productivity.
 – DW systems are aimed at supporting any views of data, so the possible number of use scenarios is practically infinite and only few them are known from the beginning.
 – It is almost impossible to predict all the possible types of errors that will be encountered in real operational data.

How to Create Custom Stages in Datastage Parallel Jobs ?

In addition to the wide range of parallel stage types available, the Designer allows you to define your own stage types, which you can then use in parallel jobs.
For creating custom stage in datastage which will functions as per our requirement and we will be able to modify that stage as we may want. Here is some information on custom stage functionality in datastage in parallel jobs and how can we create our own wrapped custom stage.

Setting up "CRON" Jobs in Nix

If you want to use the emacs editor for editing cron jobs, then, set the following in your "/home/user/.bash_profile"

EDITOR=emacs

Then, to edit cron jobs

$ crontab -e

You may want to put in the following header

Get list of locked item on DataStage Server with Script

This script can be run with or without any argument and provide the list of Job, item locked in DataStage environment by any user. This script will help release the locks if user doesn't locked the item.

Script Name : DSgetJobLocks.sh
Usage : ./DSgetJobLocks.sh [one or more argument]

Linux & Unix Basics – Day 2

Directory Structure

/dev  -  Special files that represent devices
/sbin -  System utilities for system startup
/etc  -  System configuration files used by System Admins
/usr  -  contains the subdirectories bin, include, share etc
/home – contains all users’ home directories
/var  -  contains user’s mail files, crontab files etc.
/tmp – directory used by system processes to keep their temporary files. 

Shell

• User interface - Shell provides an interface to the user, wherein the user could issue his commands, and Shell displays output and error messages to the user.
• Command interpreter - Shell accepts command from user, and interprets it to the kernel.
• Command processor - Shell parses the command line arguments, expands the special meaning of meta characters, searches for the command, and if the command is found, then transfers control to the command.
• Programming language - Shell provides a native programming language.