Regular Expressions in Linux

What Are Regular Expressions?

A regular expression is a pattern template you define that a Linux utility Uses to filter text. A Linux utility (such as the sed editor or the awk program) matches the regular expression pattern against data as that data flows Into the utility. If the data matches the pattern, it's accepted for processing.

              If the data doesn't match the pattern, it's rejected. The regular expression pattern makes use of wildcard characters to represent one or more characters in the data stream.

Types of regular expressions:

There are two popular regular expression engines:

The POSIX Basic Regular Expression (BRE) engine

The POSIX Extended Regular Expression (ERE) engine

A.     Defining BRE Patterns:

The most basic BRE pattern is matching text characters in a data stream.

Get Datastage Job Lock List by Script

** This script will fetch all locks currently on Datastage Server irrespective of Projects.
** We can modify this script to correct the data display format as per our need
** Can modify sort command as it changes its behavior in different Linux flavour

Encrypt Password for DataStage Command Line

As datastage user,  we need lots of scripts help our daily  jobs,  most of these scripts need authenticated by datastage engine,  that means we need to pass our Datastage user password in script or command line. 
                     But there is a problem if we are using "Password" here, It can be view by checking the content of script or command history. To secure password, there is a way provided by datastage and that is ENCRYPT the password before using. Let's see how we can do that ---

Rescuer 'xxd' command in Linux

"xxd" - a Rescuer , you guys are thinking why I am saying it a Rescuer so this is why so -

In our day to day work in Datastage/Linux, sometimes we have to deal with "Non Printable Character" ( bad characters ;). To confirm these we have many commands but for me this one is more reliable as this is displaying Hexadecimal value for each char from input file.

Every 2 no is representing 1 char from a file, here I have represent a char "a" which hexadecimal value is "61".

TimeStamp Comparison in Linux

Comparing Date, Time or TimeStamp is tricky thing in Linux. This is how you can achieve this -


By converting date to Seconds
$ date +%s
the above command will current time in seconds since 1970-01-01 00:00:00 UTC

if you have time in string form you can use below command to convert them like -

VI Cheat Sheet

Cursor movement

• h - move left
• j - move down
• k - move up
• l - move right
• w - jump by start of words (punctuation considered words)
• W - jump by words (spaces separate words)
• e - jump to end of words (punctuation considered words)
• E - jump to end of words (no punctuation)
• b - jump backward by words (punctuation considered words)
• B - jump backward by words (no punctuation)
• 0 - (zero) start of line
• ^ - first non-blank character of line
• $ - end of line
• G - Go To command (prefix with number - 5G goes to line 5)

Note: Prefix a cursor movement command with a number to repeat it. For example, 4j moves down 4 lines.

\n not creating new line in Linux

When we are using Linux, Some Shell are quite good to understand the Escape characters we want to print but some are not.

This is how we have to make them understood these characters -


$ echo "Atul \n Singh"
Atul \n Singh

Connect to Oracle DB with Linux Shell Script

In this post, we will see how to connect to Oracle DB in Linux Shell Script. There are some prerequisite for a Linux server to connect to Oracle DB. There should be oracle db driver installed on the Linux server from which you want to connect to Oracle DB. Check whether you can call/connect SQLPLUS from command line.
$ sqlplus
if you are not able to, check whether oracle path is set in .profile file. Lets assume, Oracle DB driver is installed on Server and Sqlplus path is set.

Now, there are some basic steps to connect to Oracle DB-
a) Connect to Oracle DB with UserID and Password
b) Execute the SQLs
c) Store the output in a file or send to standard output (terminal)

Handling Filename with Spaces in Linux

Spaces in Unix/Linux Filename is not a good habit as usually it is used as delimiter for field values rather than being in filename itself. In Linux, SPACE is used as a IFS ( Internal Field Separator) that is used for word splitting after expansion and to split lines into words with the read build in command. The default values of IFS is SPACE, TAB, NEW LINE.

     This is why whenever we are doing something with File-name having spaces in Linux, we got error like..
: No such File or Directory

   What if you got some files from a "Unix dumb" Server source, How you will handle the filename with spaces, Here are some simple ways to deal with it.

Interview Questions : Unix/Linux : Part-8

1. Display all the files in current directory sorted by size?
ls -l | grep '^-' | awk '{print $5,$9}' |sort -n|awk '{print $2}'


2. Write a command to search for the file 'map' in the current directory?
find -name map -type f

3. How to display the first 10 characters from each line of a file?
cut -c -10 filename

4. Write a command to remove the first number on all lines that start with "@"?
sed '\,^@, s/[0-9][0-9]*//' < filename

Get list of locked item on DataStage Server with Script

This script can be run with or without any argument and provide the list of Job, item locked in DataStage environment by any user. This script will help release the locks if user doesn't locked the item.

Script Name : DSgetJobLocks.sh
Usage : ./DSgetJobLocks.sh [one or more argument]

Linux & Unix Basics – Day 2

Directory Structure

/dev  -  Special files that represent devices
/sbin -  System utilities for system startup
/etc  -  System configuration files used by System Admins
/usr  -  contains the subdirectories bin, include, share etc
/home – contains all users’ home directories
/var  -  contains user’s mail files, crontab files etc.
/tmp – directory used by system processes to keep their temporary files. 

Shell

• User interface - Shell provides an interface to the user, wherein the user could issue his commands, and Shell displays output and error messages to the user.
• Command interpreter - Shell accepts command from user, and interprets it to the kernel.
• Command processor - Shell parses the command line arguments, expands the special meaning of meta characters, searches for the command, and if the command is found, then transfers control to the command.
• Programming language - Shell provides a native programming language.

Check whether DataStage Job is Multi-Instance or not with Sctipt

The following script can be used to check if the given datastage job is multi instance or not.

Arguments to the Scripts:

Arg1:Datastage Project Name
Arg2:JobName

Search the string in the file : Linux/Unix

Here, I am going to share a small shell utility which will ask source file name and the pattern which need to be searched in file and give the formatted output with the line no with line. You can select the no of lines which display before and after the string.

Colourizing Unix Programs & Shell Scripts

 

In an ANSI compatible terminal (like xterm, rxvt, konsole ...) text may be shown in colors different from black/white. Here first the colour coding is shown for shell programming .

Each color code is present after an escape sequence. This can be given by

echo  –e "\033[1m Demo Test \033[0m\n";

DataStage client receives error when compile jobs: Cannot get exclusive access to executable file for job sampleJob - job may be being monitored

Problem :

DataStage user sees the following error in Designer client when attempting to compile a specific job:
Cannot get exclusive access to executable file for job sampleJob - job may be being monitored.

Similar problems may occur in other releases.

Interview Questions : Unix/Linux : Part-7

For more : visit here

1. How to display the processes that were run by your user name ?
ps -aef | grep <user_name>

2. Write a command to display all the files recursively with path under current directory?
find . -depth -print

Interview Questions : Unix/Linux : Part-6

1.       How do you write the contents of 3 files into a single file?

cat file1 file2 file3 > file

2.       How to display the fields in a text file in reverse order?

awk 'BEGIN {ORS=""} { for(i=NF;i>0;i--) print $i," "; print "\n"}' filename

3.       Write a command to find the sum of bytes (size of file) of all files in a directory.

ls -l | grep '^-'| awk 'BEGIN {sum=0} {sum = sum + $5} END {print sum}'

The Unix "SCRIPT" command : a command recorder

script is a standard Unix command that records a script of your interaction with the Unix system. Once it's started, it works "in the background", meaning that you continue to work normally, but the script session is dumping everything that shows up on your screen (more or less*) into some file. It probably would be better called carbon-copy. 

The most common use of script  is to document terminal session. By running script you log all the information displayed on your terminal. You can then print the log file or view it with an editor. In a way script is a specialized tee for the shell.

Saving Shell Script with executable permissions: tips & tricks

In our tech routine we daily usually need to write small script to do task easily. Sometimes its irritating to edit/create and save and give the executable permissions.

What if when we create a file it have the executable permissions, Yes, we can do that with the help of umask but this will applicable on all files so don’t want that. So here is the solution….