Regular Expressions in Linux

What Are Regular Expressions?

A regular expression is a pattern template you define that a Linux utility Uses to filter text. A Linux utility (such as the sed editor or the awk program) matches the regular expression pattern against data as that data flows Into the utility. If the data matches the pattern, it's accepted for processing.

              If the data doesn't match the pattern, it's rejected. The regular expression pattern makes use of wildcard characters to represent one or more characters in the data stream.

Types of regular expressions:

There are two popular regular expression engines:

The POSIX Basic Regular Expression (BRE) engine

The POSIX Extended Regular Expression (ERE) engine

A.     Defining BRE Patterns:

The most basic BRE pattern is matching text characters in a data stream.

Script to UnLock DataStage Jobs

This script will help you to unlock the all jobs, if you are a DS admin, or your datastage jobs , if a DS developer, without doing human error while executing command on Datastage server UV shell or Datastage Administrator Client.

Script is quite simple, Execute it , it will ask for Project Name and Job Name and Done if you have appropriate access on Datastage server.

May be need a modification as per your Datastage platform or if you need some fancy stuff to add on.  Happy Scripting !!

Get Datastage Job Lock List by Script

** This script will fetch all locks currently on Datastage Server irrespective of Projects.
** We can modify this script to correct the data display format as per our need
** Can modify sort command as it changes its behavior in different Linux flavour

Encrypt Password for DataStage Command Line

As datastage user,  we need lots of scripts help our daily  jobs,  most of these scripts need authenticated by datastage engine,  that means we need to pass our Datastage user password in script or command line. 
                     But there is a problem if we are using "Password" here, It can be view by checking the content of script or command history. To secure password, there is a way provided by datastage and that is ENCRYPT the password before using. Let's see how we can do that ---

Rescuer 'xxd' command in Linux

"xxd" - a Rescuer , you guys are thinking why I am saying it a Rescuer so this is why so -

In our day to day work in Datastage/Linux, sometimes we have to deal with "Non Printable Character" ( bad characters ;). To confirm these we have many commands but for me this one is more reliable as this is displaying Hexadecimal value for each char from input file.

Every 2 no is representing 1 char from a file, here I have represent a char "a" which hexadecimal value is "61".

TimeStamp Comparison in Linux

Comparing Date, Time or TimeStamp is tricky thing in Linux. This is how you can achieve this -


By converting date to Seconds
$ date +%s
the above command will current time in seconds since 1970-01-01 00:00:00 UTC

if you have time in string form you can use below command to convert them like -

VI Cheat Sheet

Cursor movement

• h - move left
• j - move down
• k - move up
• l - move right
• w - jump by start of words (punctuation considered words)
• W - jump by words (spaces separate words)
• e - jump to end of words (punctuation considered words)
• E - jump to end of words (no punctuation)
• b - jump backward by words (punctuation considered words)
• B - jump backward by words (no punctuation)
• 0 - (zero) start of line
• ^ - first non-blank character of line
• $ - end of line
• G - Go To command (prefix with number - 5G goes to line 5)

Note: Prefix a cursor movement command with a number to repeat it. For example, 4j moves down 4 lines.

\n not creating new line in Linux

When we are using Linux, Some Shell are quite good to understand the Escape characters we want to print but some are not.

This is how we have to make them understood these characters -


$ echo "Atul \n Singh"
Atul \n Singh

File descripters in Unix

A file descriptor is an abstract value of type Unix.file_descr, containing information necessary to use a file: a pointer to the file, the access rights, the access modes (read or write), the current position in the file
or we can say
Under the covers of the Unix operating system, files are referenced, copied, and moved by unique numbers known as file descriptors.

Three descriptors are predefined. They correspond to standard input, standard output, and standard error.

0 - stdin (Standard Input)  "<"
1 - stdout ( Standard Output) "1>" or ">"
2 - stderr ( Standard Error) "2>"

Connect to Oracle DB with Linux Shell Script

In this post, we will see how to connect to Oracle DB in Linux Shell Script. There are some prerequisite for a Linux server to connect to Oracle DB. There should be oracle db driver installed on the Linux server from which you want to connect to Oracle DB. Check whether you can call/connect SQLPLUS from command line.
$ sqlplus
if you are not able to, check whether oracle path is set in .profile file. Lets assume, Oracle DB driver is installed on Server and Sqlplus path is set.

Now, there are some basic steps to connect to Oracle DB-
a) Connect to Oracle DB with UserID and Password
b) Execute the SQLs
c) Store the output in a file or send to standard output (terminal)

Handling Filename with Spaces in Linux

Spaces in Unix/Linux Filename is not a good habit as usually it is used as delimiter for field values rather than being in filename itself. In Linux, SPACE is used as a IFS ( Internal Field Separator) that is used for word splitting after expansion and to split lines into words with the read build in command. The default values of IFS is SPACE, TAB, NEW LINE.

     This is why whenever we are doing something with File-name having spaces in Linux, we got error like..
: No such File or Directory

   What if you got some files from a "Unix dumb" Server source, How you will handle the filename with spaces, Here are some simple ways to deal with it.

Interview Questions : Unix/Linux : Part-8

1. Display all the files in current directory sorted by size?
ls -l | grep '^-' | awk '{print $5,$9}' |sort -n|awk '{print $2}'


2. Write a command to search for the file 'map' in the current directory?
find -name map -type f

3. How to display the first 10 characters from each line of a file?
cut -c -10 filename

4. Write a command to remove the first number on all lines that start with "@"?
sed '\,^@, s/[0-9][0-9]*//' < filename

Get list of locked item on DataStage Server with Script

This script can be run with or without any argument and provide the list of Job, item locked in DataStage environment by any user. This script will help release the locks if user doesn't locked the item.

Script Name : DSgetJobLocks.sh
Usage : ./DSgetJobLocks.sh [one or more argument]

Linux & Unix Basics – Day 2

Directory Structure

/dev  -  Special files that represent devices
/sbin -  System utilities for system startup
/etc  -  System configuration files used by System Admins
/usr  -  contains the subdirectories bin, include, share etc
/home – contains all users’ home directories
/var  -  contains user’s mail files, crontab files etc.
/tmp – directory used by system processes to keep their temporary files. 

Shell

• User interface - Shell provides an interface to the user, wherein the user could issue his commands, and Shell displays output and error messages to the user.
• Command interpreter - Shell accepts command from user, and interprets it to the kernel.
• Command processor - Shell parses the command line arguments, expands the special meaning of meta characters, searches for the command, and if the command is found, then transfers control to the command.
• Programming language - Shell provides a native programming language.

Removing last or more character of string

Want to remove last character from a string in Linux, We can do this with help of SED commands.

Use this

a) Remove last char
sed 's/.$//' input
or
sed 's/.\{1\}$//' input

i.e. -
echo 123456 | sed 's/.$//'
12345

echo 123456 | sed  's/.\{1\}$//'
12345

Search the string in the file : Linux/Unix

Here, I am going to share a small shell utility which will ask source file name and the pattern which need to be searched in file and give the formatted output with the line no with line. You can select the no of lines which display before and after the string.

Useful Perl Scripts : Part-4

For more please visit :  Perl Series

 Array referencing

 

 

Colourizing Unix Programs & Shell Scripts

 

In an ANSI compatible terminal (like xterm, rxvt, konsole ...) text may be shown in colors different from black/white. Here first the colour coding is shown for shell programming .

Each color code is present after an escape sequence. This can be given by

echo  –e "\033[1m Demo Test \033[0m\n";

Interview Questions : Unix/Linux : Part-7

For more : visit here

1. How to display the processes that were run by your user name ?
ps -aef | grep <user_name>

2. Write a command to display all the files recursively with path under current directory?
find . -depth -print

Count User logins in Nix

This script will count the User logins on the Server and if you pass the username to script, It will fetch the last login details of the user.