MD5 checksum on a file in Linux.

Installing MD5SUM package:

$ apt-get update
$ apt-get install coreutils

Generate checksum file
$ md5sum transaction.log > MDSSS

Check the checksum:

$ cat MDSSS
ae43fda27bf612e2d7128c3072220078 transaction.log

mount windows directory to linux

$cd /mnt
$sudo mkdir windrive
$sudo mount -t cifs _windows_drive_path_ -o username=_user-name_,password=_password_ /mnt/windrive/

To see the contents of vdrive:

$ls -la /mnt/windrive

Recursively find all files and convert to unix format (excluding those in svn dirs)

Ref : http://snippets.dzone.com/posts/show/108

find . -type f \! -path "*svn*" -exec dos2unix {} \;

After svn checkout you should run this command in your local directory.

Command line for and while loop.. Shell Linux

Command line while loop:
$while [ 1 ]; do traceroute www.slac.stanford.edu; done;

Command line for loop:
$for x in 1 2 3; do ls -l; done

Enjoy :)

end-of-line character conversion: unix2mac dos2unix mac2unix

All these scripts are one liners:


"mac2unix"
cat $1 | tr '\r' '\n'

"unix2mac"
cat $1 | tr '\n' '\r'

"dos2unix"
cat $1 | tr -d '\r'

NOTE: "tr" (translate character) utility.

Linux date command

$ date '+%d-%m-%y %H:%M:%S'
04-02-11 14:16:57

Enjoy!

Problem running shell script in Cygwin

If you are getting these kind of error in cygwin:

# ./compare.sh
./compare.sh: line 2: $'\r': command not found
./compare.sh: line 5: $'\r': command not found
./compare.sh: line 7: $'\r': command not found
./compare.sh: line 9: $'\r': command not found
' for reading (No such file or directory) `./conf/compare.conf
' for reading (No such file or directory) `./conf/compare.conf
' for reading (No such file or directory) `./conf/compare.conf
' for reading (No such file or directory) `./conf/compare.conf
./compare.sh: line 14: $'\r': command not found

Its because you have to convert this file into a unix format. So use the following command:

# dos2unix compare.sh
compare.sh: done.

This will solve this problem

Using Curl to connect to https site to download and Extract a tar file...

Here is the command:

curl -k -u_username_ -O _URL_ | tar -xvf

Example:

curl -k -uhellouser -O https://192.168.2.3:8443/svn/project/V5.0.1.tar.gz | tar -xvf

Using Curl to download and Extract a tar file...

Here is the command:

$curl -k -O _Tar File Path_ | tar -xvf

Example:

$curl -k -O http://192.168.4.189:8443/svn/testproject/releases/V1.0.1.x.tar.gz | tar -xvf

Setting Up an MPI Cluster on Ubuntu...

Setting up a MPI cluster on Ubuntu involves the following steps:

1. Install OpenMPI on all machines.
$sudo apt-get install libopenmpi-dev openmpi-bin openmpi-doc

2. Installing openSSH on all machines.
$sudo apt-get install openssh-server openssh-client

3. Select one Master machine say M and others will be slaves say S.

4. Now we need to make sure M can logon to all S machines with out a password.

Create a user with same name on all machines. Say 'fawad'.

Login on M using 'fawad' and generate public/private key pair.

$ssh-keygen -t dsa

This command will generate id_dsa and id_dsa.pub files in ~/.ssh folder. id_dsa is the private key and id_dsa.pub is the public key. Now we need to keep the private key and copy the public key on to all the slaves machines.

5. Use scp to copy id_dsa.pub in the home directory of fawad on all S machines.

6. On the S machines do:

Login as fawad.
$cd ~/.ssh
$ cat id_dsa.pub >> authorized_keys

7. Now the M machine will be able to logon to all machines without password.

8. On the M machines.
Write a MPI code and save it. In my case i used the following code and saved it in hello_nodes.c

--------------hello_nodes.c--------------------

/*

A simle example program using MPI - Hello World

The program consists of one receiver process and N-1 sender
processes. The sender processes send a message consisting
of their hostname to the receiver. The receiver process prints
out the values it receives in the messages.

Compile the program with 'mpicc hello_nodes.c -o hello_nodes'
To run the program on four processors do 'mpiexec -n 4 ./hello'

*/

#include
#include
#include

int main(int argc, char *argv[]) {
const int tag = 42; /* Message tag */
int id, ntasks, source_id, err, i;
MPI_Status status;
char msg[80]; /* Message array */

err = MPI_Init(&argc, &argv); /* Initialize MPI */
if (err != MPI_SUCCESS) {
printf("MPI_init failed!\n");
exit(1);
}

err = MPI_Comm_size(MPI_COMM_WORLD, &ntasks); /* Get nr of tasks */
if (err != MPI_SUCCESS) {
printf("MPI_Comm_size failed!\n");
exit(1);
}

err = MPI_Comm_rank(MPI_COMM_WORLD, &id); /* Get id of this process */
if (err != MPI_SUCCESS) {
printf("MPI_Comm_rank failed!\n");
exit(1);
}

/* Check that we run on at least two processors */
if (ntasks < 2) {
printf("You have to use at least 2 processors to run this program\n");
MPI_Finalize(); /* Quit if there is only one processor */
exit(0);
}

/* Process 0 (the receiver) does this */
if (id == 0) {
int length;
MPI_Get_processor_name(msg, &length); /* Get name of this processor */
printf("Hello World from process %d running on %s\n", id, msg);
for (i=1; i err = MPI_Recv(msg, 80, MPI_CHAR, MPI_ANY_SOURCE, tag, MPI_COMM_WORLD, \
&status); /* Receive a message from any sender */
if (err != MPI_SUCCESS) {
printf("Error in MPI_Recv!\n");
exit(1);
}
source_id = status.MPI_SOURCE; /* Get id of sender */
printf("Hello World from process %d running on %s\n", source_id, msg);
}
}

/* Processes 1 to N-1 (the senders) do this */
else {
int length;
MPI_Get_processor_name(msg, &length); /* Get name of this processor */
err = MPI_Send(msg, length+1, MPI_CHAR, 0, tag, MPI_COMM_WORLD);
if (err != MPI_SUCCESS) {
printf("Process %i: Error in MPI_Send!\n", id);
exit(1);
}
}

err = MPI_Finalize(); /* Terminate MPI */
if (err != MPI_SUCCESS) {
printf("Error in MPI_Finalize!\n");
exit(1);
}
if (id==0) printf("Ready\n");
exit(0);
}


-------------------------------------------

9. Compile the code.
$ mpicc hello_nodes.c -o hello_nodes

10. Run the code on the same machine.
fawad@fawad-virtual-machine:~$ mpirun -np 4 hello_nodes
Hello World from process 0 running on fawad-virtual-machine
Hello World from process 2 running on fawad-virtual-machine
Hello World from process 1 running on fawad-virtual-machine
Hello World from process 3 running on fawad-virtual-machine
Ready

11. Create a file called hostfile, which contains the slave computers in the MPI cluster:

fawad@fawad-virtual-machine:~$ cat hostfile
172.16.92.130
172.16.92.129

12. Copy the source code hello_mpi to all slave machines using scp.

13. Then run the following command to run the job on all slave machines:

fawad@fawad-virtual-machine:~$ mpirun --hostfile hostfile -np 10 hello_nodes
Hello World from process 0 running on ubuntu
Hello World from process 6 running on ubuntu
Hello World from process 4 running on ubuntu
Hello World from process 8 running on ubuntu
Hello World from process 3 running on fawad-virtual-machine
Hello World from process 2 running on ubuntu
Hello World from process 9 running on fawad-virtual-machine
Hello World from process 5 running on fawad-virtual-machine
Hello World from process 7 running on fawad-virtual-machine
Hello World from process 1 running on fawad-virtual-machine
Ready


DONE :).

Now the next step is that we need to try creating a NFS so that we do not have to copy the binary file all the time.

Best MPI Books i would recommend:

SED : Tutorial 1

REF: http://www.grymoire.com/Unix/Sed.html

$echo day | sed s/day/night/
Output: night

$echo Sunday | sed 's/day/night/'
Output: Sunnight

slash/underscore/colon as a delimiter
---------------

$echo "/home/fawad/sed" | sed 's/\/home\/fawad\/sed/\/home\/nazir\/fawad/'
Output: /home/nazir/fawad

$echo "/home/fawad/sed" | sed 's_/home/fawad/sed_/home/nazir/fawad_'
Output: /home/nazir/fawad

$echo "/home/fawad/sed" | sed 's:/home/fawad/sed:/home/nazir/fawad:'
Output: /home/nazir/fawad

Using & as the matched string
-----------------------
$echo "123 Fawad Nazir" | sed 's/F[a-z]*/(&)/'
Output: 123 (Fawad) Nazir

The values is saved in &. So now you can use it in different ways...

$echo "123 Fawad Nazir" | sed 's/F[a-z]*/(&) [&] _&_ *&*/'
Output: 123 (Fawad) [Fawad] _Fawad_ *Fawad* Nazir

Fun: Only Print a and nothing else
$echo "Fawad Nazir Is a Great Man hahahaha" | sed s/[^a]/\ /g
Output: a a a a a a a a a a

Using Mail Server on Linux...

Send Email:

$mailx -s "Email Sibject" User_Name@Host_Name < Message_File_Name

Check new emails:
$mail

View all emails:
Goto Home Directoty $cd ~
Then, you should find a mbox folder
$view mbox

Installing and updating the kernel to Linux-2.6.16 (Ubuntu – IDE)

1. Install Ubuntu 5.10.

2. To setup the root password, reboot the machine and press ESc to go into grub mode.

To to the recovery mode of Linux and press enter.

You will get to # prompt.

Type #passwd root, enter new password and reboot #init 6.

3. Goto System -> Administrator -> Networking and enable the eth0 card for networking support.

4. Goto System ->Administrator -> Synaptic Package Manager

5. Goto Settings -> Repositories. Add Community Maintained (Universe) and press OK.

6. Press “Reload” to update the package list.

7. Goto Terminal: and run all the following commands:

$sudo apt-get update
$sudo apt-get install build-essential
$sudo apt-get install kernel-package
$sudo apt-get install gcc
$sudo apt-get install libncurses5
$sudo apt-get install libncurses5-dev
$sudo apt-get install libqt3-mt-dev

8. Now configure ftp server to transfer new linux 2.6.16 files.

9. I will go for vsftp. You can also configure ftpd, ws-ftpd or proftpd etc.

To install vsftpd:

$sudo apt-get install vsftpd, This will install vsftpd server and start it.

Now we need to make some modifications to the configuration file.

comment #anonymous_enable=YES

Uncomment local_enable=YES

Uncomment write_enable=YES

Save this file and restart vsftpd by running this command. $sudo /etc/init.d/vsftpd restart

10. Upload linux-2.6.16.tar using ftp.

11. copy linux-2.6.16.tar to /usr/src/ folder.

12. cd /usr/src

13. $sudo tar –bzip2 -xvf linux-2.6.12.tar.bz2

14. $sudo ln -s /usr/src/linux-2.6.12 /usr/src/linux

15. Run $sudo make menuconfig or $sudo make xconfig to configure your new kernel.

16. $sudo make-kpkg clean

17. $sudo make-kpkg -initrd –append-to-version=-custom kernel_image modules_image kernel_headers

18. $sudo dpkg -i kernel-image-2.6.16-custom_10.00.Custom_i386.deb
19. $sudo dpkg -i kernel-headers-2.6.16-custom_10.00.Custom_i386.deb

20. Reboot the system $sudo init 6.

Setting up a wireless Testbed…(Host-AP drivers)

S#1: Machine A Master Mode and Machine B Managed Mode.

Machine A:

$sudo iwconfig wlan0 mode Master
$sudo iwconfig wlan0 essid “Hello-1″

Set the Static IP Address: 10.10.5.2 Mask: 255.0.0.0

root@Ubuntu-01:/etc/network# iwconfig wlan0
Warning: Driver for device wlan0 has been compiled with version 19
of Wireless Extension, while this program supports up to version 18.
Some things may be broken…

wlan0 IEEE 802.11b ESSID:”Hello-1″
Mode:Master Access Point: 00:00:00:00:00:00 Bit Rate:11 Mb/s
Sensitivity=1/3
Retry min limit:8 RTS thr:off Fragment thr:off
Encryption key:off
Power Management:off
Link Quality:0 Signal level:0 Noise level:0
Rx invalid nwid:0 Rx invalid crypt:0 Rx invalid frag:0
Tx excessive retries:0 Invalid misc:0 Missed beacon:0

root@Ubuntu-01:/etc/network# ifconfig wlan0
wlan0 Link encap:Ethernet HWaddr 00:02:6F:34:0B:79
inet addr:10.10.5.2 Bcast:10.255.255.255 Mask:255.0.0.0
inet6 addr: fe80::202:6fff:fe34:b79/64 Scope:Link
UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
RX packets:69 errors:0 dropped:1 overruns:0 frame:0
TX packets:63 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:0
RX bytes:4970 (4.8 KiB) TX bytes:5220 (5.0 KiB)
Interrupt:3 Base address:0×3100

Machine B:

$sudo iwconfig wlan0 mode Managed

Set the Static IP Address: 10.10.5.3 Mask: 255.0.0.0

Important : Gateway wlan0
Connect to EssID : Hello-1

fawad@nicta123:~$ iwconfig wlan0
Warning: Driver for device wlan0 has been compiled with version 19
of Wireless Extension, while this program supports up to version 18.
Some things may be broken…

wlan0 IEEE 802.11b ESSID:”Hello-1″
Mode:Managed Frequency:2.422 GHz Access Point: 00:02:6F:34:0B:79
Bit Rate:2 Mb/s Sensitivity=1/3
Retry min limit:8 RTS thr:off Fragment thr:off
Power Management:off
Link Quality=56/70 Signal level=-19 dBm Noise level=-75 dBm
Rx invalid nwid:0 Rx invalid crypt:3 Rx invalid frag:0
Tx excessive retries:30 Invalid misc:12165 Missed beacon:0

fawad@nicta123:~$ ifconfig wlan0
wlan0 Link encap:Ethernet HWaddr 00:60:B3:29:92:EE
inet addr:10.10.5.3 Bcast:10.255.255.255 Mask:255.0.0.0
BROADCAST MULTICAST MTU:1500 Metric:1
RX packets:8799 errors:0 dropped:0 overruns:0 frame:0
TX packets:12472 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:0
RX bytes:853532 (833.5 KiB) TX bytes:1505635 (1.4 MiB)
Interrupt:3 Base address:0xe100

Now try pingning from the both ends. I think they should work.

S#2: Both Machine A & B in Ad-Hoc Mode.

Now in this case we have to see that in order for the machines to work in Ad-Hoc mode they should have same essid and channel.

Now lets configure machine A:

$sudo iwconfig wlan0 mode Ad-Hoc
$sudo iwconfig wlan0 channel 3
$sudo iwconfig wlan0 essid “Home”

fawad@Ubuntu-01:~$ iwconfig wlan0
Warning: Driver for device wlan0 has been compiled with version 19
of Wireless Extension, while this program supports up to version 18.
Some things may be broken…

wlan0 IEEE 802.11b ESSID:”Home”
Mode:Ad-Hoc Frequency:2.422 GHz Cell: 02:60:1B:27:92:EE
Bit Rate:11 Mb/s Sensitivity=1/3
Retry min limit:8 RTS thr:off Fragment thr:off
Power Management:off
Link Quality=56/70 Signal level=-40 dBm Noise level=-96 dBm
Rx invalid nwid:0 Rx invalid crypt:0 Rx invalid frag:0
Tx excessive retries:22 Invalid misc:40 Missed beacon:0

fawad@Ubuntu-01:~$ ifconfig wlan0
wlan0 Link encap:Ethernet HWaddr 00:02:6F:34:0B:79
inet addr:10.10.5.2 Bcast:10.255.255.255 Mask:255.0.0.0
inet6 addr: fe80::202:6fff:fe34:b79/64 Scope:Link
UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
RX packets:144 errors:0 dropped:2 overruns:0 frame:0
TX packets:163 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:0
RX bytes:13304 (12.9 KiB) TX bytes:17560 (17.1 KiB)
Interrupt:3 Base address:0×3100

On Machine B:

$sudo iwconfig wlan0 mode Ad-Hoc
$sudo iwconfig wlan0 channel 3
$sudo iwconfig wlan0 essid “Home”

fawad@nicta123:~$ iwconfig wlan0
Warning: Driver for device wlan0 has been compiled with version 19
of Wireless Extension, while this program supports up to version 18.
Some things may be broken…

wlan0 IEEE 802.11b ESSID:”Home”
Mode:Ad-Hoc Frequency:2.422 GHz Cell: 02:60:1B:27:92:EE
Bit Rate:11 Mb/s Sensitivity=1/3
Retry min limit:8 RTS thr:off Fragment thr:off
Power Management:off
Link Quality=55/70 Signal level=-18 dBm Noise level=-73 dBm
Rx invalid nwid:0 Rx invalid crypt:0 Rx invalid frag:0
Tx excessive retries:52 Invalid misc:984 Missed beacon:0

fawad@nicta123:~$ ifconfig wlan0
wlan0 Link encap:Ethernet HWaddr 00:60:B3:29:92:EE
inet addr:10.10.5.3 Bcast:10.255.255.255 Mask:255.0.0.0
inet6 addr: fe80::260:b3ff:fe29:92ee/64 Scope:Link
UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
RX packets:144 errors:0 dropped:0 overruns:0 frame:0
TX packets:271 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:0
RX bytes:13300 (12.9 KiB) TX bytes:24228 (23.6 KiB)
Interrupt:3 Base address:0xe100

Now try pinging each of the hosts i guess they should work.

Linux Commands…

Web Page With All Linux Commands…

Command to find what all files are open: $lsof
Command to see what is done when a command is executed: $strace , e.g. $strace ls

Command to change the file system of a disk
$sudo mkfs.ext3 -cv -L usbdisk2 /dev/sdc1

dpkg – a medium-level package manager for Debian
————————————————

This is like an rpm for Red-hat linux.

Man Page: http://www.fifi.org/cgi-bin/man2html/usr/share/man/man8/dpkg.8.gz

To list packages related to the editor vi:
dpkg -l ‘*vi*’

To search the listing of packages yourself:
less /var/lib/dpkg/available

To remove an installed elvis package:
dpkg -r elvis

To install a package, you first need to find it in an archive or CDROM. The “available” file shows that the vim package is in section “editors”:
cd /cdrom/hamm/hamm/binary/editors dpkg -i vim_4.5-3.deb

How to load Kernel modules at boot time…

fawad@Ubuntu-01:~$ more /etc/modules
# /etc/modules: kernel modules to load at boot time.
#
# This file contains the names of kernel modules that should be loaded
# at boot time, one per line. Lines beginning with “#” are ignored.

lp
mousedev
psmouse

How to Blacklist a module…

fawad@Ubuntu-01:/etc/hotplug$ echo ‘blacklist orinoco’ | sudo tee -a
/etc/modprobe.d/my_blacklist
blacklist orinoco
fawad@Ubuntu-01:/etc/hotplug$ echo ‘blacklist orinoco_cs’ | sudo tee
-a /etc/modprobe.d/my_blacklist
blacklist orinoco_cs

Where orinoco_cs & orinoco are the modules names.

The above tutorial will blacklist the kernel modules at the starup. But you modules might be loaded which anything is hotplugged into your computer.

For that i could not actually come-up with a solution but i think the way is to intsert modules names in the file:

root@Ubuntu-01:/etc/network# more /etc/hotplug/blacklist

JBoss Remote Debugging... LINUX.

Normally people modify the run.sh file and add the following line:

JAVA_OPTS=%JAVA_OPTS% -Xdebug -Xnoagent -Xrunjdwp:transport=dt_socket,address=8787,server=y,suspend=n

I would recommend not to modify the run.sh file, rather just uncomment
the following line in the run.conf file in the same folder:

# Sample JPDA settings for remote socket debuging
JAVA_OPTS="$JAVA_OPTS -Xdebug -Xrunjdwp:transport=dt_socket,address=8787,server=y,suspend=y"

SIMPLE :).

Experience While Copying a 42GB file into a Seagate USB Hardrive in Ubuntu Linux…

First of all i will split the file into 10GB pieces. The file name is: enwiki-20070908-stub-meta-history.xml

fawad@crete:~/wiki$ split -b 4000m enwiki-20070908-stub-meta-history.xml

Now i will copy a piece by piece into my seagate external hardrive. I am copying this file from a remote server to my machine

root@fawad-laptop:/home/fawad# scp fawad@crete.ex.nii.ac.jp:/home/fawad/wiki/enwiki-20070908-stub-meta-history.xml /media/usb0/niidata/split/
fawad@crete.ex.nii.ac.jp’s password:
enwiki-20070908-stub-meta-history.xml 10% 4091MB 10.5MB/s 57:03 ETAFile size limit exceeded (core dumped)
This means only files less than 4GB are supported. So i again spilled the files to 4GB file each.

Another important thing to note is that i had to use root login to copy file to my external seagate USB hardrive.

Now once i have splited the files into 4GB chunks. Now use the following command to copy all the files from this folder to one of the folder in the seagate harddrive.

root@fawad-laptop:/home/fawad# scp -r fawad@crete.ex.nii.ac.jp:/home/fawad/wiki/wikidata/ /media/usb0/niidata/wikifawad@crete.ex.nii.ac.jp’s password:
.nfs0000000000d5005100000001 100% 435MB 10.6MB/s 00:41
xad 27% 1101MB 9.5MB/s 05:05 ETARead from remote host crete.ex.nii.ac.jp: Connection reset by peer
xad 100% 4000MB 9.8MB/s 06:49
xag 100% 1233MB 9.8MB/s 02:06
xac 38% 1522MB 10.6MB/s 03:53 ETAh
xac 43% 1734MB 11.1MB/s 03:25 ETA
xac 100% 4000MB 10.1MB/s 06:38
xaf 100% 4000MB 10.2MB/s 06:32
xae 100% 4000MB 10.5MB/s 06:22
xaa 100% 4000MB 9.9MB/s 06:46
xab 39% 1573MB 9.9MB/s 04:05 ETARead from remote host crete.ex.nii.ac.jp: Connection reset by peer
xab 100% 4000MB 9.6MB/s 06:57

Installing MySql on Ubuntu Linux

fawad@fawad-desktop:~$ sudo apt-get install mysql-server

fawad@fawad-desktop:~$ mysqladmin -u root password 123456

fawad@fawad-desktop:~$ mysql -u root -p

mysql> CREATE DATABASE realitymining;
Query OK, 1 row affected (0.00 sec)