1 Hackinars in BioinformaticsUnix basics and usage of Computerome Erland Hochheim DTU Systems Biology Center for Biological Sequence Analysis

2 Log on the Danish National Life Sciences Supercomputer – akaLog on the Danish National Life Sciences Supercomputer – aka. Computerome Prerequisites Some sort of personal computer SSH client Linux and Mac have built-in terminal programs which support SSH Terminal, Xterm, etc. Windows does not… PuTTY is a good choice for most people MobaXterm is widely used, but 2-factor authentication can be a bit tricky How to do Use User-ID and Password provided from DTU Login to homesystem$ ssh ~]$

3 Login to Computerome

4 Be familiar with the strengths and weaknesses of each computer including their own laptop.Easy to carry Provides access to resources elsewhere Computerome 500+ nodes, cores 3PB available storage Somewhat difficult to carry Provides massive resources from (almost) everywhere

5 Exercise: Open terminal and login to ComputeromeFind the terminal program on your computer Linux and Mac should be built-in Windows might want to download PuTTY Login to Computerome

6 Basic Linux/UNIX commandsManual pages: man Help flag: --help, -h List files and directories: ls List, long form: ls -l List all: ls -a Change directory: cd Move files and directories: mv Copy file and directories: cp Copy recursively: cp -r Copy, and protect permissions: cp -p Remove files: rm Remove recursively: rm -r Force remove: rm -f Create directory: mkdir Create structure directory with parents: mkdir -p

7 man command – example ‘man ls’

8 --help flag

9 ls

10 cd, cp, mv, rm

11 Exercise: Experiment with commands‘man ls’ - see what ls command can do Try some different flags to ls: ls -l ls -a others… Change directory: cd Get help on mv command mv --help mv -h Copy file and directories: cp Copy recursively: cp -r Copy, and protect permissions: cp -p Remove files: rm Remove recursively: rm -r Force remove: rm -f Create directory: mkdir Create structure directory with parents: mkdir -p

12 Commands for viewing filesConcatenate files: cat Output first part of file: head Output last part of file: tail View contents of file: more “Opposite of more”: less

13 Viewing - cat

14 Viewing - head, tail

15 Viewing - more, less, example ‘less outfile’

16 Pipes and redirects | Pipe, example: date | awk '{print $3,$2,$6}'> [FILE] Overwrite file, example: echo Hello World > hello.txt >> [FILE] Append to file, example: echo Hello CBS >> hello.txt

17 Pipes and redirects - |, >, >>

18 Editors on Linux/UNIX machinesvi and vim Programmers text editors nedit NEdit is a standard GUI (Graphical User Interface) style text editor gedit Graphical text editor for Gnome

19 Editors

20 Editing - vim hello.txt

21

22 Basic file analysis Report lines, words and bytes: wcSort contents: sort

23 Basic analysis - wc

24 Basic analysis - sort

25 File and directory permissionsls –l to see permissions on files drwxrwxr-x 3 erhh erhh Mar 12 09:00 course -rw erhh erhh Mar 10 12:48 course/STDIN.o3929 -rw-rw-r-- 1 erhh erhh Mar 10 12:55 course/tmpdir/Graph ugo User Group Owner rwx Read Write eXecute Change owner: chown Change group: chgrp Change permissions: chmod

26 Permissions and their consequences

27 File transfer between computersCreate a portable archive: tar Secure copy: scp Get stuff from the Web wget curl

28 Transfer - tar, scp

29 Transfer - wget, curl

30

31 Computerome Wiki Main information source for all things Computerome A living document Corrections to be mailed to Special page for Tips and Tricks

32 Module environment module avail module load module initaddReference:

33 module avail, module load, module initadd

34 Running jobs on ComputeromeJobs are run in a Batch environment Moab scheduler Torque resource manager Reference:

35 Job submission qsub msub xqsub xmsubReference:

36 Monitoring jobs qstat showq checkjob pestatReference:

37 Simple job submission with qsub

38 Monitoring - watch showq

39 Monitoring - showq -c

40 Monitoring - qstat

41 Monitoring - checkjob

42 Monitoring - pestat

43 Job control canceljob showstart tracejobReference:

44 Job control – canceljob, showstart

45 Job control - tracejob

46 Job control – checkjob –v –v

47 Exercise with Velvet Login to ComputeromeGo to your subdirectory in hackinars project directory /home/projects/pr_hackinars/people/ Copy data/Strain_H fastq to you own directory $ cp /home/projects/pr_hackinars/data/Strain_H fastq .

48 Exercise with Velvet – cont.Create a basic jobfile (script) $ vi velvet.sh #!/bin/bash velveth tmpdir 21 -fastq Strain_H fastq velvetg tmpdir Make it executable $ chmod +x velvet.sh

49 Exercise with Velvet – cont.Load modules $ module load tools $ module load moab torque $ module load velvet/1.2.10 Run velvet job using xmsub $ xmsub -W group_list=pr_hackinars -A pr_hackinars \ > -l nodes=1:ppn=2,mem=100m,walltime=3600 \ > -V -d $PWD -ro outfile -re errorfile -de ./velvet.sh Watch job $ showq -u or watch showq -u $ qstat -u or watch qstat -u $ checkjob $ tracejob $ cat outfile $ cat errorfile $ cat tmpdir/Log

50 Exercise with Velvet – cont.Extend existing jobfile (script) $ vi velvet.sh #!/bin/sh ### Account information #PBS -W group_list=pr_hackinars -A pr_hackinars ### Job name #PBS -N velvet_test ### Output files #PBS -e errorfile #PBS -o outfile ### Number of nodes #PBS -l nodes=1:ppn=2 ### Memory #PBS -l mem=100m ### Requesting time (format dd:hh:mm:ss or just number of seconds) #PBS -l walltime=1:00:00 ### Script goes below here # Go to the directory from where the job was submitted (initial directory is $HOME) echo Working directory is $PBS_O_WORKDIR cd $PBS_O_WORKDIR module load tools module load velvet/ velveth tmpdir 21 -fastq Strain_H fastq velvetg tmpdir Run job $ qsub ./velvet.sh

51 BONUS: Estimating number of coresgzip gunzip gzip PERL Script gzip