Next: Presentation/communication Up: AY575 class notes Previous: Introduction to computing hardware

Subsections

Working in a Unix environment

see tabular summary at unix.html

Linux help

man pages: man {command}, or on the web, e.g., linux.die.net/man/

various summaries: http://freeengineer.org/learnUNIXin10minutes.html

http://www.tutorialspoint.com/unix/unix-useful-commands.htm

Basics: directories

File systems: disks (physical units) and partitions (logical units), df

paths, relative and absolute: path denotes location of a file, including directories. Absolute path starts with /, relative path does not, and is evaluated relative to current directory

pwd : working directory

cd : change directory. shorthand notations: (home directory), no argument (home directory), . (current directory) , .. (one directory up), - (previous directory)

creating directory: mkdir, rmdir (note rm -r for recursively removing directories)

separating file name from directory name: basename and dirname

Basics: files

ls : list files, useful options: -a, -l, -F, -r, -t, -R, others

creating files: touch, using output redirection (e.g., echo), mv, cp

permissions: chmod, chown, umask. Unix permissions cover three types of users: user (owner), group, other. Each of these can have read, write, execute permission, expressed in three bits, in order rwx. You can change permissions using characters, e.g., o+w,g-x, or via a permissions mask, e.g., 755 (rwxr-xr-x) or 644 (rw-r-r-)

If a file is an executable (program), you can run it simply by typing its name; either need absolute path, or juse filename if file is in the current PATH environment (see below).

advanced file types: links and pipes. Links come in two types: symbolic (soft) and hard. A symbolic link points to another file, and can work across file systems: the link is distinct from the destination file, and if the destination file is removed the link will be broken. A hard link works only within a file system, and provides an independent link to the same file: if either source or destination is deleted, the file still remains with the linked name. Links are created with the ln command, using -s for a symbolic link.

Pipes allow for interprocess communication through a pseudo-file. They are created using: mkfifo

file types: executable/directories vs regular files vs links

standard file extensions and file naming conventions, e.g. https://www.openoffice.org/dev_docs/source/file_extensions.html https://kb.iu.edu/d/affo

stdin, stdout and stderr: input and output redirection (note shell dependence!)

piping commands: you can direct the output of one command into the input of another using the | character, e.g. ls | more

pattern matching: wildcards (globbing): *, ?, []; regular expressions

locating files:

find ( -name, -type, -mtime, etc.). find will find all files under current directory
locate: uses pre-built system database of files

file contents:

cat
more
less
od
head
tail

You can use `` to put output of a command into another command

$\begin{shaded} \textit{ Be comfortable working with Unix files and directories, including \lq\lq globbing'' (wildcards) and the find command. } \end{shaded}$

Unix useful file manipulation tools

grep: search for specified pattern within files, useful flags, -i (ignore case), -v (search for all lines that do not have the specificed pattern.
sed:
- line-by-line processing using regular expressions
- e.g., remove lines that match specified pattern: sed '/{pattern}/d' {filename}
- e.g., do search and replace: sed 's/{old}/{new}/' {filename}
tr: line-by-line processing with simple replacement. Mostly useful to translating special characters, e.g., Windows files ending with <CR> <LF> to Unix files
sort:
- sorting alphabetically and numerically by column
- use -key=column, e.g. sort -key=2 {filename} will sort alphabetically by column 2
- use -n for numerical sort
paste/join: pasting files together
awk:
- provides a command-line programming interface, convenient for simple operations (but can handle more complex ones)
- e.g., arithmetic based on columns: awk '$1>x {print $2, $3, 5*$4, $6/$3}' {filename}
- can put awk commands in a separate file and use this to execute: awk -f {awkfile} {filename}
- can use awk on variables,e,g.
```
 echo $var | awk '{printf("set %s = %s.new\n)",$1,$1}'
```
diff: show differences between files
wc : counts lines, words, characters

$\begin{shaded} \textit{ Be familiar with Linux commands: wc, grep, sort, diff, a... ...pulation of files. Know how to use awk to modify shell variables. } \end{shaded}$

archiving and compression tools

tar :

bundles multiple files into a single archive
e.g., put files into archive: tar -cvf {archivename} {file}|{directory}
to extract: tar -xvf {archivename} to extract

gzip : compresses files losslessly, creates .gz files

compress : another compression, creates .Z files

xzip: yet another compression, creates .xz files

fpack: tool for compressing FITS files

Unix system information, resources and usage

Determining CPU, disk, and memory capability: /proc/cpuinfo, /proc/meminfo, df, various grapical interface tools (Mac OS X: system_profiler)

(note man command to get information on any command!)

machine types, etc.: uname -a

Determining CPU, disk, and memory usage: top, ps, w, du

w, whoami

hostname

$\begin{shaded} \textit{ Understand how to monitor system resources: CPU usage, disk usage, memory usage. } \end{shaded}$

Unix job control

ps: show current processes

top: show processes sorted by resource usage, updates in real-time

kill: sends signals to processes multiple signals available

CTRL-Z: sends a stop signal to running process

CTRL-

: sends a kill signal to running process

foreground/background: using signals (note signal trapping in code)

cron jobs: allow for job to run on a regular schedule, through use of a crontab (crontab -e to edit).

$\begin{shaded} \textit{ Understand how to put jobs into the background. Understand how to kill jobs that are running. } \end{shaded}$

intermachine communication

ssh: used for secure remote login, but also can be used to execute commands on remote host (e.g., ssh hyades w)

scp: copy files over ssh connected

sftp: open a session with a remote host to enable transfer of one or multiple files/directories

ftp: older, less secure, method for file transfer, but still often used for anonymous ftp, where external users can access a restricted area to grab files, or even to transfer in, if that is enabled. On our cluster, we run an FTP server on astronomy.nmsu.edu: the reserved area is under /home/ftp/pub for outgoing file, /home/ftp/incoming for incoming files.

rsync: used to "sync" files/directories, i.e., transfer only files that differ between systems. Can run locally or between machines using ssh protocol (-e ssh)

Globus?

ssh-keys: provides an alternative to password authentication. Instead of sending a password to a remote machine, a key pair is generated, with a public and private key. The public key is initially transferred to the desired remote server; on subseqeuent connection requests, the server returns the public key to get a match with private key, and if so, the connection is established. Usually, the matching of the keys requires a passphrase as well. One advantage is better security. Another is convenience, as it is possible to register your private key with an "ssh-agent" so that the passphrase is entered only once for a session, allowing remote ssh logins for the rest of the session without needing a password. Commands: ssh-keygen, ssh-agent, ssh-add. Keys are created in /.ssh/; public keys are appended to /.ssh/authorized_keys

$\begin{shaded} \textit{ Understand how to use ssh, scp, and sftp to remotely acc... ...stand how ssh can be used to issue a command on a remote machine. } \end{shaded}$

Unix environment

Unix allows for "environment" variables that are visible to all shells (as opposed to shell variables that are local to a given shell). These are often used for general configuration, and are very useful in the context of software setup and package/data management. This is especially true when you may be setting up an environment on multiple machines, where root directory names differ.

setenv, printenv: commands to set and show environment variables in csh/tcsh.

In bash, set environment variables using: export var=value

some common/useful environment variables: EDITOR/VISUAL, DISPLAY, SSH_ASKPASS, SHELL, PATH

$\begin{shaded} \textit{ Understand what environment variables are and how to set and display them. } \end{shaded}$

Editors

An editor is a basic tool used for nearly all computing tasks, so it is worthwhile to have a strong command of the editor that you choose to use.

Given today's networked environment, it is very likely that you will need at some point to edit files on remote machines. Working with editors that open graphical windows can become a significant challenge when working over the network, and for this reason, I would strongly discourage them.

The historical editor associated with Unix is vi/vim, and this is still in widespread use. Another extremely widespread editor is emacs, which can be used within a terminal window using the -nw option. Another terminal based window that is sometimes installed is nano.

With whatever editor you use, you should be able to

efficiently, move the cursor within files. Goto specific line numbers
Search for text strings
Search for text string and replace with alternate string
delete line or blocks of lines with single command
be able to efficiently move blocks of text around, e.g. copy and paste

$\begin{shaded} \textit{ Be extremely comfortable with editing files. Strongly re... ...ble to editing files on a remote machine within a login terminal! } \end{shaded}$

Unix shells and shell scripting

startup: .cshrc, .bashrc, with useful customization commands:

alias definitions
environment varibles
set path for searching for executables using PATH environment variable (printenv PATH to see what path is)

which command: find full directory path of a given command (or find out whether the command is in the path)

history command: lists past N commands

command completion (TAB-completion) and recall (!N executes command N; !x executes last command that begins with x.

When to use shell scripts? Often most convenient for operations have to do with files and simple file modification. Not usually a good choice for numerical work!

running command files: source vs starting new shell (#!), file permissions (a file must have executable permission for it to be identified in the path as a command).

scripting:

variables: reference using $varname
command line arguments: $0 is first word (the command), $1 the first argument, etc.; $* refers to all arguments (after command), which is very useful, e.g., in conjunction with for loops (below) to loop over all command line arguments
conditionals: == (equal), != (not equal), && (and), || (or)
looping

see unix.html for syntax of shell commands in bash and tcsh

see PDF presentation from Utah CHPC on shell scripts

csh:

set
if / then / else
- often used with file inquiry conditionals (e.g. -e tests for existence)
foreach /end
while / end

example

bsyn &
set bsynjob = $!
set tmax = 300
set runtime = `ps -q $bsynjob -o cputime | tail -1 | awk -F: '{print ($1*3600)+($2*60)+$3}'`
while ( $runtime < $tmax )
  sleep 2
  set runtime = `ps -q $bsynjob -o cputime | tail -1 | awk -F: '{print ($1*3600)+($2*60)+$3}'`
  if ( `ps -p $bsynjob -o comm=` == "" ) then
    echo process done, exiting!
    exit
  endif
end
echo expired, killing job
kill $bsynjob

$\begin{shaded} \textit{ Understand the basics of shells and know what shell you ... ...and how to specify input to a shell script from the command line. } \end{shaded}$

remote & virtual desktops

e.g., VNC: vncserver and vncviewer.

Next: Presentation/communication Up: AY575 class notes Previous: Introduction to computing hardware

Jon Holtzman 2015-12-11