Unix

The operating system (OS) of a computer is the system software that is responsible for file management (creating, copying, modifying, destroying files), process management, scheduling access to resources such as CPU cycles and RAM, and monitoring/configuring itself as well as the hardware. Users tell the OS what to do with the computer's hardware via a user interface (usually a "shell"; see below). These days, there are two main operating systems: Windows and Unix-like operating systems. Most astronomers work on computers that use a Unix-like operating system, at least for data analysis (maybe not for basic office-like stuff, paper writing). Mac OS X is a Unix-like operating system.

Graphical (file manager windows) vs text interface: most astronomers use text interfaces such as:

Terminal application (Mac OS X)
xterm, konsole, gnome-terminal (Unix/Linux)

File organization:

files and directories
- File names (name.ext) : in astronomy, people generally do NOT use spaces in names, and keep file names meaningful, but compact!
- File extensions: various conventions, e.g. .txt, .dat, .fits, .tex, .html
- directories: relative paths (don't start with /) and absolute paths (start with /)
- some simple directory shortcuts: . means current directory, .. means one directory up, ~ means home directory, - often means the previous directory
disks and partitions
- local and remote disks: make sure you know which is which!

Unix file/directory-related commands (italics mean you fill in desired quantity, square brackets mean optional parameter):

pwd : (print working directory) shows what directory you are in
ls [file(s)] : lists files in current directory (or recursively with -R)
cat/more/less file(s): show contents of a file
mkdir dir: make a subdirectory under current directory
cd path : change directory
cp source destination : copy a file
mv source destination : move a file
rename : rename lots of files in a similar way with a single command
df : show available partitions including how much free disk space on each
du : show how much space is taken up by a file, or by a directory and all its subdirectories

Unix shortcuts:

History and command line recall with arrow keys, history, !
tab completion
Wildcards: *, ?, [], regular expressions
Special files: links and symbolic links: ln -s destination linkname

Unix help/references:

man command : get information about specified command, including command options
Web interface to man pages
Unix command summary (Holtzman)
Unix commands/examples (Kristian)
http://www.tutorialspoint.com/unix/unix-useful-commands.htm
http://freeengineer.org/learnUNIXin10minutes.html

Exercise 1:

log into astronomy: ssh -Y username@astronomy.nmsu.edu
what directory are you in (pwd)?
create a subdirectory called ~/bin
create a file and copy it into that subdirectory
figure out what filesystem your directory is on. How much space is currently available on that file system?
figure out what other filesystems are accessible. Are these on the local computer or a remote computer?
use man pages to figure out how to get df to show you free disk space in ``human-readable" format

Creating/editing files:

warning: graphical editors (e.g., gedit, kwrite) may not be a good choice: problems when working over a network (common!)
emacs : powerful common text editor. If you are editing a file on a remote machine, then avoid the need for X11 forwarding by running in a local window using: emacs -nw
vi : historical main Unix editor. This is still what many use because it is powerful and universally available. Additionally, if you are editing a file on a remote machine, using vi does not require X11 forwarding.
nano : simple text editor. Less functional than vi, but does not require graphics forwarding and all the basic commands for writing, saving, and exiting are listed at the base of the screen, making it easier to ramp up.
Textedit (Mac OS X); Notepad (Windows); gedit (Linux), kwrite (Linux) : not generally used (not available on Unix machines)

Looking at files

cat/more/less file: print file contents to screen, all at once, or by screenful. Less is better because it supports searching.
head/tail [-n number] file: display first or last (n) lines of file, note tail -f displays lines as they are added
wc file: counts lines, words, characters in a file

Practice:

If you are not already logged into astronomy, do that now
create a file with 10 million random characters using "base64 /dev/urandom |head -c 10000000 > file.txt"
output the first 3 lines using head -3
output the final 3 lines using tail -3
count how many lines there are using wc -l
count how many of those lines contain the string "she" appears using grep -c

Running programs:

Programs are run by typing the name of the program
the location(s) through which directories are searched to find the program is determined by the users PATH
- the path is stored in a special UNIX kind of variable called an environment variable
- you can see what this is set to using: printenv PATH
- you can set this using: setenv PATH dir1:dir2:dir3:...
- you can add to this using: setenv PATH ${PATH}:dir1:dir2:dir3:...
- path is searched in order
- you can see what file/program is being run using the command: which progname

File permissions:

Unix has file permissions, separately for three different categories: owner, group, world
Permissions are read, write(/delete), execute
ls -l (long listing) shows permissions
Program file must have execute permission to be found in path
Directories must have execute permission to see contents
Permissions can be specified by character permission code, or by octal code (4=read, 2=write, 1=execute)
default permissions usually rw-r--r-- for files, rwxr-xr-x for directories
chmod command to change permissions, e.g.
- chmod g+w file (changes file to have group write permission)
- chmod 660 file (changes file to have rw-rw----)

Signals/job control:

Unix has signals that can be used to control processes:
- CTRL-Z : suspend process
- CTRL-C : interrupt process
- CTRL-\ : kill process
Can also send signal to a running process: kill processid
show running processes: ps [aux], also top for listing processes in order of resource usage (good to see what's going on if things are slow!)
Running jobs in the background:
- add & to end of command line to start command in the background
- start job in foreground, CTRL-Z, then enter: bg (to put in background, fg will return job to foreground)
- Unix-like operating systems can have "screen" application, which is a persistent shell that you can open and close. This means, for example, that the time-consuming application that you're running on a remote resource won't die when your connection is lost.

Let's do some exercises with starting and stopping jobs: Exercise 2

More UNIX utility commands:

grep string file(s) : search for string in files
- grep -i error *.out (finds all occurances of "error", ignoring case, in all files that end in .out)
find path : find files in a directory hierarchy, optionally filtering by name, type, size, modification time, etc.
awk : process files line-by-line, with options for column arithmetic/manipulation, with possible conditional expression
- awk '{print $3, $5, $7, $10/2, $8*$9}' file (no conditional means all lines, prints 3rd, 5th and 7th columns, half the tenth, product of column 8 and 9, ...)
- awk '$8>0' file
- awk '$8<0 {printf("%8.2f\n",$2*$3)}' file
sed : line-by-line scripting editor
- sed 's/hello/goodbye/g' file (substitutes all occurances of hello with goodbye)
sort [-n] [--key={columnnumber}] : sorts files on a specifed column value (-n is numerical sort, else alphabetical)
diff file1 file2: show line-by-line differences between files

Unix redirection and pipes

Input and output redirection: > (output redirection to new file), >> (output redirection appends to existing file), < (input redirection from file)
stdout and stderr
Piping I/O between commands: |
- kill -9 `ps aux | grep idl | awk '{print $2}'`

Exercise 3: Create a file with some names and numbers, and practice sorting and awking! Exercise 3

Intermachine communication:

ssh, scp, sftp: secure communication between machines
- ssh: remote login: ssh user@address or ssh address -l user
- scp: copy file(s) between machines: scp file address:path
- sftp: open a connection for multiple file transfers, sftp address, then use, cd, lcd, put, get
rsync: "sync" between machines, i.e., only copy files that have changed
ftp: older protocal to allow people to share files between machines, generally now only used in "anonymous" mode: you can make your files available to others (e.g., collaborators) even if they don't have local accounts, by putting them in a special ftp location (/home/ftp)
httpd: you can make files available through a web browser by putting them in a special location (/home/httpd/html)

Shells and shell scripts:

Concept of a shell: allow for scripting of operating system commands (although this often can also be done using scripting languages)
Most common shells: bash and tcsh
you can run a shell script by:
- source scriptname
- make it executable (chmod +x scriptname), then run it like a program (with full path name, or by putting directory in the path)

Example:

#!/bin/csh
# silly script that counts the number of lines in all files specified on command line, and
# substitutes the string goodbye for the string hello in all of these files that have more than 10 lines
foreach file ( $* )
  set nlines = `wc $file | awk '{print $1}'`
  if ( $nlines > 10 ) then
    sed 's/hello/goodbye/' $file > $file.new
  else
    echo $file "is a short file!"
  endif
end

Unix shell command summary

Customizing your UNIX environment:

shell startup scripts: .bashrc and .cshrc
- alias shortname command
- setenv environmentvariablename value

Exercise 4:

Create an astr555 subdirectory in the small disk partition of your desktop, or in your home directory (what's the difference?)
Edit your .cshrc file, and create an alias to cd into that directory (from anywhere) with the shortcut command a555,
Add your ~/bin directory to the PATH in your .cshrc file so that you can run programs from there every time you log in. Make sure you append/prepend this directory to the PATH and not wipe out the pre-existing PATH! What happens if you don't do this?