Linux and CLI

Some Useful commands

Tail/Head

To view the last 10 lines of our bash history we can use:

tail ~/.bash_history

Here tail is the command that allows us to view the last 10 lines, and ~/.bash_history is simply the file we wish to view. Tail and its counterpart head allow us to quickly see the first/last 10 lines of a file respectively.

Bang Bang

Known as 'bang bang', using two exclamations will run the previous command. It may be questioned why this is useful when we can simply use the up arrow to get the previous command, but consider:

chmod 700 <filename>
ERROR
sudo !!
SUCCESS

Running chmod without super-user permissions may result in an error as you are not allowed, however instead of using the up arrow and then navigating to the front, and typing sudo, we can simply use sudo !!.

Less

Less allows you to read a file. It has a similar look and feel to vim, however you cannot edit a file, only read it. Whilst viewing, 'q' is used to exit, and '/' can be used to search for words/phrases from within the file.

less <filename>

Files and directories

Touch is used to make files, and mkdir for directories. Below are some examples.

mkdir newDirectory
cd newDirectory
touch myFile.txt
cd ..
rm -r newDirectory

This will create a new directory labelled newDirectory, change directory to this, and then create a file called myFile.txt. We then back one directory, and using rm to delete with the specified -r (recursively delete) flag, we delete both the directory and its files.

Copying Files/directories

Copying files can be tedious. Below are some common commands when it comes to copying files locally, or to different directories.

touch myFile.txt # Create New File
cp myFile.txt myCopy.txt # Copy myFile.txt to a new file myCopy.txt
cp myCopy.txt /home/myFolder # Copies the copied file to a different directory
cp /home/myFolder/* /home/mySecondFolder # Copies all the files from the first directory into the second directory
cp myFile.txt myCopy.txt /home/mySecondFolder # Copies only the specified files to a second directory

Wildcards and Replacements

Say we want to write out many files with similar names. Writing 'touch file1.txt file2.txt ...' would quickly become tedious. We can instead create new files like one of the ways below:

touch file{1, 2, 3, 4}.txt # Creates 4 text files
touch file{1..4}.txt # Creates 4 text files
touch file{1..4..2}.txt # Creates 2 files (every 2nd number from 1 to 4)
touch file{a..z}.txt # Creates a file for each letter in the alphabet
touch file{a..c}{1..3}.txt # A file for each possible combination

But now after running these commands we have so many different files in our directory, and deleting them all will be a problem. Using the '*' wildcard, we can remove everything much easier.

rm file*.txt # Would remove all files we created
rm *.txt # Deletes everything ending in .txt
rm file* # Deletes everything starting with file

Streams and Pipes

One other way of creating and writing directly to a file in a single line is using streams.

echo 'hello' 1> newFile.txt

This command does not actually print 'hello' to the screen as expected by echo, but instead (specified by 1>) redirects the stdout of echo and points it to newFile.txt (whether or not this file exists yet). Using 2> instead of 1> redirects stderror rather than stdout in the same fashion. Removing either 1 or 2, and simply using '>' will redirect both stdout and stderror to the same place. Using a single '>' replaces text in a file, and using '>>' appends to the contents in a file.

echo 'Some Text' > newFile.txt
cat newFile.txt
# Some Text
echo 'Some More Text' >> newFile.txt
cat newFile.txt
# Some Text
# Some more Text
echo 'This Will Replace' > newFile.txt
cat newFile.txt
# This Will Replace

Stdin can direct file contents into a program. In the below scenario, the contents of list.txt is taken, and parsed to grep which looks for a line which contains 'error-log.txt'. The second line demonstrates how you can split streams to different locations. /dev/null is considered to be the abyss, and routing output there means it will be lost.

grep "error-log.txt" < list.txt
grep "error-log.txt" < list.txt 1> output.txt 2> /dev/null

Pipe ('|') takes what one program outputs and puts it into the next one. The previous example can be re-written as below:

cat list.txt | grep "error-log.txt"

This prints out the contents of list.txt, then grep searches for the string "error-log.txt".

sudo

Sudo implies switch user and do. It performs a single actions as the superuser, and then exits from the elevated privilege. Instead of always being the super-user, sudo is used to ensure least privilege is enabled within a system.

Permissions

drwxr-x-r-x is an example of permissions on a directory (as denoted by the d at the beginning). The first 3 characters refer to the file/directory permissions for the owner. In this case they have rwx (read-write-execute) permissions meaning they have full access. The second triplet refers to the group that owns the file. In this case they have r-x (read-execute), meaning that users of the group can either read, or execute but not write to the file. This is common. The final triplet refers to everyone that is not the user or the group. They also have r-x (read-execute), the same as the group.

chown

To change the owner of a file or directory, we can use the chown command. If a directory were owned by root for example, we could run the following command to change directory ownership to user and group ubuntu.

sudo chown ubuntu:ubuntu <directory>

chmod

chmod is different to chown, in that it directly changes file permissions rather than the owner of a file. To edit the permissions we discussed earlier, specifically that of 'everyone' (the final triplet), we could run a command.

sudo chmod u=rw,g=rw,o=rw myFile.txt
sudo chmod 777 myFile.txt # Adds read-write-execute permissions for everyone.

Now everyone (user, group and 'everyone') have both read and write permissions to the file. In the second command we used 3 digits rather than specifying using u=rw etc. Each digit corresponds to user, group, and everyone respectively. The formula for the digit is adding 4 for read permissions, 2 for write permissions, and 1 for executable permissions. 744 would imply full permissions for the owner user, and read permissions for group and everyone.

SSH

Secure shell allows you to remotely execute commands on another computer from your own machine. Using the command:

ssh-keygen -t rsa

We can generate a public/private key pair for use. You do not want to give out your private key, only your public one. The authorized_keys file (will be output from the above command) is used to store other people's public keys whom you trust. If a private key owner can verify that the public key is owned by them, then they can remotely login to your machine. The command below allows you to connect to the remote machine (it will ask you if you are sure first):

ssh <username>@<ipaddress>

SFTP

Secure file transport protocol is used to securely send/receive files from a remote machine (similar to ssh). SFTP is automatically setup when ssh is (as they use the same protocol and ports). You must be careful when using SFTP as you manage two directories at the same time.

pwd # Outputs the present working directory for the remote machine
lpwd # Present working directory for local machine
put <filename> # Upload a file from your machine to remote working directory
get <filename> # Get a file from remote machine to your pwd

wget and curl

wget downloads something from an internet based off a URL. It works similar to cp, but instead of copying a local file, it copies something from the net. While wget works similar to cp, curl is more like a linux program. It is like 'cat', in that it can operate on input and output streams. For example we may want to curl a website into a text-file.

curl <url> > output.txt

Sometimes we may want to POST, PUT, DELETE instead of GET. Utilising the -x flag, we can specify which verb we want to use. Cookies and headers can also be sent using curl with their respective flags.

Writing Your Own Scripts

When we want to run multiple commands at once, we use a shell script. A file containing commands that has the .sh tail. In the first line of our .sh file, if we specify '#! /bin/bash' we can run the file using './[filename]'. And if we added the command to our path, we would not need the './' and could simply run it by name.

Variables/User-Input/Arguments

Setting a variable is as easy as NAME=value. Sometimes we want user input to be processed in the file. Using the read program, we can achieve this. For example:

read name && echo hello $name
# Matthew
# (Output): hello Matthew

A -p (prompt) tag can be used to specify what a user needs to enter.

read -p "Enter your name here: " name && echo hello $name
# Enter your name here: Matthew
# hello Matthew

Arguments/parameters can also be passed into a shell script file. To access them from within the file you can use $0, $1, $2 and so on for however many arguments were supplied.

Conditionals

As we know, a conditional is a statement that only runs if a condition is true. For example, we may wish to only use ~/temp path if the user does not supply an alternative path as an argument.

#! /bin/bash

DESTINATION=$1
read -p "Enter a file prefix: " FILE_PREFIX

if [ -z $DESTINATION ]; then # Checks length of DESTINATION (the argument)
    echo "No path provided, deafualting to ~/temp"
    DESTINATION=~/temp
fi

mkdir -p $DESTINATION
cd $DESTINATION
touch ${FILE_PREFIX}{1..10}.txt
echo done

This script allows us to create 10 files with a file-prefix, followed by their number, in a directory of our choosing. If we do not specify a directory, it defaults to ~/temp. It also moves us to the directory, and prints done when completed.

Switch Case

Below is a simple example taken from the course.

case $1 in
  "smile")
    echo ":)"
    ;;
  "sad")
    echo ":("
    ;;
  "laugh")
    echo ":D"
    ;;
  "sword")
    echo "o()xxx[{::::::::::::::>"
    ;;
  "surprise")
    echo "O_O"
    ;;
  *)
    echo "I don't know that one yet!"
    ;;
esac

Loops and Arrays

#!/bin/bash

friends=(Kyle Marc Jem "Brian Holt" Sarah)
echo My second friend is ${friends[1]}
for friend in ${friends[*]}
do
    echo friend: $friend
done
echo "I have ${#friends[*]} friends"

Above is a simple example of a for-loop, looping through an array. This would output 6 different names (Brian Holt is seen as two names), but will state "I have 5 friends" due to the length of the array.

Cron

Cron is a feature which allows a task to be completed on a schedule (this could be daily, weekly etc.). There are cron folders like /etc/cron.daily (along with their other versions) and when a script is placed in this directory, it will be run each day. Just make sure the file you place in there has executable privilege.

Crontab is used to define a cron schedule. This so the script can be executed on a more defined schedule like every 3 minutes for example. 'crontab -e' will open up the editor where we can specify what we want to run.

* * * * * <filename> # Asterisks respectively: <minutes><hours><day of month><month><day of week>
5 * * * * <filename> # Would execute every 5 minutes
@reboot <filename> # Would execute each time the system is rebooted