CLI Fundamentals for the Cloud

Operating a computer with BASH is the reality the end-user faced prior to Window's graphical user interface. In the end however, what we're actually doing is rather much the same. By deploying actions via a CLI, we're empowered in our agency in what we can schedule and automate in the environment. 

Ensuring we're starting off at the right directory is key, for that we use;

pwd

This outputs are current directory which informs us were we're currently sitting at in the system.
A basic rule of command syntax is that is delivered in below format;

command -> option -> argument

eg.
sudo usermod -a -G Sales arosalez
sudo usermod -> -a -G  -> Sales arosalez

The most basic commands revolve around providing quick user or system data.
I won't elaborate on the most of these as they are either ubiquitous or pretty self evident.

ls, cd, whoami, id, hostname, uptime, date
cal, clear, echo, history, touch, cat

Don't forget to use the tab autocomplete function when inputting CLI commands, it should save you lots of time!

When you create users in Linux, by default they are included on the /etc/passwd file.

You can use the head or tail command to display either the first or last 10 lines of a file. One of the options of a head or tail command is -n, which lets us determine how many lines we display. So in the example we wanted to print only the first 3 lines of the passwd file, we'd enter the following;

head -n 3 /etc/passwd

Moving forward quickly, useradd is another vital command with some handy options such as determining the users home directory path -d, adding an account expiration date -e and even a comment -c feature usually used to house a users full name. 

If we want to check we've successfully added a user onto our Linux machine, we can use the grep command. This command searches within documents again with the command/option/argument syntax we saw above. If I want to want to check if SgtSunshine is a user, I'd input;

grep SgtSunshine /etc/password

Other related commands to users are passwd and userdel. But users are only the atoms of our building blocks as we'll be predominately utilising groups to manage user permissions making groupadd, groupmod and groupdel completely essential. Another powerful related command is gpasswd used to administer the /etc/group file allowing for users to be added -a or deleted -d with members set with -M option and administrators by -A.

The building blocks of users and groups boils down to user permission levels. Fundamental to this is the root user, which has complete access over a computer systems, highlighting the need to create a standard user with the right privileges to avoid any unauthorized access across the system.

By also using the /etc/sudoers file, we can however issue permissions for specific users or groups to use particular commands via the visudo command. As a user with these permissions, we utilize the sudo (delegated permissions) command as it's safer then using su (full administrative permissions) where we would need to know the administrator password! If we want to actually list our sudo permissions we'd enter;

sudo -lu

This is extremely handy when troubleshooting if a command isn't working for you. Moreover, the use of the sudo command is also automatically logged at /var/log/messages, meaning it's usage can be adequtely monitored.

When we want to edit files in the command line, there are two popular text editors that work directly within the shell, vim and nano (the latter may need installation into debian/ubuntu distributions - sudo apt-get install nano). Now because of the excellent Vimtutor mode, I won't spend much time reviewing the commands here or the interaction between command and insert mode. However below I've briefly listed how we enter each editor;

vi <filename>
nano <filename> 

Nano is much more straight forward with CTRL+<key> being used for the majority of options. Whilst Vim operates much more inline with a CLI with commands like forced quit without saving :q! and forced save and quit :wq! commands. 

A further core element of the CLI is utilising it to get around the file system. Below I've listed some of the core directories commonly used in Linux distributions;

/            root of file system
/boot           boot files and kernal
/dev            devices
/etc            configuration files
/home           standard user's home dir
/media          removeable media
/mnt            network drives
/root           root user home dir
/var        log files, print spool, network services

As mentioned at the very beginning of this post, the ls command is one of the most commonly used CLI commands as it essentially lists the files & folders listed in our current directory. This command has a number of options -l for the long format, which allows us importantly to view permissions. -a show us the hidden files, -R shows us the subdirectories within and it's also possible to sort by extension, file size, modification time and ver number, but I haven't listed these here. 

To view files, it's usual to just deploy the cat command. But in the case of exploring log files, for example, which are likely to contain more lines of text then could fit in our entire command line, it makes more sense to use the head, tail, more or less commands to determine the way in which we will most optimally view view the file. Like many, if not all Linux commands the aforementioned, come each with their own bespoke set of options to most accurately execute your desire.

Very interestingly for me as a Windows user for all my computing life, Linux treats everything as a file! And I mean commands, devices, directories and documents - EVERYTHING. Plus, unlike Windows, it treats everything case sensitive, so this is always worth paying close attention to. 

Like the Windows shortcut for copy pasting (CTRL+C, CTRL+P) to copy a file or directory you simply need to use cp <file-name> <destination> to do so. If we want to then remove the copied file, we'd use rm command instead. However, very usefully, to avoid removing a directory with files within, you're required to utilise the additional option feature of the command line to achieve this. 

mkdir makes a directory and can make multiples without an option. But if you use the -p option, you can make multiple nested directories at once which is pretty useful;

mkdir -p /home/user/dir1/dir2

Now to move a file/dir we follow a similar pattern to cp. We use mv <option> <destination>. Again there are a flurry of different options to ensure we don't encounter a number of issues, having to rename the file with an additional command and avoiding overwriting existing files as an example. rmdir is a specific command, which is practically an equivalent to rm -d.

The command I've now got a twitch reaction to inputting the moment I SSH into an EC2 instance is pwd. This just confirms where you are in the file directory structure, really important as it will determine the outcomes of your commands.

A concept which isn't unique to the Linux file system however is the use of absolute and relative paths. Simply the distinction here is that the absolute path is authoritative, it leads straight from the root directory. This idea is familiar to me as when building this simple website, I would use relative hyperlinking from within the directory the webpage sat within. 

Building on working with the file system within Linux, we have an additional bundle of commands to work with;

• hash: lists the history of programs and commands executed from within the CLI.
• cksum: verifies a file has not changed, can be used to check downloaded files too.
• find: searches for files with additional options of specific criteria like file name: size and owner. Can also be additionally deployed with wildcards to enhance it's operation too. It has options to automatically delete the returned list -delete or even execute a determined command straight away -exec <command>.
• grep: looks inside the content of a file to assess if there is a text pattern that matches your desired input. It also is powerful enough to search files within a directory grep <string being searched> <directory path>, making it really useful. 
• diff: used whenever you wish to find out the difference between two different files quickly. 
• tar: turns multiple files into a single file. To later extract a .tar file, you use -x option. To get this .tar extension however, you may also wish to use the -f option, to specify it!
• gzip: can only compress one single file, so is usually used with the tar command.
• zip: can compress multiple files and even directory hierarchies.
• unzip: decompresses the contents of a file. 

Linux also uses links and inode. Despite using shortcuts on Windows myself quite a bit, I found this unnecessarily confusing, however I found the diagram below the most immediately illuminating between the differences of hard and soft links;

Next I will talk about how we manipulate the file permissions, ie. who has access. The most important part of this is the approach we need to take;

The principle of least privilege.

The principle that a security architecture should be designed so that each entity is granted the minimum system resources and authorizations that the entity needs to perform its function.

Now working on this basis, we can say, never ever use chmod 777! But what does that even me? So thinking about Linux treating everything as a file, we have 3 seperate kind of permissions available to files. 

read - allows a user to access the file.

write - allows a user to access & edit the file.
execute - allows a user to run a script. If you use Windows, you can think of this as a .exe. It also allows users to view a file, so can be considered a higher level of access then just 'read'.

Linux gives us two ways to see the permissions associated with files. Symbolic and absolute. For the symbolic example, I'm going to be cheeky and borrow the slide from the AWS re/Start materials as it describes symbolic mode very succinctly;

The command to view the file list displaying permissions in this way is ls -l.
The above slide also shows how to make the symbolic command determine between the individual user or by the group - the latter is also vital as it allows us to manage user access levels most effectively. However, there is an easier option within the chmod command to employ associating permissions with files. Using numbers as the options, we can easily determine what permissions we wish to employ.

This table below further exemplifies how the permissions work out;

0	- - -	no access
1	- - x	execute only
2	- w -	write access only
3	- w x	write and execute
4	r - -	read only
5	r - x	read and execute
6	r w -	read and write
7	r w x	read, write and execute (full access)

 

But as before, we can still use the u/g/+/- options [shown on the symbolic mode slide] within the chmod command along with the absolute mode to have a really powerful command without requiring too much input. 

Now I will try to explain, briefly as possible, additional options we can use within the CLI and further build on all the commands we've already explored.

Right at the start, we saw how Bash is constructed in the command/option/argument syntax format. Using quotation marks (" ") overrides the usual Bash interpretation of an argument with a space as two separate arguments - this just means you can now use spaces in your arguments! eg;

usermod -c Dev user maurelius

usermod -c "Dev user" maurelius

This is important as commands have limited parameters depending on it's function. This therefore allows us further flexibility. Wildcards, used with find or grep, can be used to specify an unknown characters or even a set of limited or specific values in a search;

* eg. find *.txt documents/
^ Will return any file with a .txt extension.

? eg. find tex??.txt documents/ 
^ Will return any file matching, thus here would return potentially text1.txt, text2.txt but not text.txt as it's looking for two values in this example.

For this section about expanding the CLI usage, we will look at pipe and redirectors along with the tee command. These commands are all about doing something additional after we've already deployed a command. I will, for speed and efficiency again rely on online resources to quickly demonstrate these;

Examples of these redirectors in practical use could be;

uptime > info.txt
hostname >> info.txt 

^ So above, we see that I first output the computer's uptime information, and then appended the host name information with the following inputs.

The next tool in our arsenal is known as the pipe and is represented as |. 
With this tool, we can simply pipe in a further command;

cat <filename> | less

Credit: guru99.com

There is also lots to be mentioned around the sed, sort, and awk commands. However, I'd each of these commands are quite involved and deserve a separate article on themselves. However important to know that they are used for text manipulation and searching. 

Now I will touch on using the CLI to access the computer processes, but only briefly. I've been using Windows long enough to experience Firefox being amazing to having constant memory leaks and requiring closure via the task manager. The commands for accessing the running processes in the CLI are ps and pstree (which has child processes nested). This is exactly where piping and a command like grep comes in useful;

ps -ef | grep sshd

^  the above command displays all processes, however is then piped to filter the process we're wanting to check on, in the example above, sshd. You can then also use top to check mem/cpu usage, again similar to the Windows Task Manager.

Services aren't dissimilar to processes. However, the best way to think about them is as sitting behind the desktop doing things you wouldn't notice on a GUI, like picking up on the fact you inserted a USB flash drive, for example. To manage we use the systemctl command. Outside of the CLI, you can use Amazon CloudWatch instead if you want a GUI.

We also have the power with the CLI to schedule actions to occur on our computer.
The most popular method to do this is with the at and cron job commands. 
cron is more powerful, as it's a scheduler of tasks (the kind of thing we would use with logs, which I will mention soon!). 

The little cheat sheet above shows us the clever format used by the command to determine when we'd want to run particular tasks. To further illustrate the point, I've included two examples below to further illuminate the concepts;

0 1 * * 1-5 /scripts/script.sh

^ the above syntax would execute script.sh every weekday at 1am.
Worth checking your system time is accurate eh! 😁 Another example;

*/10 * * * * /scripts/script.sh

^ the above syntax now executes every 10 minutes! Using the slash operator function.

Now we turn to the management of log files. This is core to the work of any computer system administrator and this is no different in the cloud. The importance of logs for audit purposes, troubleshooting, ensuring SLA's agreements and monitoring industry requirements make them a vital tool of the job. This is exactly why grep, head, tails commands come in so useful for a cloud role. 

I will now do a flyby around BASH scripting as in my intention is to do some practice, either during or post-programme (most likely the latter due to the intensity!). Moreover, many of the conceptual elements - variables, loops, conditionals, many other logical control statements etc all follow the basic tenets of object oriented programming. 

Anyway - Bash scripts offer us a way to automate complex, multi-command operations into a single file. This means with something like the cron command we just looked at, we'll be able to make the most efficient use our time. A little bit like Wallace & Gromit, except with things hopefully working!

• Scripts can be entered at the launching stage of an EC2 instance, which means we can customise our instances very quickly, albeit, depending on use cases you may wish to simply create your own custom AMI instead

• Scripts also help us by providing documentation (if best practice is in place) elaborating on the purpose and actions being undertaken, 

• Running a script requires proper permissions. That's why we talked about it alot above!

For the purposes of including a shebang in this post, here's a quick little example of a first bash script;

#!/bin/bash
echo "What is your name?"
read name
echo £Hello $name"

Whilst not mentioned, you of course would need to create the file nano getname.sh as well as ensure you have the permissions to execute it, chmod +x getname.sh. A short but sweet example.

And last but not least, installing software. This wasn't new to me due to all my faffing with a Raspberry Pi, but having only had experience with one Linux distribution, there was some new bits for me. 

In essence, each of these represent different kind of package managers (bar the last of course[!]). They manage how we install, update, remove and inventory the software installed on our computer. 

^ The above example of wget & curl only really is relevant in the case of the soft we're downloading being source code or not hosting within a repository. 

And that's it! A super fast run down of the absolute basics of the CLI for the Linux and applicable for most Cloud roles. 🙏