unknown_document

We have seen how to execute commands , and how to build pipelines that pass information between commands to perform more complex tasks. In this section, we will touch on how this information is passed between commands, using " streams ". A stream refers to the information that flows through the pipeline, from command to command. A command receives information from the "input stream", processes it, then passes the result to the "output stream" (or possibly sends error information to the "error stream", if needed). To help solidify the concept of a stream, suppose there is a command that reads text from the input, modifies it in some way, then passes the result to the output. One might expect the process of executing that command to be: Read the input into memory Modify the content in memory Pass the modified content to the output However, consider two cases: Suppose that the input comes from a file that requires more memory than is available in the system. In this case it would be impossible to load the entire file into memory, and the command would fail. Suppose that the input comes from a continuous source, such as a temperature sensor that reads at regular intervals. Since the temperature is constantly being monitored, there is no such concept as "the end of the input". In order to handle these situations effectively, the process is a bit more like this: open the input source read a line of data if the end of the input stream is detected, stop processing and close the stream perform some operation on that line and pass the result to the output repeat from step 2 With this in mind, once the input stream is opened and a line is read, the command doesn't know how much data is contained in the input; it simply reads a line, processes it and passes the result to the output, then reads another until (possibly) all input data has been consumed. Similarly, the next command in the pipeline simply receives each line of data, processes it, then passes the result to its output. This flow of lines of data led this concept to be called a "stream". Standard Streams When any of input , output , or error are not specified, they each default to a specific "standard" stream. Under the hood, each stream is implemented as a file that can be read from or written to. There are three standard streams: Standard Input Standard input, called stdin and sometimes referenced numerically as "0", is the stream from which a program reads its input data, if not otherwise specified. Not all commands require an input stream. For example, the ls command, which displays information about files contained in a directory, reads input from the filesystem without any input data stream. Standard Output Standard output, called stdout and sometimes referenced numerically as "1", is the stream to which a program writes its output data. By default this is usually connected to the terminal, so that the results of a command are printed to the screen. Not all commands generate output. For example, the mv command, which renames a file on the filesystem, does not generate any output when it is successfully invoked. Standard Error Standard error, called stderr and sometimes referenced numerically as "2", is an alternative output stream that used by commands for error or diagnostic information. The main purpose of stderr is to allow a command to generate diagnostic feedback without polluting the output stream. Note This section is intended to provide just enough information to allow users to begin understanding and using streams in many common applications. If you want to dig a bit deeper into the streams and how they work a good place to start is Everything is a File . One of the useful features of the standard streams is that they can be replaced by other streams, combined etc, sent to in a process called redirection , which is the topic of the next section.

Overview At a high-level, command-line commands represent a piece of code that: Optionally receives input data, then Optionally performs an operation on that data, then Passes the (possibly modified) data to the output, and Optionally generates an error message. The following diagram provides a useful representation of this: The "natural" flow goes from the input, to the command, then to the output. If an error occurs, then information about that error is sent to the error. With this diagram in mind, let's proceed to see some examples that will help solidify this concept. Executing Commands Commands are executed from the command-line by typing the name of the command, possibly followed by options and/or arguments. Arguments provide the command with information that it needs to perform its task. For example, a command that operates on a file might be passed the name of the file to perform those operations on. Options provide "knobs" that allow the user to customize how the script behaves. The combination of command, arguments, and options can be combined into a call signature , which describes the format to be used when calling the command. As a simple example, the echo command is often called with a single string as an argument: ninja$: echo Hello World! Hello World! ninja$: which takes the argument (Hello World!) and passes it to the output which, by default, prints the output to the terminal screen. In the next section we will see how to use this output as the input for other commands.

By default, information flows through the standard streams , but can be directed to other other locations. For example, we already saw that pipelines can be used to direct the stdout of one command to the stdin of another. Redirection provides a mechanism through which standard streams can be combined and/or sent to another location. For example, by default stdout is generally connected to the terminal, so that command output is displayed in the terminal. Suppose you want to save the command output to a file. One option would be to copy and paste it from the screen, but this is manual and error-prone. In this case, it makes more sense to redirect the command's output to a file. Redirecting stdout Redirecting stdout can be achieved using the following call signature: {command} > {filename} where command is the command to execute, > indicates that stdout should be redirected, and filename is the file to direct the output to. Let's see this in action: First, repeat the simple "Hello World!" example and redirect the output to output.txt : ninja$: echo Hello World! > output.txt ninja$: Note that unlike the original example the "Hello World!" text is not displayed in the terminal. Let's now cat the output file to see the content: ninja$: cat output.txt Hello World! ninja$: Note that if filename already exists, it will be overwritten, which may not be what is desired. As an alternative, you can append output to the specified file using the following call signature: {command} ]] {filename} Let's execute that command: ninja$: echo Hello Again! >> output.txt ninja$: then see the result: ninja$: cat output.txt Hello World! Hello Again! ninja$: Redirecting stdin In a similar way, we can also redirect content to a command's stdin using the call signature: {command} < {filename} which is similar to the previous case, except the < indicates that stdin should be redirected. As expected, executing this sorts this lines from output.txt , sorts them, and prints the result to the terminal: ninja$: sort < output.txt Hello Again! Hello World! ninja$: Note that there are two additional variations of stdin redirection, which support here documents and strings , but will not be discussed here. Multiple Redirection Both stdin and stdout can be redirected by combining these two features into a single call, using the call signature: {command} < {input file} > {output file} As an example, let's now sort our file in reverse and redirect the result into a new file, reversed.txt : ninja$: sort --reverse < output.txt > reversed.txt ninja$: Now we can display the new file contents: ninja$: cat reversed.txt Hello World! Hello Again! ninja$: which confirms that the command executed as expected. Now that we have learned the basics of working with the command line, let's next look at how to combine multiple commands into shell scripts , which we can execute like a single, customized command.

We saw in the previous section that by default commands (generally) print their output in the terminal window. This section introduces the concept of "pipelines", where the output from one command is passed as input to the next command, in order to perform more complex operations. This concept can be visualized using the following diagram, which shows a pipeline made up of 2 commands: This pipeline can be executed from the command line as follows: {command 1} | {command 2} where the "|" symbol is called a "pipe". This pipeline can be extended to n commands by "piping" the output to additional commands, using: {command 1} | {command 2} | {command 3} | ... | {command n} Example Pipeline As simple example, suppose we have a file containing a list of color names, and we want to generate a sorted list of 5 randomly-selected color names from that list. This can be achieved by combining the following commands: Execute the shuf command to read colors.txt from the file system, shuffle the lines, then pass the result to stdout , then Pipe that result to the head command to limit the result to 5 lines: ninja$: shuf colors.txt | head -n 5 SkyBlue MediumAquaMarine Olive Orchid LimeGreen ninja$: Next, pipe that result to the sort command to sort the lines: ninja$: shuf colors.txt | head -n 5 | sort LimeGreen MediumAquaMarine Olive Orchid SkyBlue ninja$: which achieves our original goal.

Much of the terminology used when discussing modern terminals comes from history, which can make understanding terminals seem more confusing than it should be. A small amount history can help provide some background information to help some of the concepts be more relatable. Early computers were comprised of multiple refrigerator-sized cabinets, each of which performed one of the basic tasks of the system. For example, one cabinet would contain the " central processing unit " (CPU), while other cabinets held tape drives , disk drives , punched-card readers , a line printer , and any other peripherals that were available to the computer. The user-interface to these computers consisted of a keyboard and some means of displaying the output from the computer, such as a sequence of light bulbs or later a Cathode-ray Tube (CRT) display , connected to the computer via cables. The user-interface was typically arranged into a " console ", where the operator could sit and interact with the computer. Since the user-interface represented the point at which electrical signals "enter" and "exit" the network, the user-interfaces themselves became known as " terminals ". Standardization As often happens in technology development, there were initially a wide range of competing technologies, but over time market forces and economies of scale led these competing technologies to converge towards standardization around a handful of those technologies, the names of which have become much of the jargon that surrounds the terminal today. Early terminals were electro-mechanical "Teletype writers", generally shortened to " TTY ", which is a term still used today when interacting with the terminal. As with a modern keyboard, each key on the TTY represented a human-readable character such as a letter, number, punctuation, etc. As the user would press a key to input data into the computer, each key would be " encoded " into a sequence of 8 1s and 0s called bytes , which could be understood by the computer. These bytes were sent over the cable to the CPU, which performed the requested operations, then the resulting byte(s) were transmitted back to the terminal where they were "decoded" back into characters for human-consumption. The rules used for encoding and decoding each character eventually became the ASCII standard. Despite the benefits of standardization, the ASCII standard was written to support US English, which led to several extensions in order to support other languages, which eventually led to the UTF-8 standard which unified the various extensions into a single encoding that is broadly in use today. Over time display technologies evolved which led to the introduction of "video terminals" (VT) which allowed text and other graphical information to be displayed on a screen. Some of the most popular video terminals were the VT100 and later VT200 series, which introduced support for the ANSI escape codes that have become standards that are still in use today. Emulation As microprocessor technology advanced and the cost of memory and other peripherals dropped, terminals began to handle increasingly-advanced operations, which led to the introduction of " intelligent terminals ", which differentiated them from the "dumb terminals" they replaced. In order to make the newer terminals " backward compatible " with existing software, these terminals included hardware to " emulate " the older devices. Over time the emulation functions were implemented in software, which eventually led to the introduction of fully-software " terminal emulators " that could offer features that would have been difficult to impossible to implement in hardware, such as command-line completion and syntax highlighting . With the introduction of terminal emulators, "the terminal" became a software window that is opened in order to gain access to the operating system . The software that provides this access is called "the shell", which is the subject of the next section.

The ability to execute commands from the command line provides a great deal of power and efficiency, but executing common commands can get repetitive. In these cases, shell scripts allow groups of commands to be executed together with a single command. As a very simple example of how to execute shell scripts, lets repeat the "Hello World" example from the previous section. First, create a file named "myscript.sh" with the following contents: #!/usr/bin/bash echo Hello World! The first line of this file is called the shebang , which starts with #! followed by the path to the interpreter that should execute this script, /usr/bin/bash . Now, you can execute this script using the bash command, by executing bash myscript.sh as follows: ninja$: bash myscript.sh Hello World! ninja$: In this example the bash command is used to execute the commands contained in myscript.sh . Shell scripts can also be executed directly by calling the full path to the script. If the script is location in the current working directory, then the script name can be prefixed with a . , which is an alias to the current working directory a shortcut. Let's try: ninja$: ./myscript.sh bash: ./myscript.sh: Permission denied ninja$: Whoa, what happened? Executing files can present security risks, so permission must be explicitly granted before a script can be executed. This is accomplished using the chmod command as follows: ninja$: chmod u+x myscript.sh ninja$: In short, chmod is short for "change mode". In this case we want to add executable (x) permission to the current user, which is stated as "u+x". Finally, we state the name of the file to apply this change to. Now, let's try executing the script again: ninja$: ./myscript.sh Hello World! ninja$: With that, we have enough background information to start learning the fundamentals of working with the command-line in the following sections.

As described in the previous section, when a terminal is opened it runs software that provides access to the computer's operating system. This software has became known as "the shell ". Let's take a quick at how that name came to be used. At a high level, operating systems are designed with two distinct parts: A first part which contains the low-level software that has complete control of system resources. A second part which "surrounds" the first to prevent access to it, except for specific, pre-defined actions that application software are allowed to take. This concept has been visualized as a nut contained within its shell, where the meat of the nut (the " kernel ") contains the low-level software, and the hard shell that surrounds it prevents outside access except for specific, pre-defined " system calls ", such as those defined by " the Linux API . With this in mind, when users open a terminal they are presented with an application that accepts commands that are typed in the terminal, executes them, and returns the output to the terminal interface. This application provides users an interface to make system calls, call other applications, and execute scripts that define higher-level interactions with the operating system. Since these "shell applications" allow users to interact directly with the shell, they have become synonymous with "the shell". On Linux systems, the most common shell application is Bash , which is often installed by default, though other popular shell applications include ksh , zsh , and fish . Since Bash is the most common shell we will use it for examples, but most shells feature backwards-compatibility with Bash so most examples should work across other shells. As terminals evolved from hardware to software, the boundaries between the terminal and the shell became increasingly blurred, to the point where the concepts of "the terminal", a "terminal emulator", "the console", and "the shell" have become interchangeable in many conversations, which has led to confusion about what each term actually refers to. Possibly the most common term for these concepts among users is "the command-line", which is the topic of the next section.

As described in the previous section , the concepts of the terminal, a terminal-emulator, the shell, etc have all become interchangeable with the "command-line" in many contexts. The command line is the text interface to your computer, named after (well...) the line on the screen where the user can execute commands. To demonstrate, let's see how to execute the ubiquitous "hello world!" example: ninja$: echo Hello World! Hello World! ninja$: In this example, we executed the echo command to pass the string "Hello World!" to stdout . More broadly, the command-line is the central component of the Command-line Interface (CLI) that creates the user interface in many terminal applications. Next, let's take a look at some of the more common commands .

Userdata is a type in Lua that basically represents a "black box" of data that is used in the code, but can't be printed or otherwise inspected. Using userdata is an advanced topic that generally involves integrating C code with Lua. In most cases anyone other than advanced users who are integrating Lua and C will only see userdata when debugging a script and printing values, and the only reason we bring it up is so that reader have some idea what they are looking at when they see it.

To start vim with an empty buffer , simply type: nvim To open an existing file in Vim, simply add the path to the file: nvim [path/to/file] You can also include multiple files, to open them all at once: nvim [path/to/file/one] [path/to/file/two] ... Neovim also supports " globbing ", which allows you to open multiple files by specifying a simple pattern. For example, to open all python files located in the current directory, simply type: nvim *.py In additional to opening files, Neovim supports a variety of options to control how it starts and which features are enabled. While these may not be used on a daily basis, it is good to know that they exist and how to find them. List some of the common options from the terminal with the command: nvim --help or visit the Neovim docs for the complete list. To start with options, specify one or more options when the nvim command is invoked. For example, use the -R option to open a file in read-only mode: nvim -R [path/to/tile] Finally, as a terminal program Neovim can be called from command-line scripts to implement some types of powerful workflows. For example, suppose you want to edit the current directory listing you could execute: ls -a | vim - this command lists the current directory ( ls -a ), then pipes ( | ) the result to Vim which is listening on stdin ( - ). While this example is a bit contrived, this demonstrates a simple way to leverage a wide range of command-line tools to get content into Vim.