Conquering the Command

Line
Unix and Linux Commands for Developers

Mark Bates

Contents
Foreword
Preface
Acknowledgments
Introduction
Chapter 1 Basics and Navigation
1.1 Home Directory (~)
1.2 Present Working Directory (pwd)
1.3 List Files and Directories (ls)
1.3.1 Listing a Different Directory
1.3.2 Listing All Files (-a)
1.3.3 Long Form Listing (-l)
1.3.4 Human Readable Sizes (-h)
1.3.5 Sorting by Size (-S)
1.3.6 Sorting by Last Modified Time (-t)
1.3.7 Reverse Sort (-r)
1.4 Links (ln)
1.4.1 Hard Links
1.4.2 Forcing a Link (-f)
1.4.3 Symbolic Links (-s)
1.5 Change Directories (cd)
1.5.1 Navigating Up (..)
1.5.2 Navigating to the Home Directory
1.6 Creating Directories (mkdir)
1.6.1 Create Intermediate Directories (-p)
1.6.2 Verbose Output (-v)
1.7 Copying Files (cp)
1.7.1 Copying a Single File
1.7.2 Copying Multiple Files
1.7.3 Verbose Output (-v)
1.7.4 Copying Directories (-R)
1.7.5 Force Overwriting of a File (-f)
1.7.6 Confirm Overwriting of a File (-i)
1.8 Deleting Files (rm)
1.9 Moving Files (mv)
1.10 Input/Output (|, >)
 1.10.1 Redirecting Output (|)
1.10.2 Writing to a File (>)
1.10.3 Reading from a File (<)
1.11 Conclusion
Chapter 2 Ack/Ag
2.1 Installation
2.2 The Basics
2.2.1 Regular Expression Searches
2.2.2 Literal Expression Searches
2.3 Listing Files (-l)
2.4 Case Insensitive Searches (-i)
2.5 Scoping to Files (-G)
2.6 Ignoring Paths
2.6.1 The –ignore-dir Flag
2.6.2 VCS Ignore Files (Ag only)
2.6.3 The .agignore File (Ag only)
2.6.4 The .ackrc File (Ack only)
2.7 Searching from Standard Input
2.8 Performance
2.9 Conclusion
Chapter 3 cURL
3.1 Installation
3.2 The Basics
3.3 Following Redirects (-L)
3.4 Downloading Files (-O)
3.4.1 Custom File Names (-o)
3.5 Changing the HTTP Request Method (-X)
3.5.1 Sending Parameters
3.5.2 Passing Data in a Request Body (-d)
3.5.3 Using a File for a Request Body (-d)
3.5.4 Form Parameters (-F)
3.6 Setting Headers (-H)
3.7 Basic HTTP Auth (-u)
3.8 Saving and Sending Cookies/Headers (-D, -b)
3.8.1 Using the Cookie Jar (-c)
3.9 Conclusion
Chapter 4 Find
4.1 The Basics (-name)
4.2 Searching Paths (-path)
4.2.1 Find Only Files or Directories (-type)
4.3 And/Or Expressions (-or)
4.4 Not Matching Query (!, -not)
4.5 Find by Last Modified Time (-mtime)
4.6 Find by File Size (-size)
4.7 Performing Operations (-delete)
4.8 Conclusion
Chapter 5 Grep
5.1 The Basics
5.1.1 Regular Expression Searches
5.1.2 Counting Occurrences (-c)
5.1.3 Displaying Line Numbers (-n)
5.1.4 Case Insensitive Search (-i)
5.2 Searching Multiple Files
5.2.1 Searching Specific Files
5.2.2 Recursive Searching (-R)
5.3 Searching from Standard Input
5.4 Invert Searches (-v)
5.5 Conclusion
Chapter 6 Ps
6.1 The Basics
6.1.1 Columns
6.2 Display User-Oriented Output (u)
6.2.1 Columns
6.3 Display All Processes (-e)
6.4 Display Processes by User (-U)
6.5 Customize Displayed Columns (-O, -L)
6.6 Sorting
6.6.1 Sort by Memory Usage (-m)
6.6.2 Sort by CPU Usage (-r)
6.7 Conclusion
Chapter 7 Sed
7.1 Installation (Mac OS X)
7.2 The Basics (s)
7.2.1 Global Replace
7.2.2 nth Replace
7.3 Saving Files
7.3.1 Editing in Place (-i)
7.3.2 Using Backup Files
7.4 Case Insensitive Searches
7.5 Matching Lines
7.6 Manipulating Matches
7.6.1 Changing Case
7.7 Controlling Output (-n)
7.7.1 Printing Modified Content (p)
7.7.2 Writing Modified Content to a File (w)
7.7.3 Displaying Specific Lines
7.8 Delete Lines (d)
7.9 Running Multiple sed Commands (;)
7.10 Conclusion
Chapter 8 Tar
8.1 The Basics (-c)
8.2 Saving to a File (-f)
8.2.1 Verbose Mode (-v)
8.2.2 Multiple Directories/Files
8.2.3 Building from find (-T)
8.3 Listing Files (-t)
8.4 Appending Files (-r)
8.5 Updating a File (-u)
8.6 Extracting Files (-x)
8.6.1 Extracting a Single File
8.6.2 Extracting to a Different Folder (-C)
8.7 Compressing with gzip (-z)
8.8 Extracting Compressed Files (-z)
8.9 Conclusion
Chapter 9 Extras
9.1 Cal
9.1.1 Display Current Month
9.1.2 Display Current Year
9.1.3 Display an Alternative Year
9.1.4 Display an Alternate Month/Year
9.2 Cat
9.2.1 Read a File
9.2.2 Concatenate Multiple Files
9.2.3 Save Multiple Files to a New File
9.2.4 Display Line Numbers
9.3 Kill
9.3.1 Forcefully Terminate a Process
9.4 Man
9.5 Pbcopy (Mac OS X Only)
9.6 Pbpaste (Mac OS X Only)
9.7 Tail
9.7.1 Display Last n Lines of a File
9.7.2 Reverse Output of a File
9.7.3 Watch a “Live” File
9.7.4 Search “Live” Output
9.8 Tree
9.8.1 Installation
9.8.2 Listing a Directory’s Contents
9.8.3 Limiting the Depth of Listing (-L)
9.8.4 Listing Directories Only (-d)
9.8.5 Listing Another Directory
9.9 Wc
9.9.1 Count Words in a File (-w)
9.9.2 Count Lines in a File (-l)
9.9.3 Piping Input to WC
Foreword
I distinctly remember, as a freshman in college, asking a local guru
how to rename a file in Unix. “It’s mv,” he said “for ‘move’.” To me,
he was a Unix god, and here I was not even knowing mv. Hey, we all
have to start somewhere.

No matter where you’re starting, Mark Bates’s Conquering the

Command Line is for you. Mark is an expert developer, lucid writer,
and acclaimed speaker, and I can’t think of a better guide to the ins
and outs of the Unix command line from the perspective of a
working developer rather than a Unix sysadmin. Mark’s approach is
not to give you an exhaustive and overwhelming catalog of each
command and everything it can do, but rather to carefully select the
most important commands and the most useful options. In other
words, this is a curated introduction to the Unix command line.

Conquering the Command Line starts with the essential basics—

things like pwd, ls, and cd—and then takes you on a tour of the more
exotic animals in the Unix zoo. The result is that you’ll quickly
move past mv to master commands like curl, grep, find, and ack—
not to mention ps, tar, and sed. Because of the ubiquity of Unix,
the knowledge you gain will serve you well whether you use OS X,
Linux, or *BSD. (And if you’re stuck running Windows—well, you
can always install a Linux virtual machine.)
I certainly don’t consider myself a Unix god, or indeed any other
kind of OS deity. For all I know, neither did that local guru. But we
can all aspire at least to demigod status, and Conquering the
Command Line will help get you there. Please remember, though,
to use your new-found Unix powers for good, not for evil—with
great sudo comes great responsibility.

Michael Hartl
Author, The Ruby on Rails Tutorial

Preface
Who is This Book For?
This book is for new developers, experienced developers, and
everyone in between who wants to master Unix and Linux
commands. This book was designed to showcase some of the most
useful commands that a developer can know to help them in their
daily tasks.

This book is not an exhaustive manual for each of these commands;

in fact, it is just the opposite. Instead of just collecting up MAN
pages, this book pulls out the most useful flags, options, and
arguments for each of the commands in the book.

By learning and understanding the subset of flags, options, and

arguments for each command, you’ll be able to more efficiently use
each one in your daily development workflow - while understanding
where to look to find the more esoteric options you’ll only need
occasionally.

Code Examples
For most of the examples in this book we will use the Rails source
code as a way to demonstrate a lot of the commands we will look at.

The source code for Rails can easily be downloaded from GitHub
using Git.

$ git clone [email protected]:rails/rails.git

$ cd rails
If you don’t have Git installed, which you should, you can download
the zip of the Rails source code at
https://github.com/rails/rails/archive/master.zip.

Because Rails is an ever changing code base, your results may vary
from those in the book as the source code changes.

Acknowledgments
Books are not written in a vacuum, and this book was no exception.
It is easy to assume that since there is only one name on the cover
of this book, that there is only person responsible for bringing it all
to fruition - when in fact, this couldn’t be any further from the
truth.

I have a lot of people to thank for helping to get this book in your
hands, yourself included. If I didn’t think people such as yourself
would want to read this book, I wouldn’t have written it, so thank
you so much.

I would like to thank Michael Hartl, and the entire team at

Softcover, for developing a great set of publishing tools meant for
people such as myself, the technical writer. I have written two
books prior to this one, and I have to say that I have never used
tools as good as these for writing a book before. The platform just
got out of my way and let me focus on writing, and not on
constantly trying to figure out how to format things, why my PDF
wasn’t generating correctly, etc…

Next I would like to thank my fantastic team of technical reviewers.

The tech reviewers on this book were Pat Shaughnessy, Michael
Denomy, Pete Brown, Johnny Boursiqout, and Dan Pickett. They
tirelessly waded through each chapter as I wrote it, and they
submitted some amazing pull requests to improve both the quality
of the content as well as the prose itself. Their questions,
suggestions, subtractions, and additions, made this book
significantly better than I could have ever written on my own. I’m
eternally grateful to each of them for donating their time, and
mindshare, so freely. If you see them at a conference, user group, or
in the street, give them a big hug and say thanks.

Lastly, the most important thank you I can give is to my family. I

have to be the luckiest man on the planet to have such an amazing,
and understanding wife. She supports me, she pushes me, she
believes in me, and most importantly, she makes my life better. I
can’t express how much I owe to her for all she has done and given
to me, including our two beautiful boys. Thank you Rachel, as
always, this book is dedicated to you.

Well, enough gushing - I just hope you enjoy this book, and that it
helps you as much as I hope it will. Enjoy.

Introduction
The story of how this book came to be started in April of 2013 at
RailsConf in Portland, Oregon. At RailsConf I was hanging out with
Michael Hartl, author of the Rails Tutorial. Michael and I had
known each other for a few years through conferences, as well as
both being authors for Addison-Wesley.

Over dinner one night, Michael started telling me that he was

thinking of building a self-publishing platform for technical writers,
and was pumping me for information about what kind of a platform
I would like to write books on. I left RailsConf not thinking too
much more about Michael’s idea, as I had vowed to never write
another book.

Fast forward a couple of months to June, 2013. I was in Los Angeles

for a friend’s wedding. I drove up to Pasadena one night to meet
Michael for dinner, and the strongest Margaritas I have ever had.
During dinner Michael, as he’s prone to do, pulled out his laptop
and started to demonstrate his publishing platform to me. I have to
admit, I thought it looked really nice. He proceeded to spend the
rest of the night trying to convince me that I should be a guinea pig
for him and write another book, this time using his platform.

After I left Los Angeles, I headed back to Boston, packed quickly,

and boarded a plane to Stockholm with my wife and kids, to spend
the entire month of July working for a client I had there.

While at my client’s office, I had several conversations with some

developers who had no idea how to do even those most basic of
commands on their systems. They were petrified of the command
line. They could do simple stuff like change a directory, move a file,
etc., but they couldn’t un-tar a file, or find runaway processes. I
found this incredibly frustrating as most developers spend their
whole day in a terminal window, and here was a whole group of
people that weren’t getting the most they could from the powerful
tools at their disposal.

This lack of command line knowledge bothered me for several days.

I was obsessed with how these developers could get this far, without
this knowledge. I started to ask myself questions such as how had I
acquired these skills? The answer was I was lucky enough to have
great mentors over the years who passed on tiny snippets of
commands that were incredibly useful. I also thought about what
would keep people from learning how to use these commands.

I came to the conclusion that one of the biggest obstacles for

developers, particularly new developers, is the system that is
supposed to help developers understand these commands: man
pages. Man pages (man being short for manual) are supposed to be
manuals that explain these commands in great detail - including
every last option and flag. They are daunting, overwhelming, and
confusing beasts that bewilder even the most “senior” developer.

This realization came to me while I was standing in line to buy

tickets to the Vasa Museet. I realized what was needed was a book
that picked out the most useful commands a developer needs every
day. On top of that, the most useful flags and options need to be
explained in simple terms, with clear examples, that can be of
immediate use for a working developer.
As I stood in line, I shot off an email to Michael explaining my idea
and to get his thoughts on the subject matter and whether or not he
thought it would be a good fit for his Softcover platform. It was
seconds before he responded with an overwhelming “YES!”, and
that was how I became the first author to publish on Softcover.

I loved writing this book. It wasn’t always easy, no book ever is, but
it was incredibly educational. I found new tools, new flags, and new
options that have radically changed the way I work everyday. I have
found myself referring to this book, while I was writing it. I have
incorporated all of these tools into my daily work flow, and I hope
you do too.

So with that said, let’s dive right in.

Mark Bates
January, 2014
Chapter 1
Basics and Navigation
It’s important to have a firm grasp on the basics of how to use the
Unix (or Linux) command line. This includes navigating around
directories, copying files, creating files, etc.

For the more experienced developer, this chapter may be common

knowledge to you. If that’s the case, you can skip it, although you
might pick up one or two little things you didn’t already know.

For newer developers, this chapter contains information that will

become muscle memory after a while. You must know everything
in this chapter just to be able to do the most basic of things.

The rest of this book will assume you understand what is in this
chapter.

1.1 Home Directory (~)

On Unix and Linux systems, every user has a “home” directory. This
directory is yours and yours alone. The location of your home
directory varies depending on which operating system you’re on.
For example, on Mac OS X it’s /Users/<username>, but on Ubuntu
systems, it’s /home/<username>.

Finding your home directory is incredibly easy. When you first log
into your system you will automatically be placed in your home
directory by default. With this knowledge, you can use the pwd
command (Section 1.2) to get the path of your home directory.

Listing 1.1: $ pwd

/Users/markbates

Your home directory will be where your settings files get stored and
where your desktop is kept. The home directory (or subdirectories
under your home directory) is also where you will save documents
and files that are yours and not to be shared with others.

By default, your home directory will be unreadable and inaccessible

to all other users apart from yourself (and root). You can change
these permissions, if you want, but I would recommend against
making such a change.

It is quite common to see a tilde (~) in place of the full path for a
home directory. The ~ will expand to be the path of your home
directory when you use commands such as cd (Section 1.5).

1.2 Present Working

Directory (pwd)
Understanding where you are at any given time in your file system
is important. This information can help you build paths to other
files or help you if you get lost. To find out where you are you can
use the pwd command.

$ pwd

Listing 1.2
/Users/markbates/development/unix_commands

In Listing 1.2 you can see the directory I’m currently working in.
For me, it’s the directory that I’m writing this book in.

By default, pwd will list the “logical” path of your current directory.
This means it will treat symlinked paths (see Section 1.4.3) as if
they were the actual paths. For example, my development directory
doesn’t exist under my home folder, /Users/markbates; instead it is
symlinked to a folder inside of /Users/markbates/Dropbox.

If we want to see the actual physical path, with all of the symlinks
resolved, we can use the -P flag.

$ pwd -P

Listing 1.3
/Users/markbates/Dropbox/development/unix_commands

Listing 1.3 shows the actual full path to my current working

directory.
1.3 List Files and Directories
(ls)
Often it is useful to be able to see what files or directories are in
your current directory, or another directory. To accomplish this we
can use the ls command.

$ ls

Listing 1.4
Applications Calibre Library Desktop Documents Downloads
Dropbox Library Movies Music Pictures
Public Sites development screencasts scripts
tmp

Listing 1.4 shows the list of folders and files in my current directory,
which for me would be my home directory, /Users/markbates. I will
be using my home directory for all of the examples in this section,
unless otherwise noted.

1.3.1 Listing a Different Directory

By default, ls will operate in the current directory that you’re in.

However, you may want to find out what files are in another
directory, without first leaving your current directory. The ls
command will let you pass it a path to work on.

$ ls /usr/local/

Listing 1.5
 CONTRIBUTING.md Cellar Frameworks Library README.md
SUPPORTERS.md bin etc foreman go
heroku include lib opt sbin
share texlive var

Listing 1.5 shows the files and directories inside of the /usr/local
directory, without having to leave the current directory that I am in.

1.3.2 Listing All Files (-a)

So far we have seen all of the “visible” files and directories listed,
however, it is common on Unix and Linux based machines to have
“invisible”, or “hidden”, files that are prefixed with a “.” (dot).

Using the -a flag we can see both “visible” and “hidden” files and
directories.

$ ls -a

Listing 1.6
. .Trash .bashrc .ctags
.el4r .erlang.cookie .go .mplayer
.psql_history .rvm .swt .zshrc
Dropbox Sites .. .Xauthority
.cache .cups .el4rrc.rb .gem
.guard_history .netrc .railsrc .rvmrc
.vim Applications Library development
.CFUserTextEncoding .adobe .cisco .dbshell
.emacs .gemrc .gvimrc .node-gyp
.redis-commander .scala_history .viminfo Calibre Library
Movies screencasts .DS_Store .anyconnect
.cloud66 .dropbox .emacs.d .git-completion.sh
.heroku .npm .rediscli_history .softcover
.vimrc Desktop Music scripts
.MakeMKV .bash_history .codeintel .dvdcss
.ember .gitconfig .inkscape-etc .pry_history
.rnd .ssh .z Documents
Pictures tmp .NERDTreeBookmarks .bash_profile
.config .editedhistory .ember-data .gnome2
.irb-history .pryrc .rspec .subversion
.zlogin Downloads Public
Listing 1.6 now shows significantly more files and directories than
Listing 1.4 because the -a flag is including all of the “hidden” files,
as well as the “visible” ones.

1.3.3 Long Form Listing (-l)

So far in the section, whenever we have used the ls command, we

have been given back a list of names, arranged in a column. One of
the most useful flags for ls is the -l flag that will list out the names
of the files and directories as well as give more detailed information
about them.

$ ls -l

Listing 1.7
drwxr-xr-x 16 markbates staff 544 Aug 19 13:58 Applications
drwxr-xr-x 5 markbates staff 170 Dec 6 16:20 Calibre Library
lrwxr-xr-x 1 markbates staff 33 Jul 26 2012 Desktop ->
/Users/markbates/Dropbox/Desktop/
lrwxr-xr-x 1 markbates staff 35 Jul 26 2012 Documents ->
/Users/markbates/Dropbox/Documents/
drwx------+ 140 markbates staff 4760 Dec 16 13:44 Downloads
drwx------@ 24 markbates staff 816 Dec 15 17:24 Dropbox
drwx------@ 66 markbates staff 2244 Oct 23 08:29 Library
drwx------+ 5 markbates staff 170 Sep 15 18:06 Movies
drwx------+ 6 markbates staff 204 Mar 5 2013 Music
drwx------+ 10 markbates staff 340 Jul 31 18:20 Pictures
drwxr-xr-x+ 4 markbates staff 136 Mar 5 2013 Public
drwxr-xr-x+ 3 markbates staff 102 Mar 5 2013 Sites
lrwxr-xr-x 1 markbates staff 37 Jul 26 2012 development ->
/Users/markbates/Dropbox/development/
lrwxr-xr-x 1 markbates staff 37 Jun 1 2013 screencasts ->
/Users/markbates/Dropbox/screencasts/
lrwxr-xr-x 1 markbates staff 33 Jul 26 2012 scripts ->
/Users/markbates/Dropbox/scripts/
drwxr-xr-x 19 markbates staff 646 Dec 6 16:07 tmp

In Listing 1.7 we see a significant amount of information being

presented to us for each file and directory.
If the -l option is given, the following information is displayed for
each file: file mode, number of links, owner name, group name,
number of bytes in the file, abbreviated month, day-of-month file
was last modified, hour file last modified, minute file last modified,
and the pathname. We can also see that some paths are followed by
-> and another path. These paths are symlinked to the directories
that follow the ->. See Section 1.4.3 for more information on
symbolic links.

1.3.4 Human Readable Sizes (-h)

In Section 1.3.3 we used the -l flag to give us a more detailed view

of files and directories in our current directory, including the
number of bytes in the file.

While knowing the number of bytes can be useful, I find it more

useful to see the size of the file in human readable terms, such as
1.7K or 35M. Thankfully ls will give us this information if we use
the -h flag.

$ ls -lh

Listing 1.8
drwxr-xr-x 16 markbates staff 544B Aug 19 13:58 Applications
drwxr-xr-x 5 markbates staff 170B Dec 6 16:20 Calibre Library
lrwxr-xr-x 1 markbates staff 33B Jul 26 2012 Desktop ->
/Users/markbates/Dropbox/Desktop/
lrwxr-xr-x 1 markbates staff 35B Jul 26 2012 Documents ->
/Users/markbates/Dropbox/Documents/
drwx------+ 140 markbates staff 4.6K Dec 16 13:44 Downloads
drwx------@ 24 markbates staff 816B Dec 15 17:24 Dropbox
drwx------@ 66 markbates staff 2.2K Oct 23 08:29 Library
drwx------+ 5 markbates staff 170B Sep 15 18:06 Movies
drwx------+ 6 markbates staff 204B Mar 5 2013 Music
drwx------+ 10 markbates staff 340B Jul 31 18:20 Pictures
drwxr-xr-x+ 4 markbates staff 136B Mar 5 2013 Public
 drwxr-xr-x+ 3 markbates staff 102B Mar 5 2013 Sites
lrwxr-xr-x 1 markbates staff 37B Jul 26 2012 development ->
/Users/markbates/Dropbox/development/
lrwxr-xr-x 1 markbates staff 37B Jun 1 2013 screencasts ->
/Users/markbates/Dropbox/screencasts/
lrwxr-xr-x 1 markbates staff 33B Jul 26 2012 scripts ->
/Users/markbates/Dropbox/scripts/
drwxr-xr-x 19 markbates staff 646B Dec 6 16:07 tmp

In Listing 1.8 we see we now have an easier to understand

description of the size of the files in the directory.

1.3.5 Sorting by Size (-S)

Another useful flag we can use with ls is -S, which will sort the
results of ls by file size, instead of the default sorting by name.

$ ls -lhS

Listing 1.9
drwx------+ 140 markbates staff 4.6K Dec 16 13:44 Downloads
drwx------@ 66 markbates staff 2.2K Oct 23 08:29 Library
drwx------@ 24 markbates staff 816B Dec 15 17:24 Dropbox
drwxr-xr-x 19 markbates staff 646B Dec 6 16:07 tmp
drwxr-xr-x 16 markbates staff 544B Aug 19 13:58 Applications
drwx------+ 10 markbates staff 340B Jul 31 18:20 Pictures
drwx------+ 6 markbates staff 204B Mar 5 2013 Music
drwxr-xr-x 5 markbates staff 170B Dec 6 16:20 Calibre Library
drwx------+ 5 markbates staff 170B Sep 15 18:06 Movies
drwxr-xr-x+ 4 markbates staff 136B Mar 5 2013 Public
drwxr-xr-x+ 3 markbates staff 102B Mar 5 2013 Sites
lrwxr-xr-x 1 markbates staff 37B Jul 26 2012 development ->
/Users/markbates/Dropbox/development/
lrwxr-xr-x 1 markbates staff 37B Jun 1 2013 screencasts ->
/Users/markbates/Dropbox/screencasts/
lrwxr-xr-x 1 markbates staff 35B Jul 26 2012 Documents ->
/Users/markbates/Dropbox/Documents/
lrwxr-xr-x 1 markbates staff 33B Jul 26 2012 Desktop ->
/Users/markbates/Dropbox/Desktop/
lrwxr-xr-x 1 markbates staff 33B Jul 26 2012 scripts ->
/Users/markbates/Dropbox/scripts/

In Listing 1.9 the results of ls are displayed with the largest files at
the top, and the smallest files at the bottom.
1.3.6 Sorting by Last Modified Time (-t)

It can often be useful to be able to sort your ls results by the last

time the files were modified. To do this we can use the -t flag.

$ ls -lt

Listing 1.10
drwx------+ 140 markbates staff 4760 Dec 16 13:44 Downloads
drwx------@ 24 markbates staff 816 Dec 15 17:24 Dropbox
drwxr-xr-x 5 markbates staff 170 Dec 6 16:20 Calibre Library
drwxr-xr-x 19 markbates staff 646 Dec 6 16:07 tmp
drwx------@ 66 markbates staff 2244 Oct 23 08:29 Library
drwx------+ 5 markbates staff 170 Sep 15 18:06 Movies
drwxr-xr-x 16 markbates staff 544 Aug 19 13:58 Applications
drwx------+ 10 markbates staff 340 Jul 31 18:20 Pictures
lrwxr-xr-x 1 markbates staff 37 Jun 1 2013 screencasts ->
/Users/markbates/Dropbox/screencasts/
drwx------+ 6 markbates staff 204 Mar 5 2013 Music
drwxr-xr-x+ 4 markbates staff 136 Mar 5 2013 Public
drwxr-xr-x+ 3 markbates staff 102 Mar 5 2013 Sites
lrwxr-xr-x 1 markbates staff 33 Jul 26 2012 scripts ->
/Users/markbates/Dropbox/scripts/
lrwxr-xr-x 1 markbates staff 35 Jul 26 2012 Documents ->
/Users/markbates/Dropbox/Documents/
lrwxr-xr-x 1 markbates staff 33 Jul 26 2012 Desktop ->
/Users/markbates/Dropbox/Desktop/
lrwxr-xr-x 1 markbates staff 37 Jul 26 2012 development ->
/Users/markbates/Dropbox/development/

1.3.7 Reverse Sort (-r)

When you are listing a directory that contains many files, sorting
with ls is a great way to help quickly find the files or directories you
are looking for. By default, ls sorts all of its results alphabetically.
We also learned how to sort results by size (Section 1.3.5) and by
last modified time (Section 1.3.6).

Using the -r flag we are able to reverse the results of ls.

 $ ls -lr

Listing 1.11
drwxr-xr-x 19 markbates staff 646 Dec 6 16:07 tmp
lrwxr-xr-x 1 markbates staff 33 Jul 26 2012 scripts ->
/Users/markbates/Dropbox/scripts/
lrwxr-xr-x 1 markbates staff 37 Jun 1 2013 screencasts ->
/Users/markbates/Dropbox/screencasts/
lrwxr-xr-x 1 markbates staff 37 Jul 26 2012 development ->
/Users/markbates/Dropbox/development/
drwxr-xr-x+ 3 markbates staff 102 Mar 5 2013 Sites
drwxr-xr-x+ 4 markbates staff 136 Mar 5 2013 Public
drwx------+ 10 markbates staff 340 Jul 31 18:20 Pictures
drwx------+ 6 markbates staff 204 Mar 5 2013 Music
drwx------+ 5 markbates staff 170 Sep 15 18:06 Movies
drwx------@ 66 markbates staff 2244 Oct 23 08:29 Library
drwx------@ 24 markbates staff 816 Dec 15 17:24 Dropbox
drwx------+ 140 markbates staff 4760 Dec 16 13:44 Downloads
lrwxr-xr-x 1 markbates staff 35 Jul 26 2012 Documents ->
/Users/markbates/Dropbox/Documents/
lrwxr-xr-x 1 markbates staff 33 Jul 26 2012 Desktop ->
/Users/markbates/Dropbox/Desktop/
drwxr-xr-x 5 markbates staff 170 Dec 6 16:20 Calibre Library
drwxr-xr-x 16 markbates staff 544 Aug 19 13:58 Applications

In Listing 1.11 we can see that default sort of ls, alphabetically, has
now been reversed with tmp at the top of the list, and Applications
at the bottom.

We can also use the -r flag with other options to reverse their sort
order. For example, ls -lSr will list the files by size with smaller
files listed first.

1.4 Links (ln)

Links allow us to create an association from one file or directory to
another. A link allows us to “point” one file or directory to another.
This can be quite useful for maintaining a sane file system, or for
having multiple versions of a file or directory, and not wanting to
use disk space to store copies of those files.

An example of where links can come in handy is when deploying a

web application. Let’s say when you deploy your application you
create a new directory whose name is the current time, and you put
your application code into that directory. After your code is there,
you would have to edit your web server configuration to tell it to
point to the new directory where your latest code is. However, using
links you can create a link called “current” that your web server
points at, and you can change where the “current” link points to - in
this case the last timestamped folder you created for your code.

Links can either be “hard” or “symbolic” as we’ll see in the following

sections.

1.4.1 Hard Links

We can use the ln command to create a link between two files. By

default, the ln command will create a hard link between these files.

Hard links create an identical copy of the linked file on disk, that
gets updated automatically as the source file gets updated. That
means if the content of the source is changed, so will the target file.

To create a link to a file, we call the ln command - first giving it the

name of the file we want to link to (our source file), followed by the
name of the linked file we want to create (the target).

$ ln a.txt b.txt
Listing 1.12: $ ls -l
-rw-r--r-- 2 markbates staff 12 Dec 17 16:46 a.txt
-rw-r--r-- 2 markbates staff 12 Dec 17 16:46 b.txt

Listing 1.12 shows the newly created b.txt has all of the same
properties as the a.txt file.

An important thing to note about hard links is that they only work
on the current file system. You can not create a hard link to a file on
a different file system. To do that you need to use symbolic links,
Section 1.4.3.

While the content of the source and target files are linked, if the
source file gets deleted, the target file will continue to live as an
independent file, despite the severed link.

$ rm a.txt

Listing 1.13: $ ls -l
-rw-r--r-- 1 markbates staff 12 Dec 17 17:01 b.txt

In Listing 1.13 we can see that the b.txt file still exists, despite the
target file having been deleted.

If we were to use the cat tool from Chapter 9 Section 9.2, we can see
that the contents of b.txt are still intact.

$ cat b.txt

Listing 1.14
Hello World
1.4.2 Forcing a Link (-f)

If the source file already exists you will get an error, like that in
Listing 1.15.

$ ln a.txt b.txt

Listing 1.15
ln: b.txt: File exists

To fix the error in Listing 1.15 you can pass the -f flag to force the
link.

$ ln -f a.txt b.txt

1.4.3 Symbolic Links (-s)

We have seen that the default type of link that gets created when
using ln is that of a hard link. Unfortunately hard links do not work
for directories. To create a link to a directory, we can use the -s flag
to create a symbolic link. This can also be used for linking to files as
well, not just directories.

Symbolic links can also link to files or directories on other file

systems. This makes symbolic links (symlinks) more powerful, and
more common than the default hard links.

To create a symbolic link we can use the -s flag.

$ ln -s a.txt b.txt
Listing 1.16: $ ls -l
-rw-r--r-- 1 markbates staff 12 Dec 17 17:01 a.txt
lrwxr-xr-x 1 markbates staff 5 Dec 17 17:12 b.txt -> a.txt

As Listing 1.16 shows we have created a symbolic link from a.txt to

b.txt. See Section 1.3.3 for an explanation of how to read
Listing 1.16.

Listing 1.17 shows the results of creating a hard link, and

Listing 1.18 shows the results of a symbolic link.

Listing 1.17: $ ln a.txt b.txt

-rw-r--r-- 2 markbates staff 12 Dec 17 16:46 a.txt
-rw-r--r-- 2 markbates staff 12 Dec 17 16:46 b.txt

Listing 1.18: $ ln -s a.txt b.txt

-rw-r--r-- 1 markbates staff 12 Dec 17 17:01 a.txt
lrwxr-xr-x 1 markbates staff 5 Dec 17 17:12 b.txt -> a.txt

Notice that the file timestamps and size are identical in Listing 1.17
with the hard link, but they are different in Listing 1.18 for the
symbolic link. The reason for the difference is that with the
symbolic link the operating system creates a new, small file that
points to the target file or directory.

1.5 Change Directories (cd)

While staying in the same directory all of the time might seem like
fun, eventually you’ll need to change, or navigate, into another
directory. To do this we can use the very simple cd command.

cdtakes one argument - the path to the directory you wish to

navigate to.

$ cd ~/Documents

Listing 1.19: $ pwd

/Users/markbates/Documents

In Listing 1.19 I have successfully navigated to my Documents

directory, and we can use the pwd command (Section 1.2) to prove it.

1.5.1 Navigating Up (..)

Once inside of a directory, it can be useful to be able to navigate

back up a directory. One way to do this would be to call cd giving it
the full path to the directory you want to go. Another way to
accomplish the task of moving up to the parent directory is to use
the special path ...

$ cd ..

Telling cd to navigate to .. will cause it to navigate to the parent

directory of the current directory.

1.5.2 Navigating to the Home Directory

When you are in a directory and want to navigate back to your home
directory, you can call cd without any arguments.
 $ cd

Listing 1.20: $ pwd

$ /Users/markbates

1.6 Creating Directories

(mkdir)
Directories are a great place to store and organize files, and creating
them is easy. To create a directory we can use the mkdir command
followed by the name of the directory we wish to create.

$ mkdir foo

Listing 1.21: $ ls -l
drwxr-xr-x 2 markbates staff 68 Dec 18 13:20 foo

1.6.1 Create Intermediate Directories (-p)

The mkdir command will allow us to create nested directories using

the -p flag.

$ mkdir -p a/b/c

Listing 1.22: $ ls -la a/b/c

drwxr-xr-x 2 markbates staff 68 Dec 18 13:23 .
drwxr-xr-x 3 markbates staff 102 Dec 18 13:23 ..
In Listing 1.22 we can use the ls command (Section 1.3) to see that
the c directory exists and is currently empty.

1.6.2 Verbose Output (-v)

The mkdir command supports a verbose mode flag, -v. Adding the -
v flag will print the results of mkdir to the console.

$ mkdir -v a

Listing 1.23
mkdir: created directory 'a'

Using the -v flag can provide useful reporting when writing scripts
that will execute a lot of commands for you.

1.7 Copying Files (cp)

Copying files and directories can be accomplished using the cp
command.

1.7.1 Copying a Single File

To copy a single file we can call the cp command with two

arguments. The first argument is the file we want to copy, the
source file. The second argument, or target, is the location we want
to copy the source file to. This target can either be the name of the
file you wish to copy to, or the directory you wish to copy to. If you
specify just a directory then the original file name of the source file
will be used to create the new file in the target directory.

$ cp a.txt b.txt

Listing 1.24: $ ls -l
-rw-r--r-- 1 markbates staff 0 Dec 18 13:34 a.txt
-rw-r--r-- 1 markbates staff 0 Dec 18 13:41 b.txt

1.7.2 Copying Multiple Files

cp will allows you to list several files you would like to copy. When
you do this, however, the last argument must be a directory, and the
original file names of the source files will be used as the names of
the new files in the target directory.

$ cp a.txt b.txt foo

Listing 1.25: $ ls -l foo

-rw-r--r-- 1 markbates staff 0 Dec 18 13:47 a.txt
-rw-r--r-- 1 markbates staff 0 Dec 18 13:47 b.txt

We can also pass simple patterns to cp to achieve the same results

as Listing 1.25.

$ cp *.txt foo

Listing 1.26: $ ls -l foo

-rw-r--r-- 1 markbates staff 0 Dec 18 13:47 a.txt
-rw-r--r-- 1 markbates staff 0 Dec 18 13:47 b.txt
1.7.3 Verbose Output (-v)

The cp command will print verbose output regarding the files that
were copied using the -v flag.

$ cp -v a.txt b.txt

Listing 1.27
a.txt -> b.txt

1.7.4 Copying Directories (-R)

By default cp expects the source to be copied to be a file. If we try to

copy a directory to another directory we will get the error we see in
Listing 1.28.

$ cp foo bar

Listing 1.28
cp: foo is a directory (not copied).

To fix the error in Listing 1.28, we can use the -R flag to recursively
copy the directory’s contents to the new directory.

$ cp -Rv foo bar

Listing 1.29
foo -> bar
foo/a.txt -> bar/a.txt
foo/b.txt -> bar/b.txt
foo/c.txt -> bar/c.txt
Using the verbose flag, -v, we can see in Listing 1.29 all of the files
from the source directory, foo, have been copied to the target
directory, bar.

1.7.5 Force Overwriting of a File (-f)

In Listing 1.30 we can see that we have two files, a.txt and b.txt.

$ ls -l

Listing 1.30
-rw-r--r-- 1 markbates staff 0 Dec 18 13:58 a.txt
-rw-r--r-- 1 root staff 6 Dec 18 14:55 b.txt

When looking at Listing 1.30 we can see that the two files listed are
owned by two different users. The first file, a.txt, is owned by user
markbates, while the second file is owned by root.

What would happen if we tried to copy a.txt to b.txt?

$ cp a.txt b.txt

Listing 1.31
cp: b.txt: Permission denied

As we can see in Listing 1.31 we get an error when trying to

overwrite b.txt with a.txt. We don’t have permission to do that.
When errors such as this occur we can use the -f flag to force the
copying of the source file to the target file.

$ cp -f a.txt b.txt
Listing 1.32: $ ls -l
-rw-r--r-- 1 markbates staff 0 Dec 18 13:58 a.txt
-rw-r--r-- 1 markbates staff 0 Dec 18 14:57 b.txt

Listing 1.32 shows that by using the -f flag we were able to

successfully overwrite the target file, b.txt, with the contents of the
source file, a.txt.

1.7.6 Confirm Overwriting of a File (-i)

When you are about to copy over several files at once, and some of
the target files already exist, it may be beneficial to confirm that you
do in fact want to copy the source file and replace the target file.

cp will happily prompt you to confirm that you are about to

overwrite another file if you use the -i flag.

$ cp -i a.txt b.txt

Listing 1.33
overwrite b.txt? (y/n [n])

In Listing 1.33 we are presented with a prompt asking us to confirm,

y or n, whether to overwrite the target file. A response of y will
overwrite the file. A response of n will skip the file and move on to
the next copy, if there is one.

1.8 Deleting Files (rm)

The rm command is used to delete files and folders. It supports the
same flags and arguments as the cp command that we learned about
in Section 1.7.

$ rm -v a.txt

Listing 1.34
a.txt

Listing 1.34 uses the -v flag to list files that were deleted using the
rm command.

1.9 Moving Files (mv)

The process for moving files is almost identical to that of copying
files, which we learned about in Section 1.7.

mv supports the same flags as cp. In reality the mv command is really

just a combination of cp and rm to achieve the desired outcome of
moving a file from one location to another.

$ mv -v a.txt b.txt

Listing 1.35
a.txt -> b.txt

We can achieve the same effect from Listing 1.35 by using the cp
and rm commands. This can be useful when trying to move a file
across file systems, as the mv command doesn’t support that.

$ cp a.txt b.txt
$ rm a.txt

Listing 1.36: $ ls -l
-rw-r--r-- 1 markbates staff 0 Dec 18 16:20 b.txt

Listing 1.36 shows that we were able to replicate the behavior of mv

using the cp and rm commands.

1.10 Input/Output (|, >)

The Unix philosophy, which you’ll hear repeated often, is “do one
thing, and do it well”. This theme is evident as we move through
this book. Each command covered does one thing, and does it well.

1.10.1 Redirecting Output (|)

With these seemingly small individual commands, we can build

some pretty impressive work flows by redirecting the output of one
command to the input to another command. This is made possible
by using the “pipe” operator, |.

In the following example, we “pipe” the output of the ls command

to the input of the grep command to find all the files in my home
directory that contain an underscore, _.

$ ls -a ~ | grep _
Listing 1.37
.DS_Store
.bash_history
.bash_profile
.guard_history
.pry_history
.psql_history
.rediscli_history
.scala_history

When using the | operator we can chain together any number of

commands. For example, we can take the output of Listing 1.37 and
pass it to the sed command from Chapter 7 and change all of the
underscores to dashes.

$ ls -a ~ | grep _ | sed "s/_/-/g"

Listing 1.38
.DS-Store
.bash-history
.bash-profile
.guard-history
.pry-history
.psql-history
.rediscli-history
.scala-history

Listing 1.38 shows the result of chaining together the ls, grep, and
sed commands using the | operator.

1.10.2 Writing to a File (>)

In addition to redirecting the output from one process and sending

it to another process, we can also write that output to a file using
the > operator.
 $ ls -a ~ | grep _ > underscores.txt

Listing 1.39: $ cat underscores.txt

.DS_Store
.bash_history
.bash_profile
.guard_history
.pry_history
.psql_history
.rediscli_history
.scala_history

Listing 1.39 shows the contents of the underscores.txt file (using

cat from Chapter 9 Section 9.2), which contains the results of our
search for files in the home directory that contain underscores.

1.10.3 Reading from a File (<)

In Section 1.10.2 we saw that we can use > to write data to a file. If
we instead wanted to read data from a file, we can use <.

In Chapter 9, Section 9.6, we see a great example of this using the

pbpaste command.

1.11 Conclusion
In this chapter we covered the basics of the command line, from
how to navigate between directories, to how to move, delete, or copy
files, and more.

Hopefully this chapter has provided you with the solid footing that
you’ll need to conquer the command line. The commands in this
chapter will become second nature to you over time, as you’ll use
them continuously each day.
Chapter 2
Ack/Ag
Searching large code bases is something that most of us do on a
daily basis. Whether it be to refactor some code, or to simply find
where a variable, class, or method has been implemented. Good
searching tools are the lifeblood of development, and bad searching
tools can make our jobs more difficult than they need be.

In this chapter, we’ll look at two tools that can help make searching
a large number of files fast and easy. Those tools are Ack and Ag.
Ack and Ag are 99% feature compatible with each other, so you can
use them interchangeably.

Ag is significantly faster than Ack, and it has built-in version control

system (VCS) awareness that make it particularly useful for
software developers. I will be using Ag throughout this chapter, but
feel free and use Ack if that is all that is available on your system.

We’ll want a sizable code base to search through to show the power
of ack/ag, so we’ll be using the Rails source code, as mentioned in
the Preface.

2.1 Installation
There’s a very good chance that you already have Ack installed on
your machine already, but it is nice to be up-to-date with the latest
version.

If you are on a Mac, the best and easiest way to install either Ack or
Ag is using Homebrew.

$ brew install ack

$ brew install ag

If you are on a different platform, please see the install directions

on the project’s page for the correct installation instructions for
your platform.

2.2 The Basics

Let’s start by taking a look at the most basic type of search we can
do using Ack or Ag.

The creator of the Ruby on Rails framework is a man who goes by

the initials of DHH. Let’s see if we can find any occurrence of those
initials in the entire Rails code base.

$ ag DHH

The result of our search should look something like Listing 2.11 :

Listing 2.1
actionview/lib/action_view/helpers/atom_feed_helper.rb
43: # author.name("DHH")
76: # author.name("DHH")
 actionview/test/template/atom_feed_helper_test.rb
25: author.name("DHH")
42: author.name("DHH")
59: author.name("DHH")
71: author.name("DHH")
95: author.name("DHH")
113: author.name("DHH")
131: author.name("DHH")
149: author.name("DHH")
171: author.name("DHH")
191: author.name("DHH")
279: assert_select "author name", :text => "DHH"

activerecord/CHANGELOG.md
58: *DHH*
67: *DHH*
92: *DHH*

activerecord/lib/active_record/relation.rb
100: # users = User.where(name: 'DHH')
101: # user = users.new # => #<User id: nil, name: "DHH", created_at: ...

activerecord/lib/active_record/relation/query_methods.rb
685: # users = users.create_with(name: 'DHH')
686: # users.new.name # => 'DHH'

activerecord/test/cases/xml_serialization_test.rb
388: assert_no_match %r{^ <copyright>DHH</copyright>}, xml
389: assert_match %r{^ {6}<copyright>DHH</copyright>}, xml

activesupport/CHANGELOG.md
30: *DHH*
98: *DHH*
122: *DHH*

guides/source/caching_with_rails.md
33:INFO: Page Caching has been removed from Rails 4. ...
39:INFO: Action Caching has been removed from Rails 4. ...

guides/source/migrations.md
109:`YYYYMMDDHHMMSS_create_products.rb`, that is to say a UTC timestamp
227:`db/migrate/YYYYMMDDHHMMSS_add_details_to_products.rb` file.

As you can see, the first argument we pass in is the search term we
want to find in our directory of files. By default, this query is run
recursively against the current directory.

Should we want to specify a path to search against, we can pass in a

final argument. For example, if we just wanted to search within the
guides directory, we can modify our query to look like this:
 $ ag DHH guides/

Our result will now be scoped to just that directory as seen in

Listing 2.22

Listing 2.2
guides/source/caching_with_rails.md
33:INFO: Page Caching has been removed from Rails 4. ...
39:INFO: Action Caching has been removed from Rails 4. ...

guides/source/migrations.md
109:`YYYYMMDDHHMMSS_create_products.rb`, that is to say a UTC timestamp
227:`db/migrate/YYYYMMDDHHMMSS_add_details_to_products.rb` file.

2.2.1 Regular Expression Searches

Both Ack and Ag support a limited subset of regular expression

syntax. This can be incredibly useful for crafting more fine-grained
search queries.

Let’s search for any occurrence of the word readme at the end of a
line.

$ ag readme$

Listing 2.3 lists the results of our successful query.

Listing 2.3
railties/lib/rails/generators/rails/app/app_generator.rb
40: def readme

railties/lib/rails/generators/rails/plugin/plugin_generator.rb
32: def readme

railties/test/generators/actions_test.rb
203: def test_readme
2.2.2 Literal Expression Searches

Both Ack and Ag treat the search term as a regular expression by

default. This means that if we were to search for the pattern .rb it
would match any single character followed by rb. This might not be
what we want. Let’s search for .rb in the
railties/CHANGELOG.md file.

$ ag .rb railties/CHANGELOG.md

Listing 2.4
70:* Rename `commands/plugin_new.rb` to `commands/plugin.rb` ...
78:* Omit turbolinks configuration completely on skip_javascript ...
103:* Changes repetitive th tags to use colspan attribute ...

Listing 2.4 lists the results3 of our query. Since Ag matches the
pattern as a regular expression, our results includes some things we
don’t want, such as turbolinks.

To fix this we can change our query to the use -Q flag. This will
search for the exact pattern, .rb, in the file.

$ ag -Q .rb railties/CHANGELOG.md

Listing 2.5 lists the results4 of our successful query.

Listing 2.5
70:* Rename `commands/plugin_new.rb` to `commands/plugin.rb` ...

While regular expressions are somewhat outside the scope of this

book, Mastering Regular Expressions is an excellent resource if you
need to learn more. If you prefer interactive tutorials, RegexOne
provides a great walk-through.

2.3 Listing Files (-l)

By default, when performing a search with Ag or Ack, the results
include the line number as well as the line where the term was
found.

This level of detail is incredibly useful in a lot of cases, but what if

we really just want to see a list of the file names, without seeing the
context in which the term was found? The -l flag can help us with
this.

$ ag DHH -l

The result set is now presented as just a list of files, Listing 2.6.

Listing 2.6
actionview/lib/action_view/helpers/atom_feed_helper.rb
actionview/test/template/atom_feed_helper_test.rb
activerecord/CHANGELOG.md
activerecord/lib/active_record/relation.rb
activerecord/lib/active_record/relation/query_methods.rb
activerecord/test/cases/xml_serialization_test.rb
activesupport/CHANGELOG.md
guides/source/caching_with_rails.md
guides/source/migrations.md

The list of files presented to us is now a little bit more manageable

and can give us a much better sense of the number of files that are
in the result set for our query.
2.4 Case Insensitive Searches
(-i)
By default, all search queries are case sensitive, but we can easily
change that using the -i flag.

To demonstrate this, let’s change our search query. This time, we’ll
search for the term readme. We’ll also use the -l flag that we
learned about earlier to keep our result set easier to read.

$ ag readme -l

You can see the results of our query in Listing 2.7.

Listing 2.7
guides/code/getting_started/Gemfile
guides/source/asset_pipeline.md
guides/source/generators.md
guides/source/getting_started.md
railties/lib/rails/generators/actions.rb
railties/lib/rails/generators/app_base.rb
railties/lib/rails/generators/rails/app/app_generator.rb
railties/lib/rails/generators/rails/plugin/plugin_generator.rb
railties/test/generators/actions_test.rb

Now let’s add the -i flag to our query and see how that changes our
result set.

$ ag readme -l -i

Listing 2.8 shows the results of adding the -i flag to our search
query.

Listing 2.8
 actionmailer/actionmailer.gemspec
actionpack/actionpack.gemspec
actionpack/test/controller/action_pack_assertions_test.rb
actionview/actionview.gemspec
activemodel/activemodel.gemspec
activerecord/activerecord.gemspec
activerecord/lib/active_record/base.rb
activesupport/activesupport.gemspec
guides/code/getting_started/Gemfile
guides/code/getting_started/README.rdoc
guides/source/command_line.md
guides/source/asset_pipeline.md
guides/source/generators.md
guides/source/getting_started.md
guides/source/plugins.md
guides/source/rails_application_templates.md
guides/source/working_with_javascript_in_rails.md
rails.gemspec
railties/lib/rails/api/task.rb
railties/lib/rails/generators/actions.rb
railties/lib/rails/generators/app_base.rb
railties/lib/rails/generators/rails/app/app_generator.rb
railties/lib/rails/generators/rails/app/templates/README.rdoc
railties/lib/rails/generators/rails/app/USAGE
railties/lib/rails/generators/rails/plugin/plugin_generator.rb
railties/lib/rails/generators/rails/plugin/templates/%name%.gemspec
railties/lib/rails/generators/rails/plugin/templates/rails/javascripts.js
railties/lib/rails/generators/rails/plugin/templates/Rakefile
railties/lib/rails/generators/rails/plugin/USAGE
railties/lib/rails/tasks/documentation.rake
railties/railties.gemspec
railties/RDOC_MAIN.rdoc
railties/test/generators/actions_test.rb
railties/test/generators/app_generator_test.rb
railties/test/generators/plugin_generator_test.rb
README.md

As you can see, ag returns significantly more files now using a case
insensitive search.

2.5 Scoping to Files (-G)

Quite often, we will want to search only within certain file types, or
within files that have a specific file extension. To accomplish this
we can use the -G flag5 .
Let’s once again search for readme in our code base, but this time
we’ll limit our search results to filenames that contain the word
action.

$ ag readme -l -G action

As you can see in Listing 2.9 we have significantly reduced the

number of files that are returned as the result of our query.

Listing 2.9
railties/lib/rails/generators/actions.rb
railties/test/generators/actions_test.rb

Let’s modify our query slightly and search for files that end with ec.

$ ag readme -l -i -G ec

As Listing 2.10 shows us, we don’t quite have our query tuned the
way we want it, since it also returned files like
activerecord/lib/active_record/base.rb, matching the ec in
record.

Listing 2.10
actionmailer/actionmailer.gemspec
actionpack/actionpack.gemspec
actionview/actionview.gemspec
activemodel/activemodel.gemspec
activerecord/activerecord.gemspec
activerecord/lib/active_record/base.rb
activesupport/activesupport.gemspec
rails.gemspec
railties/lib/rails/generators/rails/plugin/templates/%name%.gemspec
railties/railties.gemspec
Instead of finding files that end in ec, the search results are showing
us files that contain ec anywhere in the name. How do we fix our
query to get back the results we want?

The -G flag allows us to pass in basic regular expressions, so we can

easily modify our search query to improve the results. For those
unfamiliar with regular expressions, we can use the $ to indicate
that we only want to match filenames that end with ec.

$ ag readme -l -i -G ec$

Listing 2.11
actionmailer/actionmailer.gemspec
actionpack/actionpack.gemspec
actionview/actionview.gemspec
activemodel/activemodel.gemspec
activerecord/activerecord.gemspec
activesupport/activesupport.gemspec
rails.gemspec
railties/lib/rails/generators/rails/plugin/templates/%name%.gemspec
railties/railties.gemspec

As you can see in Listing 2.11 our search results now only contain
files that end with ec. Success!

2.6 Ignoring Paths

Another important way to filter our search results is to ignore
certain directories that we don’t care about. Both Ack and Ag have a
few useful features that help us to do just that.

2.6.1 The –ignore-dir Flag

Let’s start with this query and it’s result set, Listing 2.12

$ ag readme -l

Listing 2.12
guides/code/getting_started/Gemfile
guides/source/asset_pipeline.md
guides/source/generators.md
guides/source/getting_started.md
railties/lib/rails/generators/actions.rb
railties/lib/rails/generators/app_base.rb
railties/lib/rails/generators/rails/app/app_generator.rb
railties/lib/rails/generators/rails/plugin/plugin_generator.rb
railties/test/generators/actions_test.rb

What if we aren’t interested in seeing any results from the

railties/lib directory? To solve that problem we can use the –
ignore-dir flag.

$ ag readme -l --ignore-dir=railties/lib

Listing 2.13 shows that we have successfully filtered out those files.
All results from files located in that directory will be omitted.

Listing 2.13
guides/code/getting_started/Gemfile
guides/source/asset_pipeline.md
guides/source/generators.md
guides/source/getting_started.md
railties/test/generators/actions_test.rb

We can easily chain multiple –ignore-dir calls together to filter our

results further.

$ ag readme -l --ignore-dir=railties/lib --ignore-dir=guides/code

See Listing 2.14 for the results of using multiple ignore flags.
Listing 2.14
guides/source/asset_pipeline.md
guides/source/generators.md
guides/source/getting_started.md
railties/test/generators/actions_test.rb

Despite its name, the –ignore-dir flag will also let us filter out
particular files, not just directories.

$ ag readme -l --ignore-dir="*.rb"

Listing 2.15
guides/code/getting_started/Gemfile
guides/source/asset_pipeline.md
guides/source/generators.md
guides/source/getting_started.md

In Listing 2.15 we, again searched for the term readme, except this
time we ignored all files that matched the pattern *.rb.

2.6.2 VCS Ignore Files (Ag only)

Ag has an important feature that distinguishes it from Ack. It will

automatically read in VCS (version control system) files, such as
.gitignore for Git, and filter out files from the result set based on
the content of those files. For the uninitiated, the .gitignore file
tells git what files should not be tracked by source control.

As an example, the Rails source code already has a .gitignore file,

so let’s add the railties/lib and guides/code directories to it so
the file now looks Listing 2.16.

Listing 2.16
 debug.log
.Gemfile
/.bundle
/.ruby-version
/Gemfile.lock
pkg
/dist
/doc/rdoc
/*/doc
/*/test/tmp
/activerecord/sqlnet.log
/activemodel/test/fixtures/fixture_database.sqlite3
/activesupport/test/fixtures/isolation_test
/railties/test/500.html
/railties/test/fixtures/tmp
/railties/test/initializer/root/log
/railties/doc
/railties/tmp
/guides/output
railties/lib
guides/code

Now if we were to run the same query, but without the –ignore-dir
flags, as shown in Listing 2.17, ag would return the same results as
Listing 2.14, but with significantly less typing.

$ ag readme -l

Listing 2.17
guides/source/asset_pipeline.md
guides/source/generators.md
guides/source/getting_started.md
railties/test/generators/actions_test.rb

Of course, there might be times when you don’t want to use this
feature. Thankfully, there is a flag we can pass to Ag to tell it to
ignore any VCS files.

$ ag readme -l --skip-vcs-ignores

Using the –skip-vcs-ignores flag we are able to get the original

results of our query, as seen in Listing 2.18
Listing 2.18
guides/code/getting_started/Gemfile
guides/source/asset_pipeline.md
guides/source/generators.md
guides/source/getting_started.md
railties/lib/rails/generators/actions.rb
railties/lib/rails/generators/app_base.rb
railties/lib/rails/generators/rails/app/app_generator.rb
railties/lib/rails/generators/rails/plugin/plugin_generator.rb
railties/test/generators/actions_test.rb

The –skip-vcs-ignores flag will skip the following files; .gitignore,

.hgignore, .svnignore. It will, however, continue to read the
.agignore file, which we’ll learn about in the next section.

2.6.3 The .agignore File (Ag only)

The .agignore file specifies ignore paths for Ag, and works
independently of any VCS file.

Listing 2.19
railties/lib
guides/code

This lets us keep our VCS ignore files focused on what can/can’t be
committed to source control, while still being able to give Ag
direction as to which paths we want to exclude from search.

2.6.4 The .ackrc File (Ack only)

The .ackrc is identical in nature to the .agignore file, but is for use
with Ack, instead of Ag.
2.7 Searching from Standard
Input
Both Ack and Ag will expect standard input as a source for
searching. This means we don’t need to just search files, but we can
use other commands to provide the data we’ll search over.

For example, we can use the ps command from Chapter 6 to get a

list of all of the processes running on our machine, and then search
for a specific command using either Ack or Ag.

$ ps -e | ag forego

Listing 2.20
58627 ttys005 0:00.91 forego start -p 3000

In Listing 2.20 we search through all of the running processes to

find one that contains the term forego.

This ability is incredibly useful, not just for finding things such as
running processes, but for reading file input from a command such
as cat, Chapter 9 Section 9.2, amongst other uses.

An alternative way to do this same search would be to use the grep

command, Chapter 5 Section 5.3.

2.8 Performance
Earlier in this chapter I mentioned how I prefer Ag over Ack
because of its speed. Let me demonstrate what I mean. Let’s
benchmark two pretty simple queries using the two different tools.

$ time ack DHH

real 0m0.981s
user 0m0.828s
sys 0m0.149s

$ time ag DHH

real 0m0.086s
user 0m0.059s
sys 0m0.142s

I think you would agree with me when I say that Ag is significantly

faster than Ack.

2.9 Conclusion
Well, that’s a good look at what Ack and Ag can do to help us quickly
search through large code bases.

Both pieces of software are capable of much, much more! I have

only scratched the surface of their feature sets in keeping with the
spirit of this book.

Both tools will give you a print out of their full features if you
simply call them without any parameters.
1. This output has been truncated to better fit the format of this book. ↑
2. This output has been truncated to better fit the format of this book. ↑
3. This output has been truncated to better fit the format of this book. ↑
4. This output has been truncated to better fit the format of this book. ↑
5. In Ack the -g flag is the equivalent of the -G flag in Ag, so just switch out the case of
the flag accordingly. ↑
Chapter 3
cURL
cURL is easily one of the most powerful tools in a developer’s
toolkit, as well as one of the most complex ones. It ships with a
dizzying array of options and features, most of which you will not
need on a daily basis.

cURL is a tool for working with URLs. cURL lets us query a URL
from the command line. It lets us post form data, FTP to a server,
and much, much more. One of the places that cURL comes in
handy most often is working with APIs. With cURL we can try out
new APIs simply, with just the command line, with no need for
installing, or writing complex wrappers around the API.

In this chapter, we will look at how we can use cURL to work with
remote URLs. We’ll also learn about the flags that will give us the
most bang for our buck in an average day.

3.1 Installation
There’s a very good chance that you already have cURL installed on
your machine, but it is nice to be up-to-date with the latest version.
If you are on a Mac, the best and easiest way to install cURL is using
Homebrew.

$ brew install curl

If you are on a different platform, please see the install directions

on the project’s page for the correct installation instructions for
your platform.

3.2 The Basics

The simplest thing we can do with cURL is to make an HTTP
request to a given server1 and print its response out to the console.

$ curl quiet-waters-1228.herokuapp.com/hello

As you can see in Listing 3.1, cURL prints the response of the web
page to the console.

Listing 3.1
Hello, World!
Thank you for cURLing me!

If we want to see more information about the response, we can

make the same request using the -i flag.

$ curl -i quiet-waters-1228.herokuapp.com/hello

Listing 3.2
 HTTP/1.1 200 OK
Cache-Control: max-age=0, private, must-revalidate
Content-Type: text/html; charset=utf-8
Etag: "a0bb15ce430e40738d857e3e7dfe0de7"
Server: thin 1.6.1 codename Death Proof
X-Content-Type-Options: nosniff
X-Frame-Options: SAMEORIGIN
X-Request-Id: d605f89d-bffc-4983-8ca4-8ac2b77c7b8d
X-Runtime: 0.002568
X-Ua-Compatible: chrome=1
X-Xss-Protection: 1; mode=block
transfer-encoding: chunked
Connection: keep-alive

Hello, World!
Thank you for cURLing me!

Listing 3.2 demonstrates that the -i flag will return response

information such as headers, server type, content type, and more.
This level of detail can prove to be extremely useful for debugging a
request.

3.3 Following Redirects (-L)

There are times when we are trying to download a resource from a
server, but instead of the resource we requested, the server returns
a status code of 302. The 302 response may also include a message
that the resource has moved. See Listing 3.3 for just such an
example.

$ curl -i quiet-waters-1228.herokuapp.com/redirectme

Listing 3.3
HTTP/1.1 302 Moved Temporarily
Cache-Control: no-cache
Content-Type: text/html; charset=utf-8
Location: http://quiet-waters-1228.herokuapp.com/hello
Server: thin 1.6.1 codename Death Proof
X-Content-Type-Options: nosniff
 X-Frame-Options: SAMEORIGIN
X-Request-Id: e9785070-173c-4e8a-bbf5-1686806cbd6b
X-Runtime: 0.007276
X-Ua-Compatible: chrome=1
X-Xss-Protection: 1; mode=block
transfer-encoding: chunked
Connection: keep-alive

<html>
<body>
You are being
<a href="http://quiet-waters-1228.herokuapp.com/hello">redirected</a>.
</body>
</html>

How do we follow redirects without having to first parse the

response and build a second request to the correct resource? To do
this, we can use the -L flag built into cURL.

$ curl -L quiet-waters-1228.herokuapp.com/redirectme

Using the -L flag we should now get the same response we did in
Listing 3.1.

3.4 Downloading Files (-O)

Printing responses to a console can be useful, but one of cURL’s
most popular features is its ability to easily download files.

If we were to run the following command against a file, the result

wouldn’t be very pleasing to your eyes on the screen.

$ curl http://quiet-waters-1228.herokuapp.com/assets/image.jpg

Listing 3.4
 ??JFIF,,??tExifII*
????(1?2?i??? NIKONE950,,Adobe Photoshop 7.02002:04:25 15:29:28??????"?'

Listing 3.4 shows the first two lines from a lot of gibberish that
represents a binary image file. However if we add the -O flag to the
request, the results will be saved into a file named image.jpg.

$ curl -O http://quiet-waters-1228.herokuapp.com/assets/image.jpg

Listing 3.5 and Listing 3.6 show the file downloading, and then a
confirmation of the file saved to disk.

Listing 3.5
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
100 413k 100 413k 0 0 306k 0 0:00:01 0:00:01 --:--:-- 306k

Listing 3.6: $ ls -la | grep image.jpg

-rw-r--r--@ 1 markbates staff 423159 Nov 18 10:46 image.jpg

As Listing 3.6 shows us, the file is downloaded from the server to
the current working directory using the same name as the file on
the server. Section 3.4.1 will show us how we can change the name
of this file.

See Chapter 5 for more information on how to use grep.

3.4.1 Custom File Names (-o)

More often than not, we can use the name of the file from the
server as the name of the file that we download to disk. However, in
this example case the name image.jpg leaves a lot to be desired. We
don’t know what it is an image of.

We can solve this problem in several different ways. The first way is
to use the mv command to rename the file after we’ve downloaded
it. Another way is to use the -o flag to give cURL a file name to save
the file to.

$ curl -o my_image.jpg http://quiet-waters-1228.herokuapp.com/assets/image.jpg

Listing 3.7: $ ls -la | grep my_image.jpg

-rw-r--r--@ 1 markbates staff 423159 Nov 18 10:58 my_image.jpg

Listing 3.7 shows that the file has now been downloaded using the
custom file name that we specified using the -o flag.

3.5 Changing the HTTP

Request Method (-X)
So far we have just been using cURL to make GET requests; however,
cURL supports all of the HTTP request methods, including POST,
PATCH, PUT, DELETE, etc.

To change the request method, we can use the -X flag.

Let’s make a simple POST request to an “echo” server that will print
out any parameters we send to it.

$ curl -X POST quiet-waters-1228.herokuapp.com/echo

Listing 3.8 shows the result of our request.

Listing 3.8
<!DOCTYPE html>
<html>
<head>
<title>Unix and Linux Commands for Developers</title>
<link href="/assets/application.css" media="all" rel="stylesheet" />
<script src="/assets/application.js"></script>
</head>
<body>

<p>
<strong>url:</strong>
<span>
http://quiet-waters-1228.herokuapp.com/echo
</span>
</p>
<p>
<strong>method:</strong>
<span>
POST
</span>
</p>

</body>
</html>

Let’s make the same request again, only this time using the PUT
request method.

$ curl -X PUT quiet-waters-1228.herokuapp.com/echo

Listing 3.9 confirms that we did indeed change the request method
from POST to PUT.

Listing 3.9
<!DOCTYPE html>
<html>
<head>
<title>Unix and Linux Commands for Developers</title>
<link href="/assets/application.css" media="all" rel="stylesheet" />
<script src="/assets/application.js"></script>
</head>
<body>
 <p>
<strong>url:</strong>
<span>
http://quiet-waters-1228.herokuapp.com/echo
</span>
</p>
<p>
<strong>method:</strong>
<span>
PUT
</span>
</p>

</body>
</html>

Depending on your request type (in our case this was HTTP), the -X
flag changes the request method appropriately. For example, with
an FTP request the -X flag will allow you to change the request type
from LIST to another request method.

3.5.1 Sending Parameters

So far, we haven’t sent any parameters to the server. There are a

few different ways to do this.

The first, and most obvious method, is to append query string

parameters to the URL2 itself.

$ curl -X POST "quiet-waters-1228.herokuapp.com/echo?fname=Mark&lname=Bates"

Listing 3.10 shows that the query string parameters are parsed
correctly by the echo server.

Listing 3.10
<!DOCTYPE html>
<html>
<head>
<title>Unix and Linux Commands for Developers</title>
 <link href="/assets/application.css" media="all" rel="stylesheet" />
<script src="/assets/application.js"></script>
</head>
<body>

<p>
<strong>url:</strong>
<span>
http://quiet-waters-1228.herokuapp.com/echo?fname=Mark&lname=Bates
</span>
</p>
<p>
<strong>parameters:</strong>
<span>
{"fname"=>"Mark", "lname"=>"Bates"}
</span>
</p>
<p>
<strong>method:</strong>
<span>
POST
</span>
</p>

</body>
</html>

Passing query string parameters will work with any request type,
not just POST. This is the simplest, and easiest, way to pass data to
the server in a cURL request, but not the only way.

3.5.2 Passing Data in a Request Body (-d)

When making non-GET requests, such as a POST, it is quite common

to want to pass data to the server via the body of the request. After
all, query string parameters can’t represent a complex form very
well.

cURL provides the -d flag to us pass data as part of the request

body.

Instead of passing the data via query string parameters, let’s use the
-d flag instead.
 $ curl -X POST -d "fname=Mark&lname=Bates" quiet-waters-1228.herokuapp.com/echo

As we can see in Listing 3.11, the echo server reads the body and
converts it into parameters, just as if we had used query string
parameters.

Listing 3.11
<!DOCTYPE html>
<html>
<head>
<title>Unix and Linux Commands for Developers</title>
<link href="/assets/application.css" media="all" rel="stylesheet" />
<script src="/assets/application.js"></script>
</head>
<body>

<p>
<strong>url:</strong>
<span>
http://quiet-waters-1228.herokuapp.com/echo
</span>
</p>
<p>
<strong>parameters:</strong>
<span>
{"fname"=>"Mark", "lname"=>"Bates"}
</span>
</p>
<p>
<strong>method:</strong>
<span>
POST
</span>
</p>
<p>
<strong>body:</strong>
<span>
fname=Mark&lname=Bates
</span>
</p>

</body>
</html>

We can also see in Listing 3.11 that the echo server now displays a
bodyattribute to us. This proves that the data is in fact coming as a
request body, and not just basic parameters.
We can make this more interesting by passing JSON via the -d flag.

$ curl -X POST -d "{\"name\":\"Mark\"}" quiet-waters-1228.herokuapp.com/echo

Listing 3.12
<!DOCTYPE html>
<html>
<head>
<title>Unix and Linux Commands for Developers</title>
<link href="/assets/application.css" media="all" rel="stylesheet" />
<script src="/assets/application.js"></script>
</head>
<body>

<p>
<strong>url:</strong>
<span>
http://quiet-waters-1228.herokuapp.com/echo
</span>
</p>
<p>
<strong>method:</strong>
<span>
POST
</span>
</p>
<p>
<strong>body:</strong>
<span>
{"name"=>"Mark"}
</span>
</p>

</body>
</html>

Listing 3.12 shows the JSON has been received and parsed correctly.

3.5.3 Using a File for a Request Body (-d)

Being able to send the request body via the command line is very
useful, but most of the time we’ll want to read in a file and send that
as the body of the request. An example of this would be sending
larger amounts of JSON.
Let’s say we have the file from Listing 3.13, how would we post that
to a server?

Listing 3.13: form_data.json

{
"lname": "Bates",
"fname": "Mark",
"site": "http://www.markbates.com",
"twitter": "http://twitter.com/markbates"
}

The -d flag already gives us all that we need to get the job done. By
passing the -d flag a file name prefixed with @, it will read in that file
as the request body.

$ curl -X POST -d @form_data.json quiet-waters-1228.herokuapp.com/echo

Listing 3.14
<!DOCTYPE html>
<html>
<head>
<title>Unix and Linux Commands for Developers</title>
<link href="/assets/application.css" media="all" rel="stylesheet" />
<script src="/assets/application.js"></script>
</head>
<body>

<p>
<strong>url:</strong>
<span>
http://quiet-waters-1228.herokuapp.com/echo
</span>
</p>
<p>
<strong>method:</strong>
<span>
POST
</span>
</p>
<p>
<strong>body:</strong>
<span>
{"lname"=>"Bates", "fname"=>"Mark",
"site"=>"http://www.markbates.com", "twitter"=>"http://twitter.com/markbates"}
</span>
</p>
 </body>
</html>

With this simple way of using files for request bodies, it can be very
easy to write a script using cURL to upload files to a remote server.

3.5.4 Form Parameters (-F)

Another way to pass parameters using cURL is the -F flag. Using

this flag we can send parameters that will be interpreted by the
remote server as if they had been posted from an HTML form.

$ curl -X POST -F user[fname]=Mark -F user[lname]=Bates -F foo=bar \

quiet-waters-1228.herokuapp.com/echo -H "Accept: application/json"

Listing 3.15
{
"url": "http://quiet-waters-1228.herokuapp.com/echo",
"parameters": {"user":{"fname":"Mark","lname":"Bates"},"foo":"bar"},
"method": "POST",
"body": "------------------------------d6e7f7029fa5\r\nContent-Disposition:
form-data; name=\"user[fname]\"\r\n\r\nMark\r\n------------------------------
d6e7f7029fa5\r\nContent-Disposition: form-data; name=\"user[lname]\"\r\n\r\n
Bates\r\n------------------------------d6e7f7029fa5\r\nContent-Disposition:
form-data; name=\"foo\"\r\n\r\nbar\r\n------------------------------
d6e7f7029fa5--\r\n"
}

In Listing 3.15 we can see that the parameters were interpreted by

the server as form-data. This can be used to mimic HTML forms,
and can be particularly useful to post both form parameters as well
as one, or more, files.

File Uploads with Form Parameters

Uploading a file using the -F flag is almost identical to using the -d
flag that we saw in Section 3.5.3. The only difference is that the file
must be given a parameter name. This can be see in Listing 3.16.

Listing 3.16
$ curl -X POST -F user[fname]=Mark -F user[lname]=Bates -F foo=bar \
-F user[photo]=@path/to/image.jpg quiet-waters-1228.herokuapp.com/echo
-H "Accept: application/json"

3.6 Setting Headers (-H)

So far, all of the responses we have received from the server have
been in HTML, but the server is also capable of sending us back data
formatted in either JSON or the dreaded XML.

In order to tell the server which type of content we want returned,

we must set the Content-Type header using the -H flag.

$ curl -X POST -d @form_data.json quiet-waters-1228.herokuapp.com/echo \

-H "Accept: application/json"

Listing 3.17 shows that by using the -H flag and giving it

"Accept: application/json"

the server responds appropriately with JSON instead of HTML.

Listing 3.17
{
"body" : {
"site" : "http://www.markbates.com",
"twitter" : "http://twitter.com/markbates",
 "lname" : "Bates",
"fname" : "Mark"
},
"url" : "http://quiet-waters-1228.herokuapp.com/echo",
"method" : "POST"
}

Setting the content type of a request isn’t the only header we can
specify using the -H flag. We can set any headers that the
responding server will accept. cURL allows multiple -H flags on a
request, making it easy to send several headers at once.

$ curl -X POST -d @form_data.json quiet-waters-1228.herokuapp.com/echo \

-H "Accept: application/json" -H "X-Auth: 1234567890"

As we see in Listing 3.18 the new header, "X-Auth", has been

received by the server.

Listing 3.18
{
"body" : {
"site" : "http://www.markbates.com",
"twitter" : "http://twitter.com/markbates",
"lname" : "Bates",
"fname" : "Mark"
},
"url" : "http://quiet-waters-1228.herokuapp.com/echo",
"x-auth" : "1234567890",
"method" : "POST"
}

3.7 Basic HTTP Auth (-u)

A lot of the internet exists behind a login form these days, so it’s
important to understand how we can use cURL to access resources
that may be protected by such an authentication requirement.
Let’s see what happens when we try to access a resource that
requires authentication.

$ curl -i -X POST quiet-waters-1228.herokuapp.com/login

Listing 3.19 shows that we received a status code of 401 and a

message that it can’t find a user.

Listing 3.19
HTTP/1.1 401 Unauthorized
Cache-Control: no-cache
Content-Type: application/json; charset=utf-8
Server: thin 1.6.1 codename Death Proof
X-Content-Type-Options: nosniff
X-Frame-Options: SAMEORIGIN
X-Request-Id: da6bbeae-1b77-4f2a-bf75-300b981e108d
X-Runtime: 0.009238
X-Ua-Compatible: chrome=1
X-Xss-Protection: 1; mode=block
transfer-encoding: chunked
Connection: keep-alive

{"error":"Unknown User"}

The -u flag will let us pass a username and a password to the server
to solve this problem.

$ curl -X POST -u "user1:password1" quiet-waters-1228.herokuapp.com/login

As seen in Listing 3.20 the login was successful and we now use the
information for user1.

Listing 3.20
{
"password" : "password1",
"name" : "User 1",
"id" : 1,
"username" : "user1"
}
With the -u flag we can now successfully authenticate ourselves
against any page that uses HTTP auth.

3.8 Saving and Sending

Cookies/Headers (-D, -b)
In Section 3.7 we learned how to use the -u flag to authenticate
against basic HTTP authentication. But how do we handle a page
that is behind a login, yet doesn’t respond to HTTP auth?

Let’s attempt to request the endpoint in question and see what the
result is.

$ curl -i quiet-waters-1228.herokuapp.com/whoami

Not surprisingly we get a 401, just like in Listing 3.19, so let’s try and
use the -u flag and see if we can gain access to the page that way.

$ curl -i -u "user1:password1" quiet-waters-1228.herokuapp.com/whoami

Again, we are greeted with a 401. The server is trying to use a

session cookie to authenticate us, and we don’t have one. To solve
this problem, we first need to hit the login endpoint using the -u
flag, save the session cookie that is returned, and then pass that
session cookie back again on subsequent requests.

The -D flag will tell cURL to dump headers and cookies into a
specified file in the current directory.
 $ curl -X POST -D headers -u "user1:password1" \
quiet-waters-1228.herokuapp.com/login

The more verbose –dump-header flag is a mnemonic synonym for -D.

$ curl -X POST --dump-header headers -u "user1:password1" \

quiet-waters-1228.herokuapp.com/login

Listing 3.21 shows the contents of the file that got created as a
result of our request.

Listing 3.21: headers

HTTP/1.1 200 OK
Cache-Control: max-age=0, private, must-revalidate
Content-Type: application/json; charset=utf-8
Etag: "768e41b20e1e385d06f9b2da6f7e0f08"
Server: thin 1.6.1 codename Death Proof
Set-Cookie: _curl_test_app_rails_session=TWpwUitjellkSE1QOHpyTEN0Q1JkazFhVVhvMi9
4RnZvZ2JEdExjR2hrb05FbGpVUCtHZ2xraTF5a01qWFc4OWhyUER6cG53dlZQZUhHdnJxTHBKSW1ITHQ
0NTdUOXdFVU5nNE05VStadUo2dS80eEhkVzJzNUdXZVg5SC9OLy8tLVFRKzc3cHB2UWlKM3FOckM2S3h
jZHc9PQ%3D%3D--3ab5702185dca000cff025fd3139348147171c34; path=/; HttpOnly
X-Content-Type-Options: nosniff
X-Frame-Options: SAMEORIGIN
X-Request-Id: a5efa267-3f9a-40ff-9a8d-49bfa8754c62
X-Runtime: 0.050191
X-Ua-Compatible: chrome=1
X-Xss-Protection: 1; mode=block
transfer-encoding: chunked
Connection: keep-alive

With this headers file now saved, we can pass it around to

subsequent requests and the server will have access to the session
cookie that it needs, along with all of the original headers that were
part of the response.

In order to pass the headers file back to the server with our request,
we need to use the -b flag.

$ curl -b headers quiet-waters-1228.herokuapp.com/whoami

By passing the -b flag and giving it the name of the file that we had
previously saved, Listing 3.22 shows that we have successfully
accessed the end point that was previously unavailable to us.

Listing 3.22
{
"password" : "password1",
"name" : "User 1",
"id" : 1,
"username" : "user1"
}

3.8.1 Using the Cookie Jar (-c)

When using the –dump-header flag we not only capture the cookies
from a response, but also all of the headers from that response.
This may or may not be what you want.

curlhas a built-in cookie jar meant for storing just the cookies and
sending them back to the server.

To store cookies in the cookie jar we use the -c flag and give it the
name of a file we wish to store the cookies in.

$ curl -X POST -c cookies.txt -u "user1:password1" \

quiet-waters-1228.herokuapp.com/login

Listing 3.23
{"id":1,"name":"User 1","password":"password1","username":"user1"}

As we can see in Listing 3.23 we get a successful response, just as

we did when we used the –dump-header flag.
If we use the cat command (Chapter 9 Section 9.2) we can look at
the contents of the cookies.txt file.

$ cat cookies.txt

Listing 3.24
# Netscape HTTP Cookie File
# http://curl.haxx.se/docs/http-cookies.html
# This file was generated by libcurl! Edit at your own risk.

#HttpOnly_quiet-waters-1228.herokuapp.com FALSE / FALSE 0 _curl_test_app_rails_

session cm53d2RJN1VncVpnK3psVUpuRHB2L2d3emIyTENIMGhBVjlPOHozM0lJYmk2WmwrSGZEbEV
UbXpnSkdGaktvZjVFeThranhuSi9HZm9YTHFSYUthUzhoVUc0T3J3MEltempsNC96UVFweXMranlsem
RPMzNYSmxzK2pJME5DMEItLWVkSDZZN0pjdU5HTmt5WHRxTW1vSUE9PQ%3D%3D--06137b60e0ff66f
3a6cad05e4fe83a0094a989be

When we look at the cookies.txt file in Listing 3.24 we see that this
time we are only storing the cookies from the response, and not the
headers as well.

Using the -b flag, as we did earlier, we can send the contents of the
cookies.txt file to the server with the request.

$ curl -b cookies.txt quiet-waters-1228.herokuapp.com/whoami

Listing 3.25
{
"password" : "password1",
"name" : "User 1",
"id" : 1,
"username" : "user1"
}

Listing 3.25 shows that we get a successful response from the server
when passing cookies.txt to it.
3.9 Conclusion
As you can see, cURL is an incredibly powerful tool. We have used
it to download files, upload files, access authenticated pages, and
more. This is a very small amount of the features that are available
in cURL.

The MAN page for cURL is very long, and for a very good reason.
The list of options that can be passed is dizzying. Hopefully this
chapter has given you the building blocks for using cURL on a daily
basis.

1. I have set up a small demo application at quiet-waters-1228.herokuapp.com to help

better illustrate the examples in this chapter. ↑
2. Because we are using an & in the URL we need to make sure to escape the URL so
that the command doesn’t interpret it incorrectly. ↑
Chapter 4
Find
In Chapter 2, we used Ack and Ag to search through a large code
base to find files that contained certain query terms, but what if we
want to search for files using criteria other than the file’s contents?

The find command is just such a search tool. With find we can
quickly search for files whose path, or name, contains a query term.
We can also find files that are owned by a particular user, are of a
certain size, have been last modified in the last n days, and much
more. In Chapter 8, we’ll use find to generate a list of files to create
an archive from.

In this chapter, we’ll look at some of the most useful features of the
find command.

The source code we’ll be using for this chapter’s examples is the
Rails source code as mentioned in the Preface.

4.1 The Basics (-name)

The find command has a very basic usage format.
 $ find PATH_TO_SEARCH OPTIONS_TO_USE PATTERN_TO_SEARCH_FOR

With that knowledge in place, let’s write a simple query to find all
files that end in model.rb.

$ find . -name model.rb

As we can see in Listing 4.1 we are only finding files that match
model.rb exactly.

Listing 4.1
./activemodel/lib/active_model/model.rb
./activerecord/lib/rails/generators/active_record/model/templates/model.rb

The reason is that the -name option is expecting a pattern to match,

and the pattern we gave it used no wild cards to indicate that we
want to match the end of the file’s name.

Let’s change our query slightly to fix the problem.

$ find . -name \*model.rb

Listing 4.2
./activemodel/lib/active_model/model.rb
./activemodel/lib/active_model.rb
./activerecord/lib/rails/generators/active_record/model/templates/model.rb
./activerecord/test/models/arunit2_model.rb
./railties/lib/rails/generators/active_model.rb

Listing 4.2 shows that we managed to search the current directory, .

(and by default its sub-directories), for files whose file name ends
with *model.rb.
You’ll notice that we used . to represent the current directory; we
could have also used the equivalent expression ./. We also had to
escape the * so that it wasn’t misinterpreted by the operating
system.

4.2 Searching Paths (-path)

In Section 4.1 we learned how to use find to search for files whose
name matches a pattern, but what about if we want to find files, or
directories, whose path matches a pattern?

To search within a path we can use the -path flag instead of the -
name flag. Let’s try to find any files whose path contains the word
session.

$ find . -path \*session\*

Listing 4.3
./actionpack/lib/action_dispatch/middleware/session
./actionpack/lib/action_dispatch/middleware/session/abstract_store.rb
./actionpack/lib/action_dispatch/middleware/session/cache_store.rb
./actionpack/lib/action_dispatch/middleware/session/cookie_store.rb
./actionpack/lib/action_dispatch/middleware/session/mem_cache_store.rb
./actionpack/lib/action_dispatch/request/session.rb
./actionpack/test/dispatch/request/session_test.rb
./actionpack/test/dispatch/session
./actionpack/test/dispatch/session/abstract_store_test.rb
./actionpack/test/dispatch/session/cache_store_test.rb
./actionpack/test/dispatch/session/cookie_store_test.rb
./actionpack/test/dispatch/session/mem_cache_store_test.rb
./actionpack/test/dispatch/session/test_session_test.rb
./actionpack/test/fixtures/session_autoload_test
./actionpack/test/fixtures/session_autoload_test/session_autoload_test
./actionpack/test/fixtures/session_autoload_test/session_autoload_test/foo.rb
./activerecord/test/models/possession.rb
./guides/assets/images/session_fixation.png
./guides/code/getting_started/config/initializers/session_store.rb
./railties/test/application/middleware/session_test.rb
Listing 4.3 shows the results of our query. The output shows us the
many files and directories that contain the word session.

4.2.1 Find Only Files or Directories (-type)

In Listing 4.3, we wanted to find files whose path name contained

the word session. Based on the results, however, note that
directories whose names matched the pattern were also supplied.

By using the -type flag, we can tell find to either look exclusively
for files, -type f, or just directories, -type d.

We can now alter our query to exclusively search for files whose
path contains session so that directories are not included in the
results.

$ find . -path \*session\* -type f

Listing 4.4
./actionpack/lib/action_dispatch/middleware/session/abstract_store.rb
./actionpack/lib/action_dispatch/middleware/session/cache_store.rb
./actionpack/lib/action_dispatch/middleware/session/cookie_store.rb
./actionpack/lib/action_dispatch/middleware/session/mem_cache_store.rb
./actionpack/lib/action_dispatch/request/session.rb
./actionpack/test/dispatch/request/session_test.rb
./actionpack/test/dispatch/session/abstract_store_test.rb
./actionpack/test/dispatch/session/cache_store_test.rb
./actionpack/test/dispatch/session/cookie_store_test.rb
./actionpack/test/dispatch/session/mem_cache_store_test.rb
./actionpack/test/dispatch/session/test_session_test.rb
./actionpack/test/fixtures/session_autoload_test/session_autoload_test/foo.rb
./activerecord/test/models/possession.rb
./guides/assets/images/session_fixation.png
./guides/code/getting_started/config/initializers/session_store.rb
./railties/test/application/middleware/session_test.rb

Similarly, we can also search for directories whose path contains

session.
 $ find . -path \*session\* -type d

Listing 4.5
./actionpack/lib/action_dispatch/middleware/session
./actionpack/test/dispatch/session
./actionpack/test/fixtures/session_autoload_test
./actionpack/test/fixtures/session_autoload_test/session_autoload_test

4.3 And/Or Expressions (-or)

We can further refine our search from Listing 4.4 to find only files
whose path name contains session and whose file name contains
mem.

$ find . -path \*session\* -type f -name \*mem\*

Listing 4.6
./actionpack/lib/action_dispatch/middleware/session/mem_cache_store.rb
./actionpack/test/dispatch/session/mem_cache_store_test.rb

As Listing 4.6 shows, we can combine both the -name and the -path
flags together to make our search very specific. This type of query is
an AND query. Both flags have to match for the file to be included in
the results.

We can also do OR queries in find using the -or operator and some
well placed parentheses.

Let’s write a query that will find files whose name either ends in
.gemspec or in .jpg.
 $ find . \( -name \*.gemspec -or -name \*.jpg \) -type f

Listing 4.7
./actionmailer/actionmailer.gemspec
./actionmailer/test/fixtures/attachments/foo.jpg
./actionmailer/test/fixtures/attachments/test.jpg
./actionpack/actionpack.gemspec
./actionpack/test/fixtures/multipart/mona_lisa.jpg
./actionview/actionview.gemspec
./activemodel/activemodel.gemspec
./activerecord/activerecord.gemspec
./activerecord/test/assets/flowers.jpg
./activesupport/activesupport.gemspec
./guides/assets/images/akshaysurve.jpg
./guides/assets/images/oscardelben.jpg
./guides/assets/images/rails_guides_kindle_cover.jpg
./guides/assets/images/vijaydev.jpg
./rails.gemspec
./railties/lib/rails/generators/rails/plugin/templates/%name%.gemspec
./railties/railties.gemspec

It’s important to remember to properly escape characters such as

parentheses and asterisks as to not confuse the operating system.
Most of the problems you’ll run into with queries such as this is
improper escaping, so make sure to keep an eye out for it.

4.4 Not Matching Query (!, -

not)

Up until this point, we have been searching for files whose names or
paths match a certain pattern. Now, let’s do the opposite. We can
query for files, or directories, that don’t match a certain pattern.

Let’s say we want to find all of the files in our directory that don’t
contain the letter t somewhere in their path.
To do that we can use the -not operator in front of the -path flag.

$ find . -not -path \*t\* -type f

Listing 4.8
./CONTRIBUTING.md
./Gemfile
./guides/CHANGELOG.md
./guides/rails_guides/helpers.rb
./guides/rails_guides/indexer.rb
./guides/rails_guides/kindle.rb
./guides/rails_guides/markdown/renderer.rb
./guides/rails_guides/markdown.rb
./guides/rails_guides.rb
./guides/Rakefile
./guides/source/command_line.md
./guides/source/configuring.md
./guides/source/engines.md
./guides/source/form_helpers.md
./guides/source/i18n.md
./guides/source/kindle/KINDLE.md
./guides/source/kindle/rails_guides.opf.erb
./guides/source/plugins.md
./guides/source/rails_on_rack.md
./guides/source/ruby_on_rails_guides_guidelines.md
./guides/source/upgrading_ruby_on_rails.md
./rails.gemspec
./RAILS_VERSION
./Rakefile
./README.md
./RELEASING_RAILS.rdoc
./version.rb

Another way to achieve the same result set as Listing 4.8 is to use !
instead of the -not operator. They will both give you the same
results, but I personally prefer the ! variant.

$ find . \! -path \*t\* -type f

4.5 Find by Last Modified

Time (-mtime)
findallows us to search for files and directories in many different
ways. Up until now, we’ve been focused on the name of the file or
the path of the file, but what about searching for files based on their
last modified date?

Using the -mtime flag we can search for files whose last modified
date was in the last n days.

Here we can find files who have been modified in the last day.

$ find . -mtime -1

Listing 4.9
./actionpack/lib/abstract_controller.rb
./ci/travis.rb
./Gemfile
./railties/test/engine_test.rb

The number we pass to the -mtime flag, in this case -1, is the
number of days we are interested in. This example returns files that
have been modified in the last 24 hours. An option of -2 would
represent the last 48 hours and so on.

Using the -mmin flag we can look for files that have been modified in
the last n minutes.

$ find . -mmin -5

Listing 4.10
./ci/travis.rb
4.6 Find by File Size (-size)
Another useful flag is -size, which, as its name states, allows us to
look for files whose size is over the specified amount.

Let’s do a search for files whose size is greater than 200 kilobytes.

$ find . -size +200k

Listing 4.11
./.git/index
./.git/objects/pack/pack-3fc5fb51b38dba971962c3dfe13c643c7c3ff357.pack
./.git/objects/pack/pack-a93c0c2849d6dba30ddb4a967548b955005ecbf1.idx
./.git/objects/pack/pack-a93c0c2849d6dba30ddb4a967548b955005ecbf1.pack
./actionview/test/template/date_helper_test.rb
./activesupport/lib/active_support/values/unicode_tables.dat

If we don’t want to see those files that are under the .git directory
we can use the ! operator we learned about in Section 4.4 to remove
them.

$ find . \! -path '*/\.*' -size +200k

Listing 4.12
./actionview/test/template/date_helper_test.rb
./activesupport/lib/active_support/values/unicode_tables.dat

Table 4.1 shows the different modifiers we can add to the -size
number to specify the size of the files we want to look for.

modifier size actual size

 k kilobytes (1024 bytes)

M megabytes (1024 kilobytes)

G gigabytes (1024 megabytes)

T terabytes (1024 gigabytes)

P petabytes (1024 terabytes)

Table 4.1: Modifiers for -size

4.7 Performing Operations (-

delete)

The find tool also has the built-in ability to execute certain
commands. One of the more useful built-in commands is the -
delete flag.

The -delete flag will, as its name suggests, delete all files and
directories that match the pattern. This makes this type of
operation incredibly efficient as we can combine the finding and
deleting into one operation instead of two.

Let’s find all of the files whose name ends with .yml and delete
them from our repository. Here I’m also using the -print flag to
show the files that will be deleted.

$ find ./guides -type f -name \*.yml -print -delete

Listing 4.13
./guides/code/getting_started/config/database.yml
./guides/code/getting_started/config/locales/en.yml
./guides/code/getting_started/test/fixtures/comments.yml
./guides/code/getting_started/test/fixtures/posts.yml

We can verify that the -delete was performed by running the find
command and seeing that no matches are found.

$ find ./guides -type f -name \*.yml

4.8 Conclusion
We’ve covered a lot of what find can do in this chapter, but it is
capable of significantly more. There is not enough space here to
cover all of the different flags, options, and ways to find the files you
are looking for, but hopefully this has given you a feel for what this
indispensable tool can accomplish.

For further evidence of how useful find is, see Chapter 8

Section 8.2.3 where we use find to build a list of files to build a tar
archive.
Chapter 5
Grep
The grep command is universal. You’ll find it installed on every
Unix/Linux based computer. It is, possibly, one of the most popular
commands you’ll find.

What is the grep command? grep is a tool for searching files for
search terms, similar to ack or ag that we learned about in
Chapter 2.

Let’s quickly take a moment to compare grep with ack or ag to

understand what the differences are.

grep is found on almost all Unix/Linux systems, whereas ack or ag

may need to be installed. grep is commonly used, because of its
prevalence, in most example code you’ll find on the internet.

ack or ag both sport more modern APIs than grep. Both are faster
than grep; ag is significantly faster.

It is a matter of personal taste which of these searching libraries

you use; I would recommend trying them all and choosing the best
one for your situation. With that said, you should, at the very least,
understand the basics of grep as there will most likely be cases
where the machine you’re using does not have one of the
alternatives installed.

5.1 The Basics

We’ll want a sizable code base to search through to show the power
of grep, so we’ll be using the Rails source code, as mentioned in the
Preface.

The basic usage of grep is pretty simple. Let’s search for the term
Pack in the README.md file.

$ grep Pack README.md

Listing 5.1
are bundled together in Action Pack. You can read more about Action Pack in its
Active Record, Action Pack, and Action View can each be used independently ou...

Listing 5.11 shows that we found two lines containing Pack in the
file.

5.1.1 Regular Expression Searches

By default grep uses Regular Expressions to search for patterns in a

file. For example, let’s search the README.md file for any lines that
contain the letter r followed by any two characters, then followed by
the letter y.

$ grep "r..y" README.md

Listing 5.2
utility classes and standard library extensions that are useful for Rails, a...
* [Getting Started with Rails](http://guides.rubyonrails.org/getting_sta...
* [Ruby on Rails Guides](http://guides.rubyonrails.org)
* [The API Documentation](http://api.rubyonrails.org)
* [Ruby on Rails Tutorial](http://ruby.railstutorial.org/ruby-on-rails-t...
[Contributing to Ruby on Rails guide](http://edgeguides.rubyonrails.org/cont...

In Listing 5.22 we see that grep has found matches such as ruby and
rary (which appeared inside of “library”).

5.1.2 Counting Occurrences (-c)

It can sometimes be useful to count the number of times a pattern

is found in a file. To do this we can use the -c flag.

$ grep -c ruby README.md

Listing 5.3
5

As we can see in Listing 5.3 the README.md file contained the pattern
ruby five times.

5.1.3 Displaying Line Numbers (-n)

When searching through a file it can be useful to know which

line(s) of the file the results can be found on. The -n flag will print
line numbers as part of the output.

$ grep -n Ruby README.md

Listing 5.4
17:be ordinary Ruby classes, or Ruby classes that implement a set of interfaces
33:Ruby code (ERB files). Views are typically rendered to generate a control...
62:4. Using a browser, go to http://localhost:3000 and you'll see: "Welcome ...
67: * [Ruby on Rails Guides](http://guides.rubyonrails.org)
69: * [Ruby on Rails Tutorial](http://ruby.railstutorial.org/ruby-on-rail...
73:We encourage you to contribute to Ruby on Rails! Please check out the
74:[Contributing to Ruby on Rails guide](http://edgeguides.rubyonrails.org/c...
83:Ruby on Rails is released under the [MIT License](http://www.opensource.o...

Listing 5.43 demonstrates grep printing line numbers for our

results.

5.1.4 Case Insensitive Search (-i)

In Section 5.1.1 we saw that grep uses Regular Expressions by

default when searching a file. Because of this, searches are case
sensitive. To change this default behavior and perform a case
insensitive search instead, we can use the -i flag.

First let’s start by searching for the term rails in the README.md file.

$ grep rails README.md

Listing 5.5
gem install rails
rails new myapp
rails server
* [Getting Started with Rails](http://guides.rubyonrails.org/getting_sta...
* [Ruby on Rails Guides](http://guides.rubyonrails.org)
* [The API Documentation](http://api.rubyonrails.org)
* [Ruby on Rails Tutorial](http://ruby.railstutorial.org/ruby-on-rails-t...
[Contributing to Ruby on Rails guide](http://edgeguides.rubyonrails.org/cont...
* [](https://travi...
* [](https://ge...

Listing 5.5 shows we are only finding results in the README.md file
that match the lower case pattern, rails.
Let’s change our search to use the -i flag and compare the results.

$ grep -i rails README.md

Listing 5.6
## Welcome to Rails
Rails is a web-application framework that includes everything needed to
Understanding the MVC pattern is key to understanding Rails. MVC divides your
your application. In Rails, database-backed model classes are derived from
methods. Although most Rails models are backed by a database, models can also
providing a suitable response. Usually this means returning HTML, but Rails ...
In Rails, incoming requests are routed by Action Dispatch to an appropriate ...
or to generate the body of an email. In Rails, View generation is handled by...
Active Record, Action Pack, and Action View can each be used independently o...
In addition to them, Rails also comes with Action Mailer ([README](actionmai...
utility classes and standard library extensions that are useful for Rails, a...
independently outside Rails.
1. Install Rails at the command prompt if you haven't yet:
gem install rails
2. At the command prompt, create a new Rails application:
rails new myapp
rails server
4. Using a browser, go to http://localhost:3000 and you'll see: "Welcome abo...
* [Getting Started with Rails](http://guides.rubyonrails.org/getting_sta...
* [Ruby on Rails Guides](http://guides.rubyonrails.org)
* [The API Documentation](http://api.rubyonrails.org)
* [Ruby on Rails Tutorial](http://ruby.railstutorial.org/ruby-on-rails-t...
We encourage you to contribute to Ruby on Rails! Please check out the
[Contributing to Ruby on Rails guide](http://edgeguides.rubyonrails.org/cont...
* [](https://travi...
* [](https://ge...
Ruby on Rails is released under the [MIT License](http://www.opensource.org/...

When we compare Listing 5.64 to Listing 5.55 we can see we have

found a lot more references to the pattern rails, such as Rails.

We can accomplish the same effect using straight Regular

Expressions, but it is a lot easier, and cleaner, to use the -i flag.

5.2 Searching Multiple Files

Up until this point we have only been searching a single file;
however, grep gives us several options for searching through
multiple files at the same time. In this section we’ll look at a few of
these options.

5.2.1 Searching Specific Files

So far we have been passing a specific file name to grep as the last
argument. To tell grep to search multiple files, we can simply pass
in multiple file names to the grep command.

$ grep rails README.md RELEASING_RAILS.rdoc

Listing 5.7
README.md: gem install rails
README.md: rails new myapp
README.md: rails server
README.md: * [Getting Started with Rails](http://guides.rubyonrails.org/g...
README.md: * [Ruby on Rails Guides](http://guides.rubyonrails.org)
README.md: * [The API Documentation](http://api.rubyonrails.org)
README.md: * [Ruby on Rails Tutorial](http://ruby.railstutorial.org/ruby-...
README.md:[Contributing to Ruby on Rails guide](http://edgeguides.rubyonrail...
README.md:* [](htt...
README.md:* [](...
RELEASING_RAILS.rdoc: http://travis-ci.org/rails/rails
RELEASING_RAILS.rdoc:* [email protected]
RELEASING_RAILS.rdoc: [aaron@higgins rails (3-0-stable)]$ git checkout -b...
RELEASING_RAILS.rdoc: [aaron@higgins rails (3-0-10)]$
RELEASING_RAILS.rdoc: [aaron@higgins rails (3-0-10)]$ git log v3.0.9..
RELEASING_RAILS.rdoc:* [email protected]
RELEASING_RAILS.rdoc:* [email protected]
RELEASING_RAILS.rdoc:issues with the release candidate to the rails-core mai...
RELEASING_RAILS.rdoc:* http://weblog.rubyonrails.org
RELEASING_RAILS.rdoc:Check the rails-core mailing list and the GitHub issue ...
RELEASING_RAILS.rdoc:* [email protected]

Listing 5.76 shows results for the term rails across both files.
Notice that grep conveniently prints the name of the file next to
each line in the results, so we know which result came from which
file.
If we don’t want to list specific files, we can tell grep to search
specific files in the current directory by giving it a pattern.

$ grep gem *.rb

Listing 5.8
install.rb: `cd #{framework} && gem build #{framework}.gemspec && gem insta...
install.rb:`gem build rails.gemspec`
install.rb:`gem install rails-#{version}.gem --no-ri --no-rdoc `
install.rb:`rm rails-#{version}.gem`
load_paths.rb:# bust gem prelude

In Listing 5.8 we told grep to search all files in the current directory
that match *.rb for the pattern gem.

5.2.2 Recursive Searching (-R)

grep will let us search recursively through directories as well. To

search recursively we can use the -R flag, which will search
whichever directory we specify, and all of its subdirectories.

$ grep -R "Read" .

Listing 5.9
./actionpack/lib/action_controller/base.rb: # Read more about writing ERB a...
./actionpack/lib/action_controller/caching/fragments.rb: # Reads a cach...
./actionpack/lib/action_controller/metal/data_streaming.rb: # Read abou...
./actionpack/lib/action_dispatch/http/request.rb: # Read the request \bod...
./actionpack/lib/action_dispatch/middleware/cookies.rb: # being written wil...
./actionpack/lib/action_dispatch/middleware/cookies.rb: "Read the...
./actionpack/lib/action_dispatch/routing/url_for.rb: # then ActionControl...
./actionpack/test/controller/log_subscriber_test.rb: assert_match(/Read f...
./actionpack/test/controller/log_subscriber_test.rb: assert_match(/Read f...
./actionpack/test/controller/log_subscriber_test.rb: assert_no_match(/Rea...
...
./guides/source/security.md:Note that _cross-site scripting (XSS) vulnerabil...
./guides/source/security.md:#### Unauthorized Reading
./railties/lib/rails/generators/actions.rb: # Reads the given file at t...
./railties/lib/rails/generators/app_base.rb: comment = 'Build JSON AP...
./railties/lib/rails/generators/app_base.rb: "Turbolinks makes fo...
 ./railties/lib/rails/generators/rails/app/templates/app/assets/javascripts/a...
./railties/lib/rails/generators/rails/plugin/templates/rails/javascripts.js:...
./railties/lib/rails/generators/test_unit/model/templates/fixtures.yml:# Rea...
./railties/test/application/middleware/session_test.rb: post '/foo/read...
./railties/test/application/middleware/session_test.rb: post '/foo/read...

Listing 5.97 shows a truncated list of results for searching for the
pattern read in the current directory (specified by the . at the end)
and all subdirectories. The exact number of results I got from this
query was 138, too many to display here.

To trim our result set down even further, we can use the –include
flag to specify a pattern that each file name much match to be
included in the results.

Let’s modify our query to only search for the term Read in yml files.

$ grep -R --include="*.yml" "Read" .

Listing 5.10
./activerecord/test/fixtures/tasks.yml:# Read about fixtures at http://api.r...
./guides/code/getting_started/test/fixtures/comments.yml:# Read about fixtur...
./guides/code/getting_started/test/fixtures/posts.yml:# Read about fixtures ...
./railties/lib/rails/generators/test_unit/model/templates/fixtures.yml:# Rea...

As we can see from Listing 5.10, we have narrowed our results down
to a much more manageable four results.

In the introduction to this chapter I mentioned that both ack and ag

have more modern interfaces than grep. We can see this by writing
the same query again, this time using ag.

$ ag Read -G yml$

Listing 5.11
 activerecord/test/fixtures/tasks.yml:1:# Read about fixtures at http://api.r...
guides/code/getting_started/test/fixtures/comments.yml:1:# Read about fixtur...
guides/code/getting_started/test/fixtures/posts.yml:1:# Read about fixtures ...
railties/lib/rails/generators/test_unit/model/templates/fixtures.yml:1:# Rea...

If you compare Listing 5.108 and Listing 5.119 you’ll see that we get
the same results for both commands.

5.3 Searching from Standard

Input
One of the most common uses for the grep command is to search
through data that has come from the results of another command,
such as the ps command that we will learn about next in Chapter 6.
You will see this usage of grep in a lot of online tutorials and sample
code.

$ ps -e | grep forego

Listing 5.12
98605 ttys005 0:00.86 forego start -p 3000

In Listing 5.12 we used the ps command to get a list of all of the

processes running on my machine. We then used the pipe operator
(|) to send that data to grep to search for the process that matched
the pattern, forego.
5.4 Invert Searches (-v)
One really useful feature of grep is the ability to invert a search
using the -v flag.

Let’s search for all lines in the README.md file that don’t contain the
letter a.

$ grep -v "a" README.md

Listing 5.13
Run with `--help` or `-h` for options.

## Contributing

## License

Listing 5.13 shows that our query was successful, but the results are
slightly strange in that it also returned blank lines as well.

By using what we learned in Section 5.3, we can create a grep query

that finds all blank lines, and then inverts that query to only return
non-blank lines. Next we can use the | operator to send that data to
our second grep query that searches for lines that don’t contain the
letter a.

Listing 5.14 shows the results of us using two grep queries to get all
non-blank lines that don’t contain the letter a.

$ grep -v "^$" README.md | grep -v "a"

Listing 5.14
Run with `--help` or `-h` for options.
## Contributing
## License

5.5 Conclusion
In this chapter we looked at how to use grep to build complex
searches across multiple files.

While I do feel that grep is a very important tool to learn, I believe

that ack and ag are better alternatives. With that said, good
knowledge of grep will certainly come in handy when you are
working on a machine that doesn’t have ack or ag installed on it.

1. This output has been truncated to better fit the format of this book. ↑
2. This output has been truncated to better fit the format of this book. ↑
3. This output has been truncated to better fit the format of this book. ↑
4. This output has been truncated to better fit the format of this book. ↑
5. This output has been truncated to better fit the format of this book. ↑
6. This output has been truncated to better fit the format of this book. ↑
7. This output has been truncated to better fit the format of this book. ↑
8. This output has been truncated to better fit the format of this book. ↑
9. This output has been truncated to better fit the format of this book. ↑
Chapter 6
Ps
Understanding what the processes running on our computers are
doing is important. Which processes are using the most memory?
Which are using the most CPU? These are important questions,
along with how to find a particular process.

The ps command helps us to answer all of these questions. Its

output can often be confusing, as well as its myriad of options and
flags. In this chapter we’ll look, as we have been doing with
commands in the previous chapters, at the most useful flags and
options to use with ps. We’ll also learn how to interpret the most
important parts of ps output.

One thing to note before we start looking at code examples in this

chapter, is that your results will vary from mine. These are the
processes that are running on my computer, which is running Mac
OS X, at the time of writing. They will be different from yours, and
unfortunately there is not much we can do to change that - so please
keep that in mind when your output is different.

6.1 The Basics

Let’s start by simply running the command and see output we get.

$ ps

Listing 6.1
PID TTY TIME CMD
24696 ttys000 0:00.19 -bash
24697 ttys001 0:00.16 -bash
24715 ttys002 0:00.14 -bash
24721 ttys003 0:00.15 -bash
24703 ttys004 0:00.15 -bash
21686 ttys005 0:00.04 ruby /Users/markbates/scripts/ss
21690 ttys005 0:00.38 forego start -p 3000
24755 ttys005 0:00.19 -bash
24851 ttys006 0:00.69 -bash
25865 ttys007 0:00.22 -bash
25876 ttys008 0:00.10 -bash

According to Listing 6.1 I am running a bunch of bash terminal

windows, one ruby process, and one forego process.

You might be wondering where the rest of the processes running on

my computer are. It’s a valid question. In this day and age there’s
no way I only have nine processes running on my machine.

When the ps command is run without any arguments, it will return

results for the processes that belong to the current user, me, that
are associated with the same terminal in which the ps command is
run. Later in this chapter we will look at how to find the rest of the
processes currently running.

6.1.1 Columns

Let’s quickly examine the output of Listing 6.1 and look at the
different columns of data that were printed out:
 Column Definition

PID The process ID of the process.

TTY1 The controlling terminal for the process.

The cumulative CPU time the process has used since

TIME the process started. The format for this time is [
dd-]hh:mm:ss.

CMD The name of the command that is running

6.2 Display User-Oriented

Output (u)
The ps command has a flag, u, that will display common useful
information, such as CPU and memory usage. It should be quickly
noted that this flag, unfortunately does not work with the rest of the
flags in this chapter. Where possible I will note alternative flags in
other sections that are compatible with this flag.

$ ps u

Listing 6.2
USER PID %CPU %MEM VSZ RSS TT STAT STARTED TIME COMMAND
markbates 57896 0.0 0.0 4529729 2720 s005 S+ 10:27AM 0:00.38
forego start -p 3000
markbates 57892 0.0 0.0 2465360 7636 s005 S+ 10:27AM 0:00.04
ruby /Users/markbates/scripts/ss
markbates 25876 0.0 0.0 2489688 2400 s008 S+ Sat10AM 0:00.10 -bash
markbates 25865 0.0 0.0 2497880 2040 s007 S+ Sat10AM 0:00.22 -bash
markbates 24851 0.0 0.0 2497880 3392 s006 S+ Sat10AM 0:00.69 -bash
markbates 24755 0.0 0.0 2497880 2004 s005 S Sat10AM 0:00.21 -bash
markbates 24721 0.0 0.0 2497880 2136 s003 S+ Sat10AM 0:00.15 -bash
markbates 24715 0.0 0.0 2497880 1552 s002 S+ Sat10AM 0:00.14 -bash
 markbates 24703 0.0 0.0 2497880 1424 s004 S+ Sat10AM 0:00.15 -bash
markbates 24697 0.0 0.0 2497880 1948 s001 S+ Sat10AM 0:00.16 -bash
markbates 24696 0.0 0.0 2497880 2396 s000 S+ Sat10AM 0:00.32 -bash

As we can see in Listing 6.22 we are getting information about the

same processes that we saw in Listing 6.1, but now we are seeing
more information about those processes.

6.2.1 Columns

Let’s look at the additional columns that the u flag gave us.

Column Definition

USER The user that the process is running under.

The percentage of time the process has used the CPU

%CPU
since the process started.

%MEM The percentage of real memory used by this process.

Indicates, as a decimal integer, the size in kilobytes

VSZ
of the process in virtual memory.

RSS The real-memory size of the process (in 1 KB units).

STAT The status of the process.

STARTED When the process was started.

The view of the data presented by the u flag is commonly referred to

as “user-oriented” output. The reason for this is simple - this
default set of data is going to be the information you’ll need most
often.
6.3 Display All Processes (-e)
Up until now we have only been getting information about
processes that are being run in a terminal window controlled by us.
There are a lot of other processes running on a modern machine,
and to access those we can use the -e flag.

$ ps -e

Listing 6.3
PID TTY TIME CMD
1 ?? 28:22.66 /sbin/launchd
17 ?? 1:19.09 /usr/libexec/UserEventAgent (System)
18 ?? 0:17.57 /usr/libexec/kextd
19 ?? 0:36.91 /usr/libexec/taskgated -s
20 ?? 0:25.96 /usr/sbin/notifyd
21 ?? 0:19.43 /usr/sbin/securityd -i
22 ?? 0:17.15 /usr/libexec/diskarbitrationd
23 ?? 0:21.38 /System/Library/CoreServices/powerd.bundle/powerd
24 ?? 5:13.79 /usr/libexec/configd
25 ?? 1:01.05 /usr/sbin/syslogd
...
24703 ttys004 0:00.15 -bash
24743 ttys005 0:00.02 login -pfl markbates /bin/bash -c exec -la bash /bin...
24755 ttys005 0:00.21 -bash
57892 ttys005 0:00.04 ruby /Users/markbates/scripts/ss
57896 ttys005 0:00.42 forego start -p 3000
24810 ttys006 0:00.02 login -pfl markbates /bin/bash -c exec -la bash /bin...
24851 ttys006 0:00.70 -bash
25864 ttys007 0:00.01 login -pfl markbates /bin/bash -c exec -la bash /bin...
25865 ttys007 0:00.24 -bash
25866 ttys008 0:00.02 login -pfl markbates /bin/bash -c exec -la bash /bin...
25876 ttys008 0:00.10 -bash

Although Listing 6.33 has been truncated, you can see that we now
have a mix of “system” processes as well as “user” processes.

If you want to use a syntax that’s similar to the u flag from

Section 6.2, you can substitute ax for the -e flag. Although you will
get slightly different columns presented, the listing of processes
should be identical.

Listing 6.44 shows using the ax flag instead of the -e flag.

$ ps ax

Listing 6.4
PID TT STAT TIME COMMAND
1 ?? Ss 28:34.79 /sbin/launchd
17 ?? Ss 1:19.61 /usr/libexec/UserEventAgent (System)
18 ?? Ss 0:17.63 /usr/libexec/kextd
19 ?? Ss 0:37.35 /usr/libexec/taskgated -s
20 ?? Ss 0:26.22 /usr/sbin/notifyd
21 ?? Ss 0:19.52 /usr/sbin/securityd -i
22 ?? Ss 0:17.24 /usr/libexec/diskarbitrationd
23 ?? Ss 0:21.71 /System/Library/CoreServices/powerd.bundle/powerd
24 ?? Us 5:15.98 /usr/libexec/configd
25 ?? Ss 1:01.66 /usr/sbin/syslogd
...
24703 s004 S+ 0:00.15 -bash
24743 s005 Ss 0:00.02 login -pfl markbates /bin/bash -c exec -la bash /b...
24755 s005 S 0:00.21 -bash
57892 s005 S+ 0:00.04 ruby /Users/markbates/scripts/ss
57896 s005 S+ 0:00.43 forego start -p 3000
24810 s006 Ss 0:00.02 login -pfl markbates /bin/bash -c exec -la bash /b...
24851 s006 S+ 0:00.71 -bash
25864 s007 Ss 0:00.01 login -pfl markbates /bin/bash -c exec -la bash /b...
25865 s007 S+ 0:00.31 -bash
25866 s008 Ss 0:00.02 login -pfl markbates /bin/bash -c exec -la bash /b...
25876 s008 S+ 0:00.10 -bash

6.4 Display Processes by User

(-U)
The ps command has a lot of filtering options built into it. One of
the most useful ones is the ability to filter for processes by the user
that is running them.
The -U flag will filter the results to show only processes that are
being run by the user ID provided.

$ ps -U root

Listing 6.5
PID TTY TIME CMD
1 ?? 28:36.67 /sbin/launchd
17 ?? 1:19.66 /usr/libexec/UserEventAgent (System)
18 ?? 0:17.63 /usr/libexec/kextd
19 ?? 0:37.49 /usr/libexec/taskgated -s
20 ?? 0:26.29 /usr/sbin/notifyd
21 ?? 0:19.52 /usr/sbin/securityd -i
22 ?? 0:17.24 /usr/libexec/diskarbitrationd
23 ?? 0:21.76 /System/Library/CoreServices/powerd.bundle/powerd
24 ?? 5:16.27 /usr/libexec/configd
25 ?? 1:01.75 /usr/sbin/syslogd
...
866 ?? 0:00.06 /System/Library/Frameworks/CoreServices.framework/...
1000 ?? 0:18.78 /usr/libexec/systemstatsd
2065 ?? 0:00.53 /System/Library/Filesystems/AppleShare/check_afp.app/...
4804 ?? 0:00.01 /usr/libexec/periodic-wrapper daily
15536 ?? 0:00.00 /usr/sbin/aslmanager -s /var/log/eventmonitor
15537 ?? 0:00.00 /usr/sbin/aslmanager -s /var/log/performance
26161 ?? 0:00.76 /System/Library/Frameworks/CoreMediaIO.framework/...
26865 ?? 0:00.01 com.apple.cmio.registerassistantservice
33266 ?? 0:01.57 /usr/libexec/syspolicyd
64100 ?? 0:00.12 /usr/sbin/ocspd
66252 ?? 0:00.00 /usr/libexec/periodic-wrapper weekly

Compare Listing 6.55 and Listing 6.66 and you can see the different
processes associated with each user.

$ ps -U markbates

Listing 6.6
PID TTY TIME CMD
351 ?? 0:23.19 /sbin/launchd
355 ?? 0:10.19 /usr/libexec/UserEventAgent (Aqua)
356 ?? 1:45.54 /usr/sbin/distnoted agent
359 ?? 0:17.99 /usr/sbin/cfprefsd agent
366 ?? 0:55.47 /System/Library/CoreServices/Dock.app/Contents/MacOS/Dock
367 ?? 6:17.86 /System/Library/CoreServices/SystemUIServer.app/...
368 ?? 2:29.01 /System/Library/CoreServices/Finder.app/Contents/...
371 ?? 0:05.79 /usr/libexec/xpcd
372 ?? 0:04.46 /usr/libexec/sharingd
 373 ?? 0:00.01 /usr/sbin/pboard
374 ?? 0:01.76 /System/Library/PrivateFrameworks/TCC.framework/...
...
24698 ttys004 0:00.02 login -pfl markbates /bin/bash -c exec -la bash /bin/bash
24703 ttys004 0:00.15 -bash
24743 ttys005 0:00.02 login -pfl markbates /bin/bash -c exec -la bash /bin/bash
24755 ttys005 0:00.21 -bash
57892 ttys005 0:00.04 ruby /Users/markbates/scripts/ss
57896 ttys005 0:00.43 forego start -p 3000
24810 ttys006 0:00.02 login -pfl markbates /bin/bash -c exec -la bash /bin/bash
24851 ttys006 0:00.72 -bash
25864 ttys007 0:00.01 login -pfl markbates /bin/bash -c exec -la bash /bin/bash
25865 ttys007 0:00.31 -bash
25866 ttys008 0:00.02 login -pfl markbates /bin/bash -c exec -la bash /bin/bash
25876 ttys008 0:00.10 -bash

On a typical development machine there are probably only two

users, you and the root user. However, on a complex server setup,
there might be several users, and this is a great way to be able to
scope your process search to those users.

6.5 Customize Displayed

Columns (-O, -L)
We can easily customize ps to display a host of different column
information using the -L and -O flags.

To get a list of the available columns that can be displayed with ps

we can use the -L flag.

$ ps -L

Signal Definitions for ps -l

Column Description

%cpu percentage CPU usage (alias pcpu)

%mem percentage memory usage (alias pmem)

acflag accounting flag (alias acflg)

args command and arguments

comm command

command command and arguments

cpu short-term CPU usage factor (for scheduling)

etime elapsed running time

flags the process flags, in hexadecimal (alias f)

gid processes group id (alias group)

inblk total blocks read (alias inblock)

jobc job control count

ktrace tracing flags

ktracep tracing vnode

lim memoryuse limit

logname login name of user who started the session

lstart time started

majflt total page faults

minflt total page reclaims

msgrcv total messages received (reads from pipes/sockets)

msgsnd total messages sent (writes on pipes/sockets)

nice nice value (alias ni)

nivcsw total involuntary context switches

nsigs total signals taken (alias nsignals)

nswap total swaps in/out

nvcsw total voluntary context switches

nwchan wait channel (as an address)

oublk total blocks written (alias oublock)

p_ru resource usage (valid only for zombie)

paddr swap address

pagein pageins (same as majflt)

pgid process group number

pid process ID

ppid parent process ID

pri scheduling priority

re core residency time (in seconds; 127 = infinity)

rgid real group ID

rss resident set size

ruid real user ID

ruser user name (from ruid)

sess session ID

sig pending signals (alias pending)

sigmask blocked signals (alias blocked)

sl sleep time (in seconds; 127 = infinity)

start time started

state symbolic process state (alias stat)

svgid saved gid from a setgid executable

svuid saved UID from a setuid executable

tdev control terminal device number

accumulated CPU time, user + system (alias

time
cputime)

tpgid control terminal process group ID

tsess control terminal session ID

tsiz text size (in Kbytes)

tt control terminal name (two letter abbreviation)

tty full name of control terminal

ucomm name to be used for accounting

uid effective user ID

scheduling priority on return from system call (alias

upr
usrpri)
 user user name (from UID)

utime user CPU time (alias putime)

vsz virtual size in Kbytes (alias vsize)

wchan wait channel (as a symbolic name)

wq total number of workqueue threads

wqb number of blocked workqueue threads

wqr number of running workqueue threads

workqueue limit status (C = constrained thread limit,

wql
T = total thread limit)

exit or stop status (valid only for stopped or zombie

xstat
process)

We can use a few of the columns from Table 6.57 to display

information such as the elapsed time (etime), the current CPU
usage (%cpu), and the current memory usage (%mem).

By passing the column names to the -O flag as a comma separated

list, we get the results we see in Listing 6.7.

$ ps -O etime,%cpu,%mem

Listing 6.7
PID ELAPSED %CPU %MEM TT STAT TIME COMMAND
24696 03-06:31:19 0.0 0.0 s000 S+ 0:00.34 -bash
24697 03-06:31:19 0.0 0.0 s001 S+ 0:00.16 -bash
24715 03-06:31:19 0.0 0.0 s002 S+ 0:00.14 -bash
24721 03-06:31:19 0.0 0.0 s003 S+ 0:00.22 -bash
24703 03-06:31:19 0.0 0.0 s004 S+ 0:00.15 -bash
24755 03-06:31:19 0.0 0.0 s005 S 0:00.21 -bash
 57892 06:11:07 0.0 0.0 s005 S+ 0:00.04 ruby /Users/markbates/scripts/ss
57896 06:11:07 0.0 0.0 s005 S+ 0:00.46 forego start -p 3000
24851 03-06:31:19 0.0 0.0 s006 S+ 0:00.72 -bash
25865 03-06:31:19 0.0 0.0 s007 S+ 0:00.31 -bash
25876 03-06:31:19 0.0 0.0 s008 S+ 0:00.14 -bash

It’s important to note that when we look at things like current CPU
and memory usage with ps, they are static snapshots of those
numbers at that moment in time. They are not “live” numbers that
will continue to update in real-time. Should you need real-time
data, you might consider using the top command instead.

6.6 Sorting
The ps command gives us a few different ways to sort the results we
get. Two of the more useful ones are the -m flag, which sorts by
memory usage, and the -r flag, that sorts by CPU usage.

While looking at both of these examples, we’ll use the -u flag that
we learned about in Section 6.4 to scope results to the root user.
We’ll also use the -O flag we saw in Section 6.5 to display either
memory or CPU usage.

6.6.1 Sort by Memory Usage (-m)

Using the -m flag we can sort processes by their memory usage.

$ ps -m -O %mem -u root

Listing 6.8
 PID %MEM TT STAT TIME COMMAND
154 1.5 ?? Ss 66:49.98 /System/Library/Frameworks/ApplicationServices...
174 1.0 ?? Ss 29:44.90 /System/Library/Frameworks/CoreServices.framew...
100 0.4 ?? Ss 49:42.27 /System/Library/Frameworks/CoreServices.framew...
70322 0.3 ?? Ss 0:04.58 /System/Library/PrivateFrameworks/PackageKit.f...
128 0.3 ?? Ss 650:05.90 /Applications/Leap Motion.app/Contents/MacOS/l...
5835 0.2 ?? Ss 0:01.46 /usr/sbin/ocspd
115 0.2 ?? Ss 1:46.32 /System/Library/CoreServices/launchservicesd
103 0.1 ?? Ss 0:16.27 /System/Library/CoreServices/loginwindow.app/C...
337 0.1 ?? S 29:54.94 /Library/Parallels/Parallels Service.app/Conte...
51 0.1 ?? Ss 11:47.34 /System/Library/Frameworks/CoreServices.framew...
17 0.1 ?? Ss 1:41.91 /usr/libexec/UserEventAgent (System)
28 0.1 ?? Ss 2:14.83 /usr/libexec/opendirectoryd
86 0.1 ?? SNs 0:38.33 /usr/libexec/warmd
32 0.1 ?? Ss 1:17.58 /System/Library/CoreServices/coreservicesd
26161 0.1 ?? Ss 0:00.96 /System/Library/Frameworks/CoreMediaIO.framewo...
349 0.1 ?? Ss 0:01.04 /System/Library/PrivateFrameworks/SoftwareUpda...
19 0.0 ?? Ss 0:48.08 /usr/libexec/taskgated -s
...

Listing 6.88 shows the top processes for the root user, sorted by
their memory usage.

6.6.2 Sort by CPU Usage (-r)

The -r flag will allows us to sort processes by their CPU usage.

$ ps -r -O %cpu -u root

Listing 6.9
PID %CPU TT STAT TIME COMMAND
130 50.6 ?? SNs 2303:23.91 /usr/bin/java -Dapp=CrashPlanService -Xmn10m...
128 41.2 ?? Rs 656:28.86 /Applications/Leap Motion.app/Contents/MacOS/...
154 3.0 ?? Us 68:09.18 /System/Library/Frameworks/ApplicationService...
115 2.3 ?? Ss 2:16.12 /System/Library/CoreServices/launchservicesd
337 1.7 ?? S 30:55.44 /Library/Parallels/Parallels Service.app/Cont...
32 1.1 ?? Ss 1:39.38 /System/Library/CoreServices/coreservicesd
23 0.4 ?? Ss 0:30.84 /System/Library/CoreServices/powerd.bundle/po...
100 0.3 ?? Ss 51:01.97 /System/Library/Frameworks/CoreServices.frame...
25 0.3 ?? Ss 1:24.04 /usr/sbin/syslogd
19 0.2 ?? Ss 0:51.95 /usr/libexec/taskgated -s
51 0.1 ?? Ss 12:20.59 /System/Library/Frameworks/CoreServices.frame...
174 0.1 ?? Ss 30:38.53 /System/Library/Frameworks/CoreServices.frame...
864 0.0 ?? Ss 0:00.06 /sbin/launchd
...
Listing 6.99 shows the top processes for the root user, sorted by
their CPU usage.

6.7 Conclusion
The ps command is one of those commands that you’ll use almost
daily. With the numerous ways you can sort, filter, and present
processes, it is easy to find the process information you are looking
for.

I would also recommend checking out Chapter 2 Section 2.7 to see

how to combine ps with Ack or Ag to find the exact process you are
looking for. Alternatively, you can find the same example, this time
using grep, in Chapter 5 Section 5.3.

1. http://en.wikipedia.org/wiki/Tty ↑
2. The listing has been wrapped to fit in the box. ↑
3. This output has been truncated to better fit the format of this book. ↑
4. This output has been truncated to better fit the format of this book. ↑
5. This output has been truncated to better fit the format of this book. ↑
6. This output has been truncated to better fit the format of this book. ↑

7. For a full list of definitions please check out:

http://publib.boulder.ibm.com/infocenter/pseries/v5r3/index.jsp?
topic=/com.ibm.aix.cmds/doc/aixcmds4/ps.htm

↑
8. This output has been truncated to better fit the format of this book. ↑
9. This output has been truncated to better fit the format of this book. ↑
Chapter 7
Sed
sed is a stream editor that lets you quickly edit files or streams
using pattern matching and replacements.

The sed command can be incredibly useful when bootstrapping a

new server, or machine, as it can easily be scripted. A common use
case for sed is to script the editing of configuration files on a new
server instance to facilitate the further setup of the needed
environment for that machine.

For this chapter we’ll be using the file in Listing 7.1 as the basis for
all of our examples.

$ cat birthday.txt

Listing 7.1
Happy Birthday to You, cha, cha, cha.
Happy Birthday to You, cha, cha, cha.
Happy Birthday Dear (name)
Happy Birthday to You, cha, cha, cha.

7.1 Installation (Mac OS X)

Mac OS X ships with a version of sed that, while mostly functional,
is feature incomplete, and in some cases just broken. To fix this, I
would recommend installing gnu-sed to replace the built-in version
of sed on OS X machines.

If you are on a Mac, the best and easiest way to install gnu-sed is
using Homebrew.

$ brew install gnu-sed --default-names

I would recommend a restart of your terminal before you continue

to ensure you are using the correct binary.

7.2 The Basics (s)

As mentioned earlier, sed is an editor that allows us to use pattern
matching to search and replace within a file, or an input stream.
This works by using Regular Expressions.

By default, the results of any edits we make to a source file, or input

stream, will be sent to STDOUT, the standard output. The original file
will be unaffected by the edits. In Section 7.3 we will see how to
update the original file directly with edits.

Let’s use sed to replace all vowels from our example file and replace
them with *.

$ sed "s/[aeiou]/*/" birthday.txt

Listing 7.2
H*ppy Birthday to You, cha, cha, cha.
H*ppy Birthday to You, cha, cha, cha.
H*ppy Birthday Dear (name)
H*ppy Birthday to You, cha, cha, cha.

In Listing 7.2 we can see that only the first vowel on each line was
replaced with a *. Before we talk about why that is and how to fix it,
let’s examine what we told sed to do.

In this example we passed sed two arguments. The first argument is

what we want sed to do, in the form of s/regexp/replacement/. It’s
important to note that I wrapped the first argument in quotes to
escape it. The second argument is the path to a file we want to read
from.

7.2.1 Global Replace

In Listing 7.2 we saw that only the first vowel in each line was
replaced. We can update our regular expression to use the g option,
which means “global”, to replace all vowels on each line with *.

$ sed "s/[aeiou]/*/g" birthday.txt

Listing 7.3
H*ppy B*rthd*y t* Y**, ch*, ch*, ch*.
H*ppy B*rthd*y t* Y**, ch*, ch*, ch*.
H*ppy B*rthd*y D**r (n*m*)
H*ppy B*rthd*y t* Y**, ch*, ch*, ch*.

As we can see in Listing 7.3 we have successfully replaced the

vowels from our example file.
7.2.2 nth Replace

sed has a rather interesting feature in that it will allow us to replace

the nth occurrence of a pattern.

Let’s write a sed script that will replace the second occurrence of the
word cha on each line with the word foo.

$ sed "s/cha/foo/2" birthday.txt

Listing 7.4
Happy Birthday to You, cha, foo, cha.
Happy Birthday to You, cha, foo, cha.
Happy Birthday Dear (name)
Happy Birthday to You, cha, foo, cha.

In Listing 7.4 we replaced the g option with the number 2 to tell sed
to replace the second occurrence of the pattern on each line that it is
found.

7.3 Saving Files

By default, sed will print its output to the standard output, STDOUT,
but sometimes we want to be able to update the source file directly.

7.3.1 Editing in Place (-i)

On most Unix and Linux systems you can use the -i flag to allow
sed to edit the file in place. This means your edits will be saved
directly to the file.
Let’s use -i to replace the (name) placeholder in the file with the
name Mark.

$ sed -i "s/(name)/Mark/" birthday.txt

When using the -i flag there will be no output if the command was
successful. We can print the contents of the file to the screen to
verify our edits using the cat command (that we’ll learn about in
Section 9.2), as seen in Listing 7.5.

Listing 7.5: $ cat birthday.txt

Happy Birthday to You, cha, cha, cha.
Happy Birthday to You, cha, cha, cha.
Happy Birthday Dear Mark
Happy Birthday to You, cha, cha, cha.

7.3.2 Using Backup Files

The -i flag lets you optionally pass it an argument that represents

an extension for a backup file.

$ sed -i.tmp "s/(name)/Mark/" birthday.txt

Listing 7.6: $ ls -la

drwxr-xr-x 7 markbates staff 238 Dec 11 15:10 .
drwxr-xr-x 12 markbates staff 408 Dec 11 10:43 ..
-rw-r--r-- 1 markbates staff 139 Dec 11 15:10 birthday.txt
-rw-r--r-- 1 markbates staff 141 Dec 11 10:22 birthday.txt.tmp

We can see in Listing 7.6 that we now have a second file,

birthday.txt.tmp, named after the original file, birthday.txt, but
the new file contains the original content before the substitution.
Using backup files allows us to perform the edits and verify that the
file is correct without losing the original content. If we are satisfied
with our changes, we can delete the backup file; or, if we need to
revert the edits, we can simply rename the .tmp file to the original
file name.

7.4 Case Insensitive Searches

sed supports the Regular Expression option i to make its search
pattern case insensitive.

Let’s replace all instances of the word happy, regardless of its case,
to the word Merry.

$ sed "s/happy/Merry/i" birthday.txt

Listing 7.7
Merry Birthday to You, cha, cha, cha.
Merry Birthday to You, cha, cha, cha.
Merry Birthday Dear (name)
Merry Birthday to You, cha, cha, cha.

7.5 Matching Lines

So far we have been operating on every line in the example file, but
what if we only want to operate on certain lines in the file?
Let’s write a sed script that will replace every character in a line that
matches cha with the character *.1

$ sed "/cha/s/./*/g" birthday.txt

Listing 7.8
*************************************
*************************************
Happy Birthday Dear (name)
*************************************

As you can see in Listing 7.8 we proceeded our search pattern with
/cha/. This tells sed to match the line against this pattern, and if
the line matches, then it performs the subsequent search and
replace.

7.6 Manipulating Matches

sed provides several ways to manipulate the matches during the
replacement phase. Let’s look at a couple of those now.

7.6.1 Changing Case

The & variable can be used in the replace section of your sed script
to represent a match from the search part of the script.

sed has several special characters we can use in the replace section
of our script to manipulate the data we get using the & variable.
The first such character we use is \u. The \u character will convert
whatever comes after it to uppercase.

Let’s write a script that combines \u and & to capitalize all of the
vowels in the example file.

$ sed 's/[aeiou]/\u&/g' birthday.txt

Listing 7.9
HAppy BIrthdAy tO YOU, chA, chA, chA.
HAppy BIrthdAy tO YOU, chA, chA, chA.
HAppy BIrthdAy DEAr (nAmE)
HAppy BIrthdAy tO YOU, chA, chA, chA.

Listing 7.9 shows that we have successfully managed to capitalize

the vowels in the file.

We can use the same technique, this time using the \l character, to
lower the case of all of the capital letters in the example file.

$ sed 's/[A-Z]/\l&/g' birthday.txt

Listing 7.10
happy birthday to you, cha, cha, cha.
happy birthday to you, cha, cha, cha.
happy birthday dear (name)
happy birthday to you, cha, cha, cha.

In Listing 7.10 we successfully converted all capital letters to

lowercase letters.

7.7 Controlling Output (-n)

Up until now, sed has been printing its output, by default, to the
console or standard output (STDOUT). In Section 7.3 we learned how
to save the output to a file, but what if we want to control what gets
printed to the console?

Using the -n flag we can suppress the default output of sed. For
example, the following command won’t print anything to the
console window.

$ sed -n "s/(name)/Mark/" birthday.txt

7.7.1 Printing Modified Content (p)

Suppressing all of the output to the console can be useful, but

usually we want to display some output.

By using the p option we can tell sed to print only the changes to the
console for us.

$ sed -n "s/(name)/Mark/p" birthday.txt

Listing 7.11
Happy Birthday Dear Mark

In Listing 7.11 we see just the one line of the file that the search and
replace query affected.

7.7.2 Writing Modified Content to a File (w)

Using the -n flag and the p option, we can write only modified lines
to a separate file using the w option and giving it a file name.

$ sed -n "s/(name)/Mark/pw birthday2.txt" birthday.txt

Listing 7.12: $ cat birthday2.txt

Happy Birthday Dear Mark

Listing 7.12 shows the contents of the birthday2.txt file containing

only the single line that we wanted to change.

7.7.3 Displaying Specific Lines

The -n flag takes an optional argument that we can use to print out
specific lines of content.

Using the -n flag and p option, we can write a query that will print
every other line of our example file to the console.

$ sed -n '1~2p' birthday.txt

Listing 7.13
Happy Birthday to You, cha, cha, cha.
Happy Birthday Dear (name)

In Listing 7.13 we passed an argument to the -n flag. This argument

can be read as starting at line 1 print (p) every 2nd line.

7.8 Delete Lines (d)

So far we have seen how to search and replace patterns with sed, but
what if we simply want to remove a particular line from a file?

Using the d option, we can tell sed to delete any lines that match a
specific pattern.

$ sed "/(name)/d" birthday.txt

Listing 7.14
Happy Birthday to You, cha, cha, cha.
Happy Birthday to You, cha, cha, cha.
Happy Birthday to You, cha, cha, cha.

In Listing 7.14 we told sed to delete any lines that match the pattern
(name). It should be pointed out that this is only changing the
output, and has left the original file untouched. If you want to write
these modifications back to the file, you can use the techniques we
learned about in Section 7.3.1.

7.9 Running Multiple sed

Commands (;)
At times it may be useful to run multiple sed commands at the same
time across a specific piece of content. sed allows us to do this
using the ; operator.

Listing 7.15 shows the contents of the file we’re going to use for the
examples in this section.
Listing 7.15: $ cat example.html
<p>
This <i>is</i> some <b>HTML</b>.
</p>
<a href="http://www.conqueringthecommandline.com">Conquering the Command Line</a>

Lets use sed to strip all of the HTML tags out of the file.

$ sed 's/<[^>]*>//g' example.html

Listing 7.16

This is some HTML.

Conquering the Command Line

Listing 7.16 doesn’t use anything new about sed that we don’t
already know, just a different search and replace pattern.

The problem with Listing 7.16 is that while it stripped out all of the
HTML tags from the file, it left us with blank lines, as we can see in
Listing 7.17

Listing 7.17

This is some HTML.

Conquering the Command Line

There are a couple of different ways we can fix this problem. The
first way would be use the pipe operator, |, to take the output of
Listing 7.16 and send it to a second sed command to delete the blank
lines using the d option we learned about in Section 7.8. We can see
the results of this approach in Listing 7.18
 $ sed 's/<[^>]*>//g' example.html | sed '/^$/d'

Listing 7.18
This is some HTML.
Conquering the Command Line

While Listing 7.18 certainly achieves our intended goal to remove

both the HTML tags and any blank lines, there is a second, more
efficient way of running both commands - and that is to use the ;
operator that sed provides.

$ sed 's/<[^>]*>//g;/^$/d' example.html

Listing 7.19
This is some HTML.
Conquering the Command Line

As Listing 7.19 shows, we are able to concatenate the two commands

together using the ; operator. The results of Listing 7.19 are the
same as Listing 7.18, but with slightly faster execution time, and
less typing.

7.10 Conclusion
As we’ve seen in this chapter, sed is a very simple editor that allows
us to quickly edit files or streams. A basic understanding of sed can
come in very handy when you’re working on a bare machine that
has yet to have a more sophisticated editor installed.
sed can easily be scripted. This makes it a great tool to use in
deployment scripts to update configuration files or other similar
operations.

1. Feel free and use this script on all of those top secret government projects you are
working on. ↑
Chapter 8
Tar
There will come a time when you want to take several files and
archive them together into a single file - this is where the tar (tape
archive) command comes in.

tar can quickly join together multiple files into one larger file, while
still preserving meta-data such as Unix permissions. By default tar
does not compress files, but it does have a flag that will compress
the archive using gzip. We’ll learn about this later in Section 8.7.

The source code we’ll be using for this chapter’s examples is the
Rails source code, as mentioned in the Preface.

8.1 The Basics (-c)

Let’s start by building a very simple archive, also commonly referred
to as a tarball. To do that, we use the -c flag on the tar command
to tell it that we want to create a new archive, and give it a list of
files we want to archive. In this case we want to archive all of the
files in the tools directory.

$ tar -c tools/
The results of our command can be seen in Listing 8.1.

Listing 8.1
tools/000755 000765 000024 00000000000 12244661361 013337
5ustar00markbatesstaff000000 000000 tools/console000755 000765 000024 00000000255
12244661361 014731 0ustar00markbatesstaff000000 000000 #!/usr/bin/env ruby
require File.expand_path('../../load_paths', __FILE__)
require 'rails/all'
require 'active_support/all'
require 'irb'
require 'irb/completion'
IRB.start
tools/profile000755 000765 000024 00000003451 12244661361 014730
0ustar00markbatesstaff000000 000000 #!/usr/bin/env ruby
# Example:
# tools/profile activesupport/lib/active_support.rb
ENV['NO_RELOAD'] ||= '1'
ENV['RAILS_ENV'] ||= 'development'

require 'benchmark'

module RequireProfiler
private
def require(file, *args) RequireProfiler.profile(file) { super } end
def load(file, *args) RequireProfiler.profile(file) { super } end

@depth, @stats = 0, []
class << self
attr_accessor :depth
attr_accessor :stats

def profile(file)
stats << [file, depth]
self.depth += 1
result = nil
elapsed = Benchmark.realtime { result = yield }
self.depth -= 1
stats.pop if stats.last.first == file
stats << [file, depth, elapsed] if result
result
end
end
end

GC.start
before_rss = `ps -o rss= -p #{Process.pid}`.to_i

path = ARGV.shift
if mode = ARGV.shift
require 'ruby-prof'
RubyProf.measure_mode = RubyProf.const_get(mode.upcase)
RubyProf.start
else
Object.instance_eval { include RequireProfiler }
end

elapsed = Benchmark.realtime { require path }

results = RubyProf.stop if mode
 GC.start
after_rss = `ps -o rss= -p #{Process.pid}`.to_i

if mode
if printer = ARGV.shift
RubyProf.const_get("#{printer.to_s.classify}Printer").new(results)
elsif RubyProf.const_defined?(:CallStackPrinter)
File.open("#{File.basename(path, '.rb')}.#{mode}.html", 'w') do |out|
RubyProf::CallStackPrinter.new(results).print(out)
end
else
File.open("#{File.basename(path, '.rb')}.#{mode}.callgrind", 'w') do |out|
RubyProf::CallTreePrinter.new(results).print(out)
end
end
end

RequireProfiler.stats.each do |file, depth, sec|

if sec
puts "%8.1f ms %s%s" % [sec * 1000, ' ' * depth, file]
else
puts "#{' ' * (13 + depth)}#{file}"
end
end
puts "%8.1f ms %d KB RSS" % [elapsed * 1000, after_rss - before_rss]

The first thing that should be pointed out is the apparent

“gibberish” preceding each of the files that makes up the archive.
This “gibberish” is, in fact, the Unix meta-data of that file; including
such things as the permissions, file’s owner, group, etc - all valuable
information to be retained.

The second thing that should be noted is that the archive was
printed to “standard out” (STDOUT)1 and not saved to a file. We will
learn how to save the archive to a file in Section 8.2.

Writing archives to the STDOUT instead of a file can be useful when

you want to pipe the output of the archive into another command.
We won’t be doing any of that in this chapter, but it is useful to
know that tar can create an archive without having to save it to a
file.
8.2 Saving to a File (-f)
Being able to write archives to STDOUT does have it uses, but a
majority of the time you’ll want to create a new file representing the
archive you’ve just created. The -f flag in tar lets us specify a file
that we want to save the archive to.

$ tar -cf foo.tar tools/

If the creation of the archive file was successful, the tar command
will produce no output.

Listing 8.2: $ ls -la

drwxr-xr-x 28 markbates staff 952 Nov 25 11:08 .
drwxr-xr-x 8 markbates staff 272 Nov 25 09:29 ..
drwxr-xr-x 14 markbates staff 476 Nov 25 11:07 .git
-rw-r--r-- 1 markbates staff 508 Nov 25 09:47 .gitignore
-rw-r--r-- 1 markbates staff 853 Nov 25 09:47 .travis.yml
-rw-r--r-- 1 markbates staff 68 Nov 25 09:47 .yardopts
-rw-r--r-- 1 markbates staff 1203 Nov 25 09:47 CONTRIBUTING.md
-rw-r--r-- 1 markbates staff 2337 Nov 25 09:47 Gemfile
-rw-r--r-- 1 markbates staff 11 Nov 25 09:47 RAILS_VERSION
-rw-r--r-- 1 markbates staff 4118 Nov 25 09:47 README.md
-rw-r--r-- 1 markbates staff 7669 Nov 25 09:47 RELEASING_RAILS.rdoc
-rw-r--r-- 1 markbates staff 2844 Nov 25 09:47 Rakefile
drwxr-xr-x 9 markbates staff 306 Nov 25 09:47 actionmailer
drwxr-xr-x 10 markbates staff 340 Nov 25 09:47 actionpack
drwxr-xr-x 10 markbates staff 340 Nov 25 09:47 actionview
drwxr-xr-x 10 markbates staff 340 Nov 25 09:47 activemodel
drwxr-xr-x 11 markbates staff 374 Nov 25 09:47 activerecord
drwxr-xr-x 10 markbates staff 340 Nov 25 09:47 activesupport
drwxr-xr-x 3 markbates staff 102 Nov 25 09:47 ci
-rw-r--r-- 1 markbates staff 5120 Nov 25 11:08 foo.tar
drwxr-xr-x 12 markbates staff 408 Nov 25 09:47 guides
-rw-r--r-- 1 markbates staff 524 Nov 25 09:47 install.rb
-rw-r--r-- 1 markbates staff 51 Nov 25 09:47 load_paths.rb
-rw-r--r-- 1 markbates staff 1159 Nov 25 09:47 rails.gemspec
drwxr-xr-x 12 markbates staff 408 Nov 25 09:47 railties
drwxr-xr-x 3 markbates staff 102 Nov 25 09:47 tasks
drwxr-xr-x 4 markbates staff 136 Nov 25 09:47 tools
-rw-r--r-- 1 markbates staff 159 Nov 25 09:47 version.rb
Listing 8.2 shows the results of querying the current directory. In
that list of files we can see the foo.tar file we created.

8.2.1 Verbose Mode (-v)

Using the -v flag we can see which files are being added to our
archive. This will output the list of files in the archive after it has
been generated.

$ tar -cvf foo.tar tools/

Listing 8.3
a tools
a tools/console
a tools/profile

Listing 8.3 shows the output of the tar command using the -v flag.
We will be using this flag for the rest of the examples in this
chapter.

8.2.2 Multiple Directories/Files

Up until now we have been creating archives by passing in a single

directory, however we can pass in any number of directories or files
to put in the archive. We just need to pass them as extra arguments
at the end of the tar command.

$ tar -cvf foo.tar tools/ tasks/ install.rb rails.gemspec version.rb

Listing 8.4
 a tools
a tools/console
a tools/profile
a tasks
a tasks/release.rb
a install.rb
a rails.gemspec
a version.rb

8.2.3 Building from find (-T)

In Chapter 4 we learned about the find tool and how to use it to

find specific files matching a pattern. Using the -T flag and the pipe
operator (|), we can use find to generate a list of files and then use
tar to archive those files.

First let’s build a find query that finds all of the files ending in
.gemspec.

$ find . -name \*.gemspec

Listing 8.5
./actionmailer/actionmailer.gemspec
./actionpack/actionpack.gemspec
./actionview/actionview.gemspec
./activemodel/activemodel.gemspec
./activerecord/activerecord.gemspec
./activesupport/activesupport.gemspec
./rails.gemspec
./railties/lib/rails/generators/rails/plugin/templates/%name%.gemspec
./railties/railties.gemspec

Listing 8.5 shows that our find query is working correctly. Now we
can use | to pipe the output of the find query to the tar command.

The -T flag will read in a list of file names from another file. In our
case we are going to use - in place of the name of a file. This tells
tar that it should read in the list of files from the find query we
piped in.

$ find . -name \*.gemspec | tar -cvf gemspecs.tar -T -

Listing 8.6
a ./actionmailer/actionmailer.gemspec
a ./actionpack/actionpack.gemspec
a ./actionview/actionview.gemspec
a ./activemodel/activemodel.gemspec
a ./activerecord/activerecord.gemspec
a ./activesupport/activesupport.gemspec
a ./rails.gemspec
a ./railties/lib/rails/generators/rails/plugin/templates/%name%.gemspec
a ./railties/railties.gemspec

In Listing 8.6 we can see that the files from our original find query
have been added to the archive.

This demonstrates how incredibly useful these types of programs

can be. They are powerful and useful individually, but when you
start to combine them together you can quickly make quite complex
applications from these simple building blocks.

8.3 Listing Files (-t)

In Section 8.2.2 we created an archive that contains multiple files,
and because we used the -v flag, we saw which files were included
in that archive when we created it. If we were to hand this archive
over to someone else, how would they know what files were in it?
For that matter, how would we know what files were in it should we
come back to it at a later date?
To list files in an existing archive we can use the -t flag.

$ tar -tf foo.tar

Listing 8.7
tools/
tools/console
tools/profile
tasks/
tasks/release.rb
install.rb
rails.gemspec
version.rb

Listing 8.7 lists the files in the archive that we created in

Section 8.2.2.

8.4 Appending Files (-r)

This flag is not compatible with compressed archives, Section 8.7.

In Section 8.2.2 we created an archive that contains several files.

Now we’ve realized that we have forgotten a few files that need to be
added. We can remedy this situation in two ways. The first way is
delete the archive and start again. The second way is to use the -r
flag to append the additional files to the archive.

$ tar -rvf foo.tar Gemfile README.md

Listing 8.8
a Gemfile
a README.md
Listing 8.9: $ tar -tf foo.tar
tools/
tools/console
tools/profile
tasks/
tasks/release.rb
install.rb
rails.gemspec
version.rb
Gemfile
README.md

We can see in Listing 8.8 and Listing 8.9 that by using the -r flag,
the files we have specified have been added to the archive.

8.5 Updating a File (-u)

This flag is not compatible with compressed archives, Section 8.7.

As was the case in Section 8.4 where we needed to add missing files
to the archive, sometimes it is necessary to update files that are
already in the archive. To do this we can use the -u flag.

$ tar -uvf foo.tar version.rb

Listing 8.10
a version.rb

We have successfully updated the archive with the new version of

the file, and it was much quicker than having to rebuild the entire
archive again.
8.6 Extracting Files (-x)
Up until this point we have focused on getting files into archives,
but what about getting them out again? To extract all of our files
into the current directory we can use the -x flag.

$ tar -xvf foo.tar

Listing 8.11
x tools/
x tools/console
x tools/profile
x tasks/
x tasks/release.rb
x install.rb
x rails.gemspec
x version.rb
x Gemfile
x README.md

Listing 8.11 shows the files that have been extracted from the
archive. These files will also be restored with their original meta-
data, permissions, owner, etc…

8.6.1 Extracting a Single File

Extracting the entire archive is the most common use case,

however, there will be times when you’ll only want to extract one
file from the archive. To extract a specific file(s), we can just list
them at the end of the extraction call - much like how we listed
them when creating the archive.

$ tar -xvf foo.tar rails.gemspec

Listing 8.12
x rails.gemspec

Listing 8.12 demonstrates how we are able to extract just the

rails.gemspec file from the archive.

8.6.2 Extracting to a Different Folder (-C)

Up until this point we have been extracting all of the files to the
current directory we are in, but we can easily extract to a different
location.

First, let’s create a new folder, named myfolder, that we will extract
all of the files to.

$ mkdir myfolder

Using the -C flag we can pass a folder to the tar command as the
location we want to extract the file to.

$ tar -xvf foo.tar -C ./myfolder

Listing 8.13
x tools/
x tools/console
x tools/profile
x tasks/
x tasks/release.rb
x install.rb
x rails.gemspec
x version.rb
x Gemfile
x README.md
Listing 8.13 shows that all of the files have been extracted. We can
use the find tool we learned about in Chapter 4 to look at the
structure of the myfolder directory.

$ find ./myfolder

Listing 8.14
./myfolder
./myfolder/Gemfile
./myfolder/install.rb
./myfolder/rails.gemspec
./myfolder/README.md
./myfolder/tasks
./myfolder/tasks/release.rb
./myfolder/tools
./myfolder/tools/console
./myfolder/tools/profile
./myfolder/version.rb

As we can see in Listing 8.14 all of the files were extracted to their
appropriate locations based on how they were archived.

8.7 Compressing with gzip (-z)

As mentioned at the very beginning of the chapter, tar does not
compress archives by default, it just generates a single large file
from multiple smaller files. With that said, tar does provide three
different compression algorithms to you. The first one, gzip (-z),
is the most common, so that is the one we will discuss here. The
others are bzip2 (-j) and Unix compress (-Z). bzip2 is considerably
slower for compression and decompression, but produces smaller
archives, and Unix compress is faster, but produces considerably
larger archives.
Let’s start by creating a regular archive. We will archive all of the
files in the activerecord directory2 .

$ tar -cf ar.tar activerecord/

Listing 8.15: $ ls -la ar.*

-rw-r--r-- 1 markbates staff 3311104 Nov 26 14:12 ar.tar

Listing 8.15 shows us that our new archive, ar.tar, is a whopping

3.3Mb. We can use the -z flag to run the archive through gzip and
compress it to make it smaller. When we do this it is customary to
append .gz to the end of the archive’s file name to let others know
that this archive has been compressed3 .

$ tar -czf ar.tar.gz activerecord/

Listing 8.16: $ ls -la ar.*

-rw-r--r-- 1 markbates staff 3311104 Nov 26 14:12 ar.tar
-rw-r--r-- 1 markbates staff 615262 Nov 26 14:14 ar.tar.gz

In Listing 8.16 we can see that the compressed version of the

archive weighs in at only 615Kb compared to the 3.3Mb of the
uncompressed archive; a significant reduction in file size.

8.8 Extracting Compressed

Files (-z)
Extracting files from a compressed archive is not much different
than how we learned to extract files in Section 8.6. All we need to
do is add the -z flag to the extract command so it first
uncompresses the archive and then extracts the files.

$ tar -xvzf ar.tar.gz

Listing 8.17
x activerecord/
x activerecord/activerecord.gemspec
x activerecord/CHANGELOG.md
x activerecord/examples/
...
x activerecord/lib/active_record/associations/builder/singular_association.rb
x activerecord/examples/.gitignore
x activerecord/examples/performance.rb
x activerecord/examples/simple.rb

Listing 8.17 shows a truncated list of the files that were extracted.

8.9 Conclusion
In this chapter we learned how to use tar to:

Build an archive from multiple files and directories, while

retaining their meta-data, such as their permissions.
Manipulate existing archives by appending new files and
updating existing files in the archive.
Extract all of the files from an archive, as well as extracting
individual files.
Compress archives to reduce the archive’s overall file size.
The tar command is a powerful and easy to master tool, as long as
you can remember the few flags and options we talked about in this
chapter.

1. Standard out is where Unix/Linux will write output by default. Typically this is the
console window. ↑
2. I’m not going to show you the verbose output of this command, as it is rather lengthy
and doesn’t really tell us anything important. Feel free and add the -v flag on your
own to see the output. ↑
3. I’m not going to show you the verbose output of this command, as it is rather lengthy
and doesn’t really tell us anything important. Feel free and add the -v flag on your
own to see the output. ↑
Chapter 9
Extras
This chapter contains a few little extra commands that I think are
both fun and useful. Some are not available on all systems, but
hopefully they’ll be available on yours.

Try combining some of these commands with other commands in

this book to create new workflows.

9.1 Cal
If you spend your life living in a terminal window, the cal command
can be quite useful for helping you check dates.

9.1.1 Display Current Month

The basic usage of cal will print out the current month’s calendar,
as seen in Listing 9.1.

$ cal

Listing 9.1
 December 2013
Su Mo Tu We Th Fr Sa
1 2 3 4 5 6 7
8 9 10 11 12 13 14
15 16 17 18 19 20 21
22 23 24 25 26 27 28
29 30 31

9.1.2 Display Current Year

If you need to see the current year’s calendar at a glance, the -y flag
will print the full year’s calendar.

$ cal -y

Listing 9.2
2013

January February March

Su Mo Tu We Th Fr Sa Su Mo Tu We Th Fr Sa Su Mo Tu We Th Fr Sa
1 2 3 4 5 1 2 1 2
6 7 8 9 10 11 12 3 4 5 6 7 8 9 3 4 5 6 7 8 9
13 14 15 16 17 18 19 10 11 12 13 14 15 16 10 11 12 13 14 15 16
20 21 22 23 24 25 26 17 18 19 20 21 22 23 17 18 19 20 21 22 23
27 28 29 30 31 24 25 26 27 28 24 25 26 27 28 29 30
31
April May June
Su Mo Tu We Th Fr Sa Su Mo Tu We Th Fr Sa Su Mo Tu We Th Fr Sa
1 2 3 4 5 6 1 2 3 4 1
7 8 9 10 11 12 13 5 6 7 8 9 10 11 2 3 4 5 6 7 8
14 15 16 17 18 19 20 12 13 14 15 16 17 18 9 10 11 12 13 14 15
21 22 23 24 25 26 27 19 20 21 22 23 24 25 16 17 18 19 20 21 22
28 29 30 26 27 28 29 30 31 23 24 25 26 27 28 29
30
July August September
Su Mo Tu We Th Fr Sa Su Mo Tu We Th Fr Sa Su Mo Tu We Th Fr Sa
1 2 3 4 5 6 1 2 3 1 2 3 4 5 6 7
7 8 9 10 11 12 13 4 5 6 7 8 9 10 8 9 10 11 12 13 14
14 15 16 17 18 19 20 11 12 13 14 15 16 17 15 16 17 18 19 20 21
21 22 23 24 25 26 27 18 19 20 21 22 23 24 22 23 24 25 26 27 28
28 29 30 31 25 26 27 28 29 30 31 29 30

October November December

Su Mo Tu We Th Fr Sa Su Mo Tu We Th Fr Sa Su Mo Tu We Th Fr Sa
1 2 3 4 5 1 2 1 2 3 4 5 6 7
6 7 8 9 10 11 12 3 4 5 6 7 8 9 8 9 10 11 12 13 14
13 14 15 16 17 18 19 10 11 12 13 14 15 16 15 16 17 18 19 20 21
20 21 22 23 24 25 26 17 18 19 20 21 22 23 22 23 24 25 26 27 28
27 28 29 30 31 24 25 26 27 28 29 30 29 30 31
9.1.3 Display an Alternative Year

If you pass cal a single integer parameter it will print out a calendar
for that year.

$ cal 2015

Listing 9.3
2015

January February March

Su Mo Tu We Th Fr Sa Su Mo Tu We Th Fr Sa Su Mo Tu We Th Fr Sa
1 2 3 1 2 3 4 5 6 7 1 2 3 4 5 6 7
4 5 6 7 8 9 10 8 9 10 11 12 13 14 8 9 10 11 12 13 14
11 12 13 14 15 16 17 15 16 17 18 19 20 21 15 16 17 18 19 20 21
18 19 20 21 22 23 24 22 23 24 25 26 27 28 22 23 24 25 26 27 28
25 26 27 28 29 30 31 29 30 31

April May June

Su Mo Tu We Th Fr Sa Su Mo Tu We Th Fr Sa Su Mo Tu We Th Fr Sa
1 2 3 4 1 2 1 2 3 4 5 6
5 6 7 8 9 10 11 3 4 5 6 7 8 9 7 8 9 10 11 12 13
12 13 14 15 16 17 18 10 11 12 13 14 15 16 14 15 16 17 18 19 20
19 20 21 22 23 24 25 17 18 19 20 21 22 23 21 22 23 24 25 26 27
26 27 28 29 30 24 25 26 27 28 29 30 28 29 30
31
July August September
Su Mo Tu We Th Fr Sa Su Mo Tu We Th Fr Sa Su Mo Tu We Th Fr Sa
1 2 3 4 1 1 2 3 4 5
5 6 7 8 9 10 11 2 3 4 5 6 7 8 6 7 8 9 10 11 12
12 13 14 15 16 17 18 9 10 11 12 13 14 15 13 14 15 16 17 18 19
19 20 21 22 23 24 25 16 17 18 19 20 21 22 20 21 22 23 24 25 26
26 27 28 29 30 31 23 24 25 26 27 28 29 27 28 29 30
30 31
October November December
Su Mo Tu We Th Fr Sa Su Mo Tu We Th Fr Sa Su Mo Tu We Th Fr Sa
1 2 3 1 2 3 4 5 6 7 1 2 3 4 5
4 5 6 7 8 9 10 8 9 10 11 12 13 14 6 7 8 9 10 11 12
11 12 13 14 15 16 17 15 16 17 18 19 20 21 13 14 15 16 17 18 19
18 19 20 21 22 23 24 22 23 24 25 26 27 28 20 21 22 23 24 25 26
25 26 27 28 29 30 31 29 30 27 28 29 30 31

Listing 9.3 shows a calendar for the year 2015.

9.1.4 Display an Alternate Month/Year

When passed two integer parameters, cal will interpret that as

month year and will generate a calendar accordingly.

$ cal 8 2016

Listing 9.4
August 2016
Su Mo Tu We Th Fr Sa
1 2 3 4 5 6
7 8 9 10 11 12 13
14 15 16 17 18 19 20
21 22 23 24 25 26 27
28 29 30 31

Listing 9.4 shows the calendar for August, 20161 .

9.2 Cat
The cat command is short for concatenate, and it does just that. It
will allow us to concatenate multiple files together, but it also has a
few extra tricks up its sleeve.

9.2.1 Read a File

Without any extra flags, cat will print out the contents of a file to
the standard output, usually your terminal window.

$ cat file1.txt
Listing 9.5
Lorem ipsum dolor sit amet, consectetur adipiscing elit. In ut felis sit amet
mi malesuada convallis. In eget rutrum lacus. Donec consequat, orci sit amet
porttitor hendrerit, risus mauris placerat erat, eget vestibulum diam dolor eu
erat. Donec a neque sit amet eros porta volutpat. Vestibulum lectus magna,
ultrices at rutrum euismod, scelerisque ac urna. Suspendisse augue ipsum,
condimentum at fringilla ut, dapibus eu nisi. Nam pellentesque ante convallis,
sagittis nunc a, vehicula turpis. Sed quis velit pulvinar, convallis magna in,
viverra sapien.

Listing 9.5 shows the contents of the file1.txt file.

9.2.2 Concatenate Multiple Files

If we supply multiple files to the cat command it will combine them

all together into a single output - in this case, to the standard
output.

$ cat file1.txt file2.txt

Listing 9.6
Lorem ipsum dolor sit amet, consectetur adipiscing elit. In ut felis sit amet
mi malesuada convallis. In eget rutrum lacus. Donec consequat, orci sit amet
porttitor hendrerit, risus mauris placerat erat, eget vestibulum diam dolor eu
erat. Donec a neque sit amet eros porta volutpat. Vestibulum lectus magna,
ultrices at rutrum euismod, scelerisque ac urna. Suspendisse augue ipsum,
condimentum at fringilla ut, dapibus eu nisi. Nam pellentesque ante convallis,
sagittis nunc a, vehicula turpis. Sed quis velit pulvinar, convallis magna in,
viverra sapien.
Integer suscipit augue a tempor pellentesque. Proin tincidunt magna in luctus
vulputate. Duis euismod elit sit amet pellentesque hendrerit. Suspendisse blandit
euismod mauris eu congue. Ut condimentum et est sed lobortis. Etiam vel neque
faucibus turpis accumsan imperdiet. Nullam nisl felis, pharetra quis orci vitae,
sagittis gravida augue. Interdum et malesuada fames ac ante ipsum primis in
faucibus. Maecenas vulputate tempus adipiscing. Cras aliquet nibh a ligula
varius, a varius massa tempus. Etiam porttitor neque sit amet ante laoreet, vitae
euismod ipsum ullamcorper. Nullam placerat sed est in tincidunt.

Listing 9.6 shows the two files joined together into one file and
printed to the standard output (terminal window).
9.2.3 Save Multiple Files to a New File

In the previous examples, the output of the cat command went to

standard output, but we can also use the redirect operator, >, to send
the output to a file.

Here is an example of how we can use the cat command to combine

2 files into a new file by using the > operator and giving it the name
of the file we want to create.

$ cat file1.txt file2.txt > new_file.txt

We can then use the cat command to view the contents of the new
file.

$ cat new_file.txt

Listing 9.7: $ cat new_file.txt

Lorem ipsum dolor sit amet, consectetur adipiscing elit. In ut felis sit amet
mi malesuada convallis. In eget rutrum lacus. Donec consequat, orci sit amet
porttitor hendrerit, risus mauris placerat erat, eget vestibulum diam dolor eu
erat. Donec a neque sit amet eros porta volutpat. Vestibulum lectus magna,
ultrices at rutrum euismod, scelerisque ac urna. Suspendisse augue ipsum,
condimentum at fringilla ut, dapibus eu nisi. Nam pellentesque ante convallis,
sagittis nunc a, vehicula turpis. Sed quis velit pulvinar, convallis magna in,
viverra sapien.
Integer suscipit augue a tempor pellentesque. Proin tincidunt magna in luctus
vulputate. Duis euismod elit sit amet pellentesque hendrerit. Suspendisse blandit
euismod mauris eu congue. Ut condimentum et est sed lobortis. Etiam vel neque
faucibus turpis accumsan imperdiet. Nullam nisl felis, pharetra quis orci vitae,
sagittis gravida augue. Interdum et malesuada fames ac ante ipsum primis in
faucibus. Maecenas vulputate tempus adipiscing. Cras aliquet nibh a ligula
varius, a varius massa tempus. Etiam porttitor neque sit amet ante laoreet, vitae
euismod ipsum ullamcorper. Nullam placerat sed est in tincidunt.

Listing 9.7 shows the newly created, new_file.txt, that is the

combination of both file1.txt and file2.txt.
9.2.4 Display Line Numbers

The -b flag will print out line numbers.

$ cat -b big_file.txt

Listing 9.8
1 Lorem ipsum dolor sit amet, consectetur adipiscing elit. In ut
felis sit amet mi malesuada convallis. In eget rutrum lacus. Donec consequat,
orci sit amet porttitor hendrerit, risus mauris placerat erat, eget vestibulum
diam dolor eu erat. Donec a neque sit amet eros porta volutpat. Vestibulum
lectus magna, ultrices at rutrum euismod, scelerisque ac urna. Suspendisse
augue ipsum, condimentum at fringilla ut, dapibus eu nisi. Nam pellentesque
ante convallis, sagittis nunc a, vehicula turpis. Sed quis velit pulvinar,
convallis magna in, viverra sapien.
2 Integer suscipit augue a tempor pellentesque. Proin tincidunt magna
in luctus vulputate. Duis euismod elit sit amet pellentesque hendrerit.
Suspendisse blandit euismod mauris eu congue. Ut condimentum et est sed
lobortis. Etiam vel neque faucibus turpis accumsan imperdiet. Nullam nisl
felis, pharetra quis orci vitae, sagittis gravida augue. Interdum et
malesuada fames ac ante ipsum primis in faucibus. Maecenas vulputate tempus
adipiscing. Cras aliquet nibh a ligula varius, a varius massa tempus. Etiam
porttitor neque sit amet ante laoreet, vitae euismod ipsum ullamcorper.
Nullam placerat sed est in tincidunt.
3 Nunc vel massa vel nibh auctor aliquam. Fusce non interdum purus.
Proin at vestibulum nisl. Mauris venenatis rutrum cursus. Nunc quis rutrum
nulla. Praesent ac faucibus ligula. Donec non velit quis risus aliquet
sollicitudin. Fusce sed vestibulum est.
4 Sed feugiat lorem eu porta sodales. Ut egestas urna orci, eget
fringilla lectus malesuada ac. Etiam malesuada nulla et luctus ullamcorper.
Fusce et enim nulla. Vivamus pharetra eros non convallis fermentum. Nulla
viverra nunc nec faucibus tempor. Duis nibh ligula, tincidunt non
ullamcorper eget, iaculis vel mi. Duis tristique imperdiet venenatis.
Praesent dictum sit amet elit sit amet aliquam. Proin convallis eros sed
tortor sodales elementum. Lorem ipsum dolor sit amet, consectetur
adipiscing elit.
5 Sed eu dignissim augue. In accumsan enim diam, eu convallis nisl
pellentesque a. Pellentesque sed lorem dui. Praesent mattis ut nulla at
condimentum. Nam ut nisi porttitor, feugiat libero interdum, sollicitudin eros.
Maecenas at metus vehicula, ornare diam vitae, iaculis arcu. Duis eu mi eu
ipsum blandit viverra in nec quam. Nulla auctor, velit vel ornare congue,
libero mi porttitor mauris, nec porta orci lorem lacinia eros. Sed tristique
aliquet congue. Interdum et malesuada fames ac ante ipsum primis in faucibus.

9.3 Kill
The kill command is, unfortunately, poorly named. Its name
would imply that its sole purpose is that of mischief, when in reality
its purpose is to simply send messages to processes. A few of these
signals will forcefully terminate a process. Others will simply tell
the process to shutdown when it is ready.

Table 9.2 shows a list of the different signals (and their meaning)
that can be sent to a process from the kill command.

Signal Value Action Comment

Hangup detected on controlling

SIGHUP 1 Term terminal or death of controlling
process

SIGINT 2 Term Interrupt from keyboard

SIGQUIT 3 Core Quit from keyboard

SIGILL 4 Core Illegal Instruction

SIGABRT 6 Core Abort signal from abort(3)

SIGFPE 8 Core Floating point exception

SIGKILL 9 Term Kill signal

SIGSEGV 11 Core Invalid memory reference

Broken pipe: write to pipe with

SIGPIPE 13 Term
no readers

SIGALRM 14 Term Timer signal from alarm(2)

SIGTERM 15 Term Termination signal

 SIGUSR1 30,10,16 Term User-defined signal 1

SIGUSR2 31,12,17 Term User-defined signal 2

SIGCHLD 20,17,18 Ign Child stopped or terminated

SIGCONT 19,18,25 Cont Continue if stopped

SIGSTOP 17,19,23 Stop Stop process

SIGTSTP 18,20,24 Stop Stop typed at tty

tty input for background

SIGTTIN 21,21,26 Stop
process

tty output for background

SIGTTOU 22,22,27 Stop
process

Table 9.2: Signal Definitions for kill

Using ps that we learned about in Chapter 6 we can find the PID, or

process ID, of the process that we need to stop.

The default signal for kill is 15, or SIGTERM.

Listing 9.9
$ kill 123456

In Listing 9.9 we are sending a signal to process with a PID of

123456 to peacefully shutdown when it is ready.

9.3.1 Forcefully Terminate a Process

One of the most common use cases for kill is to forcibly terminate
a process. There are two ways to do this, both ways are aliases of
each other.

Listing 9.10
$ kill -9 123456

Listing 9.11
$ kill -s SIGKILL 123456

In Listing 9.10 we pass in the value of the signal we want to send to

the process, -9.

In Listing 9.11 we use the -s flag and give it the name of the signal
we want to send, SIGKILL.

Both Listing 9.10 and Listing 9.11 send the same information, and
have the result, on the specified process.

9.4 Man
The goal of this book is to help you master the Unix and Linux
commands that will provide you with the most value on a daily
basis.

We’ve covered a key subset of flags and options that will be most
useful in your day-to-day work as a software developer. In some
circumstances you’ll need to go beyond the scope of this book for
more information.

Your first resource to find out more information about a particular

command should be the man command. The man command will print
out the manual for the command that you specify, including
information on all flags and options.

$ man mkdir

Listing 9.12
MKDIR(1) BSD General Commands Manual MKDIR(1)

NAME
mkdir -- make directories

SYNOPSIS
mkdir [-pv] [-m mode] directory_name ...

DESCRIPTION
The mkdir utility creates the directories named as operands, in the order
specified, using mode rwxrwxrwx (0777) as modified by the current umask(2).

The options are as follows:

-m mode
Set the file permission bits of the final created directory to the
specified mode. The mode argument can be in any of the formats
specified to the chmod(1) command. If a symbolic mode is specified,
the operation characters ``+'' and ``-'' are interpreted relative to an
initial mode of ``a=rwx''.

-p Create intermediate directories as required. If this option is

not specified, the full path prefix of each operand must already exist.
On the other hand, with this option specified, no error will be
reported if a directory given as an operand already exists.
Intermediate directories are created with permission bits of
rwxrwxrwx (0777) as modified by the current umask, plus write and
search permission for the owner.

-v Be verbose when creating directories, listing them as they are

created.

The user must have write permission in the parent directory.

DIAGNOSTICS
The mkdir utility exits 0 on success, and >0 if an error occurs.

SEE ALSO
 rmdir(1)

COMPATIBILITY
The -v option is non-standard and its use in scripts is not recommended.

STANDARDS
The mkdir utility is expected to be IEEE Std 1003.2 (``POSIX.2'')
compatible.

HISTORY
A mkdir command appeared in Version 1 AT&T UNIX.

BSD January 25, 1994 BSD

In Listing 9.12 we see the manual for the mkdir command.

The man command does a lot more than just display manual
information. For example, it helps you to generate man pages, but
that’s unlikely something you’re going to use that often.

For more information on man you can use the very “meta” command
found in Listing 9.13.

Listing 9.13
$ man man

9.5 Pbcopy (Mac OS X Only)

The pbcopy command is only available on OS X systems. On
systems other than OS X, this functionality can be recreated using
an X11 tool called xsel, which is available on most Linux systems.

So what does pbcopy do? It will read input from the standard output
and copy that data into the system’s paste buffer, to be used in other
places.
We can use the cat command from Section 9.2 to read in a file and
then use the pipe operator, |, to send the result to pbcopy.

$ cat file1.txt | pbcopy

Listing 9.14: Data in the Paste Buffer

Lorem ipsum dolor sit amet, consectetur adipiscing elit. In ut felis sit amet
mi malesuada convallis. In eget rutrum lacus. Donec consequat, orci sit amet
porttitor hendrerit, risus mauris placerat erat, eget vestibulum diam dolor eu
erat. Donec a neque sit amet eros porta volutpat. Vestibulum lectus magna,
ultrices at rutrum euismod, scelerisque ac urna. Suspendisse augue ipsum,
condimentum at fringilla ut, dapibus eu nisi. Nam pellentesque ante convallis,
sagittis nunc a, vehicula turpis. Sed quis velit pulvinar, convallis magna in,
viverra sapien.

Listing 9.14 shows the data in the paste buffer ready to be pasted
into what application you need it for.

9.6 Pbpaste (Mac OS X Only)

In Section 9.5 we learned how to use the pbcopy command to copy
data into the system’s paste buffer. It would make sense that if you
could get data into the system’s paste buffer, there would be an easy
way to get that data back out again.

The pbpaste command lets you take data from the system’s paste
buffer and write it to standard out.

Let’s copy a file into the paste buffer using the pbcopy command.
We’ll use the example file from the sed chapter, Chapter 7.

$ pbcopy < birthday.txt

With data in the paste buffer we can use the pbpaste command to
pipe that data into another command.

$ pbpaste | ag name

Listing 9.15
Happy Birthday Dear (name)

In Listing 9.15 we took the data from the system’s paste buffer and
piped it into the ag command (Chapter 2) to search for the word
“name”.

9.7 Tail
The tail command allows us to look at the end of a file, and even
more so will let us watch a file as it is being written to, making it
perfect for monitoring log files in a production environment.

9.7.1 Display Last n Lines of a File

By default, when you pass a file name to the tail command the last
ten lines of the file will be printed to the screen, and tail will exit.
Listing 9.16 shows the last ten lines of the /var/log/system.log file.

$ tail /var/log/system.log

Listing 9.16
Dec 5 15:46:22 markmini.home Google Chrome Helper[62354]: Process unable to ...
Dec 5 15:46:22 markmini.home Google Chrome Helper[62354]: Process unable to ...
 Dec 5 15:46:22 markmini.home Google Chrome Helper[62354]: CGSLookupServerRoo...
Dec 5 15:46:24 markmini.home Google Chrome Helper[62357]: Internals of CFAll...
Dec 5 15:46:24 markmini.home Google Chrome Helper[62357]: Process unable to ...
Dec 5 15:46:24 markmini.home Google Chrome Helper[62357]: Process unable to ...
Dec 5 15:46:24 markmini.home Google Chrome Helper[62357]: CGSLookupServerRoo...
Dec 5 15:49:19 markmini.home mds[100]: (Warning) Volume: vsd:0x7fcd9c5a0800 ...
DisabledRecycleCount = 8;
}

If we want to change the number of lines that are displayed by tail

we can use the -n flag.

$ tail -n 5 /var/log/system.log

Listing 9.17
Dec 5 15:46:24 markmini.home Google Chrome Helper[62357]: Process unable to ...
Dec 5 15:46:24 markmini.home Google Chrome Helper[62357]: CGSLookupServerRoo...
Dec 5 15:49:19 markmini.home mds[100]: (Warning) Volume: vsd:0x7fcd9c5a0800 ...
DisabledRecycleCount = 8;
}

Listing 9.17 shows the last five lines in the /var/log/system.log

file.

9.7.2 Reverse Output of a File

Using the -r flag we can reverse the direction of how tail displays
the selected lines of the file to us. By default, the last line of the
output is the last line of the file, but the -r flag will reverse that to
make the last line of the file the first line of the output.

$ tail -r -n 5 /var/log/system.log

Listing 9.18
}
DisabledRecycleCount = 8;
Dec 5 15:49:19 markmini.home mds[100]: (Warning) Volume: vsd:0x7fcd9c5a0800 ...
 Dec 5 15:46:24 markmini.home Google Chrome Helper[62357]: CGSLookupServerRoo...
Dec 5 15:46:24 markmini.home Google Chrome Helper[62357]: Process unable to ...

Listing 9.18 shows the last five lines of a file with the newest line at
the top of the output, and the oldest line at the bottom of the
output.

9.7.3 Watch a “Live” File

Quite possibly one of the most used commands and flags in any
production environment is the tail command with the -f flag.
When using the -f flag, tail will never exit and will continue to
output new data as it is added to the end of the watched file.

$ tail -f /var/log/system.log

As new lines are being written to /var/log/system.log they will

automatically be displayed, without the need to refresh or run the
command again.

9.7.4 Search “Live” Output

When watching a “live” file, often referred to as “tailing” a file, we

may want to filter the output so we can find exactly the lines of data
that are the most important to us. An example of this would be
wanting only to see lines from a web server log file that come from a
certain IP address.

We can easily use the pipe operator, |, to send the live stream of
data from tail to a command (such as ag that we learned about in
Chapter 2) to filter the data easily for us.

$ tail -f /var/log/system.log | ag com.apple.windowserver

Listing 9.19
Dec 5 15:46:22 markmini.home Google Chrome Helper[62354]:
CGSLookupServerRootPort: Failed to look up the port for
"com.apple.windowserver.active" (1100)
Dec 5 15:46:24 markmini.home Google Chrome Helper[62357]:
CGSLookupServerRootPort: Failed to look up the port for
"com.apple.windowserver.active" (1100)

When combining tail -f with ag we can watch as the exact data we

want from a “live” file is continuously printed to our screen without
needing to continually refresh. This combination is a must for
production (and even development) environments.

9.8 Tree
Using the tree command, you can recursively list the contents of a
directory in a tree-like format. This is handy when you want to get a
high-level view of all the directories and files inside of a project for
example.

By default, tree will list all of the directories and files in a given
directory, but it also comes with many flags that will alter the level
of detail you get back.

9.8.1 Installation
tree may or may not be installed by default on your machine. If
tree is not installed then you should visit the tree homepage for
instructions on how to install it on your system.

If you are on a Mac, the best and easiest way to install tree is using
Homebrew.

$ brew install tree

If you are on a different platform, please see the install directions

on the project’s page for the correct installation instructions for
your platform.

9.8.2 Listing a Directory’s Contents

By default when you use the tree command, it will output a list of
all of the files and directories in the current directory, recursively.

$ tree

Listing 9.20
.
├── assets
│ ├── images
│ │ └── image.jpg
│ ├── javascripts
│ │ ├── application.js
│ │ └── curly.js.coffee
│ └── stylesheets
│ ├── application.css
│ └── curly.css.scss
├── controllers
│ ├── application_controller.rb
│ ├── concerns
│ └── curly_controller.rb
├── helpers
│ ├── application_helper.rb
│ └── curly_helper.rb
├── mailers
 ├── models
│ └── concerns
└── views
├── curly
│ └── echo.html.erb
└── layouts
└── application.html.erb

In Listing 9.20 we can see a recursive list of all files and directories
in the current directory.

9.8.3 Limiting the Depth of Listing (-L)

The tree command will let us limit the depth of its recursion if we
want using the -L flag and giving it a number of levels we want the
recursion to go. This can be helpful in very deep directories, where
you only want an overview of what’s under the surface.

$ tree -L 2

Listing 9.21
.
├── assets
│ ├── images
│ ├── javascripts
│ └── stylesheets
├── controllers
│ ├── application_controller.rb
│ ├── concerns
│ └── curly_controller.rb
├── helpers
│ ├── application_helper.rb
│ └── curly_helper.rb
├── mailers
├── models
│ └── concerns
└── views
├── curly
└── layouts

In Listing 9.21 we told tree to only recurse down two levels; files
and directories that are deeper, such as those under the views/curly
directory, are not listed.

9.8.4 Listing Directories Only (-d)

By default tree will list both files and directories. Often it is useful
just to see a list of directories without the files cluttering up the
tree. We can use the -d flag to print out just the directories.

$ tree -d

Listing 9.22
.
├── assets
│ ├── images
│ ├── javascripts
│ └── stylesheets
├── controllers
│ └── concerns
├── helpers
├── mailers
├── models
│ └── concerns
└── views
├── curly
└── layouts

Using the -d flag in Listing 9.22 we see only a list of directories

under the current directory.

9.8.5 Listing Another Directory

So far we have been using tree to print out a listing of the current
directory, but like the ls command (Chapter 1 Section 1.3), we can
also give it a path to another directory.

$ tree -d /usr/local/etc/
Listing 9.23
/usr/local/etc/
├── ImageMagick
├── bash_completion.d
│ └── helpers
├── fonts
│ └── conf.d
├── openssl
│ └── misc
├── profile.d
└── rabbitmq

As we can see in Listing 9.23 we have printed out a tree listing of all
of the directories under my /usr/local/etc/ directory.

9.9 Wc
The wc command provides two really useful functions, it will both
count the number of words in a file, as well as the number of lines
in a file.

9.9.1 Count Words in a File (-w)

If we want to count the number of words in a file we can use the -w

flag.

$ wc -w /usr/share/dict/web2a

Listing 9.24
121847 /usr/share/dict/web2a
As we can see in Listing 9.24 the /usr/share/dict/web2a file
contains 121847 words.

9.9.2 Count Lines in a File (-l)

To count the number of lines in a file we can use the -l flag.

$ wc -l /usr/share/dict/web2a

Listing 9.25
76205 /usr/share/dict/web2a

Listing 9.25 shows that our file contains 76205 lines in it.

9.9.3 Piping Input to WC

As we’ve seen throughout the examples in this book, we, as

developers, often need to chain Unix commands together to
accomplish certain tasks. We can use the pipe operator, |, to supply
the input to the WC command to count the number of occurrences
of a word or line in the input.

In the section on Ag/Ack 2.2, we saw how we could search for the
occurrences of “DHH” in the Rails source code. But what if we just
wanted to count the occurrences of the phrase “DHH” in the code?
Simple, we can accomplish that by piping the ag output into wc with
the pipe operator, |.

$ ag DHH | wc -l
The result of our search should look something like Listing 9.26:

Listing 9.26
26

1. Mark your calendars for August 24th, 2016, and make sure to send me a 40th
birthday greeting. :) ↑