UNIX Find Mini-HOW-TO

Any UNIX system is a time-sharing systems which concurrently runs processes and attends to multiple users. What UNIX aims at with such skills are files ([Ritchie84]). An urban legend says that after MULTICS as abandoned "they needed to rewrite an operating system (OS) in order to play space war on another smaller machine, a DEC PDP-7". In an interview Ken Thompson explained he actually implemented UNIX as test environment for an hierarchical file-system he had designed earlier ([Seibel09]). What is true? In fact many resources under UNIX are mapped into the file system: devices such as 'tty’s, and concepts such as FIFOs and processes.

Secondly, since then it began to escape from AT&T’s Bell Laboratories in the early 1970’s UNIX follows a "component philosophy"- In UNIX programs enhance the functionality of programs. Under the UNIX shell programs are like commands in a high-level language to handle files and file contents. Where other languages are build upon scalars and few keywords, UNIX shell programs are build upon programs and few keywords.

But the going is tough. Most UNIX tools are not good in finding files, with none or simple abilities for directory traversal. Only a small set of UNIX programs actively find files: * ls, grep, chmod, chgrp, rsync, rm and cp, for example, provide -r and -R options; * tar, find and du legally process sub-directories.

UNIX programs with -r are problematic: usually they * find all file types, * run into infinite loops caused by hard- and soft-links, and * descend into mounted file-systems.

The reason why UNIX programs treat the file-system lightly are the efficient shell quoting and globbing rules. The first UNIX-shell was the Thompson Shell (1971), and it already expanded the characters * ? [ ] specially
[The Thompson Shell was introduced in the first version of UNIX 1971, and was written by Ken Thompson. It was a simple command interpreter, not designed for scripting, but it introduced pipes, quoting and glob-patterns.]
. Words that use these characters are called wildcards, glob-patterns or globs
[From the glob and fnmatch system functions.]
. To get some C source files, for example, we can use

Multiple * characters may occur in the same word, and the shell will know where to expand directory names, and where to expand filenames.

This is a deep concept. Without such a higher level of interpretation each program would believe in its worm’s-eye view. The UNIX philosophy ("programs enhance programs") wouldn’t work
[Users of MSDOS command.com could write a book about it. But UNIX missed something too: coherent command-line-parsing. Arguments are not separated into filename and option-name arguments. So some programs accept long option-names, some let short and long names both begin with "-", or use "+"; some won’t accept "--" as the end of the argument list, and some allow "-" as filename (which is a bad idea, because it names stdin).]
. But wildcards are not a general method to find files. The above command shows a few problems:

find solves all these problems. The two major standards are POSIX find and GNU find, part of the GNU findutils. The latter has useful additional capabilities, such as -regex, -delete and -print0.

The find program has become is the tool of choice to examine the file system and file contents. For example

is equivalent to the basic

but find can select files more subtly than that, by searching the file’s

The find utility recursively descends the directory hierarchy of each search-root, evaluating the Boolean expression search-expression.

find only has three real command-line options: -H, -L and -P. All three treat symbolic links. * The default behavior is to never follow symbolic links (-P). * The -L flag will cause find to follow symbolic links. * The -H flag will only follow symbolic links while processing the command line arguments. These options are mutually exclusive, but using them together is not considered an error. The last option specified determines the behavior.

By default, when no search-roots are defined, find returns a list of all files below the current working directory.

The first argument with a -, ! or ( as the first character after search-root is interpreted as start of search-expression. The search expression contains elements of a mini-language.

Elements are interpreted by find and must circumvent shell quoting rules for the characters ( ) ;. find will handle wildcards internally. Arguments to -name, -path, -wholename and -regex use wildcard characters must also be quoted. See pattern arguments for more information.

Elements are whitespace-separated, i.e. pass \( name foo \) not \(-name foo\) or '( -name foo )'.

Each element in the search-expression yields true or false. The whole expression is evaluated from left to right. Standard Boolean logic applies, and so not all elements might be evaluated.

The exit code is 0 for ok or >0 when an error occurred. Note that the exit code is 0 too when no files where found.

Several expressions require pattern arguments (globs): -name, -path, -regex and -wholename. These arguments are expanded internally and must be protected from the shell.

find expands patterns differently as does the shell:

For example, -path "foo\*bar" matches files named ./foobar, ./foo/.bar and ./foo/foo/bar.

The same pattern, when passed as -name "foo\*bar" is only applied to the filename (basename); it matches ./foobar and ./bluh/foo.bar, but not ./foo/.bar.

When the search-expression contains no actions (-exec, -execdir, -ok or -print) then -print is appended:

Therefore:

The above example finds both files, and is not equivalent to

For brevity each directory-entry is called a file here. All search-roots are processed sequentially from left to right. find processes all files in the root directory, then in sub-directories.

The complete directories on the disk are always scanned, and every file is fed into the search-expression. The search-expression consists of tests and actions (see the command-line). Tests return true or false, depending on the file qualities. Actions return true on principle. However:

find can descend into directories of arbitrary depths and will handle any valid path length.

In reality file-systems often contain loops created through the use of hard- or symbolic links. The POSIX standard therefore requires that: "The find utility shall detect infinite loops; that is, entering a previously visited directory that is an ancestor of the last file encountered. When it detects an infinite loop, find shall write a diagnostic message to standard error and shall either recover its position in the hierarchy or terminate." (POSIX find).

In Boolean logic, expressions are evaluated from left to right. When more than one logical operator is used in a statement, NOT is evaluated first, then AND, and finally OR. Arithmetic (and bit-wise) operators are handled before logical operators. find provides only logical operators.

A simple shell function example:

In the following example the file program is run on all files of the working directory, but not files under ./tmp. It prunes tmp:

The use of parentheses, even when not required, can improve the readability of expressions and reduce the chance of making a subtle mistake because of operator precedence. There is no performance penalty in using parentheses. This example is more readable than the previous one, although they are logically the same:

Next we additionally prune all .svn sub-directories:

To change the meaning of expressions, add parentheses to force evaluation of OR parts. This example uses parentheses, but is otherwise similar to the previous one:

Parentheses make the expression more readable. Apparently the example can be optimized:

As evidenced by:

Note that -and, -or and -not are GNU extensions. POSIX-compliant are -a, -o and !.

The simplest example is

which prints every directory entry of the directory and all sub-directories. Normally only specific files or directories are of interest. The following (dangerous!) example removes all files (not directories) of a user:

To prevent find from searching certain directories use -prune. Here is an example found on the find man-page: to skip the directory src/emacs and all files under it, and print the names of the other files found, do something like this:

Here is a typical example you’ll often find on the Internet:

Since -prune is always true it does not block the commands right to it. Additionally -prune has the side-effect that find considers the directory it currently iterates empty. This happens to be the directory that matches spec.

Note that -prune is not always true. It is false in conjunction with -depth and -delete (GNU find). The -depth action lets find ascend from the bottom directories, so -prune will be senseless.

-depth is stronger than -prune.

find by default descends into all sub-directories. -prune avoids certain directories; -depth forces a bottom-up search. But how to process only files in the current directory? A portable solution is

Prunes all directories except the search-root. The grouped expression is false for all files and for the directories not named like the search-root. All other directories are pruned, in which case the grouped expression is true. A more compact form of the above is

which is executed on all items expect the search-root. -print shall be a placeholder for any further search-expression here. The example prunes every file, but find will ignore the faux-pas. The only exception is the search-root, which is not pruned. So the contents of . are printed, but not . itself. To print . plus all files in it use the form
[Pointed out by Stéphane Chazelas on 3/10/2009 in comp.unix.shell]

Here the grouped expression is always true and -print is called on all items. But all directories except . are pruned; only . and items in . are found. Again, find gracefully ignores the attempt to prune non-directories.

Exactly the opposite behavior is triggered by omitting the -name part. For example

finds only the search-root, as evidenced by

This can be useful to produce some output in the proper find format.

GNU find introduced new expressions to control descending

Use -maxdepth 1 to let find find each search-root plus files in it, but don’t descend deeper. To not include the search-root additionally specify -mindepth 1.

Recurse the search-root and stop after printing the first file found. Useful to quickly test whether a directory somewhere contains a certain file or file type. If so don’t waste time to find additional files.

find has no option to do this.

The solution is find|head -1 to get the first line of find-output (original solution was posted here). As explained in How UNIX processes pipes below each program in the pipe is forked by the same sub-shell. head exits when the second line occurs. Then find receives a SIGPIPE signal to indicate that the pipe to which it writes is broken. The reaction of find is to terminate itself.

The solution is unsatisfying because find receives the signal only when it prints the second file found. So find will search a second file, and if there’s no second file it will search the whole directory. A better solution is find|sed 1q. sed reads the line (first pathname), and exits as response to q. At the same time find will receive SIGPIPE and exits too.

The following example makes use of find|sed 1q by searching a directory of RAW files (digital negatives of photos).

Again, find has no option to do this. We have to force a SIGPIPE signal to terminate find after enough files have been found. See example 3.

The general question here is to print the nth to the mth file from standard out.

Print the 2nd up to the 10th file:

Additonally print the 17th to the 20th line:

The following command prunes all directories named name. Use this form to execute a command only on directories named name, but on none of its possible entries.

find will combine all expressions with -and. Because of the left to right evaluation order commands occur last. When the current path names a directory, and the name matches name or spec, then find also evaluates commands. -prune is true, but has the side-effect of not descending into the directory on which commands are undertaken.

Removing sub-directories is a special case where -prune is necessary. Consider:

The command removes all sub-directories of the current working directory unless their name begins with a dot. But during this operation most versions of find will complain:

The message occurs after find executed rm on directory-name. The message reveals that find has seen the directories before it evaluated -exec. I leave it open whether the above message makes any sense. The only way to prevent it is -prune:

Alias:

A more complex example, which illustrates that find can make extra shell scripts unnecessary, and is also useful in makefile rules.

Let’s say in a software project we have a sub-folder libs holding internal and third-party libraries. The third-party libraries are located under libs/third-party. Now we want to clean libs from compiler-intermediate files, unnecessary files, empty directories etc.

The constraints:

Although not too simple, the expression implementing this behavior is still compact:

Alias:

Basically this prunes all directories under libs and executes cleanup on them ([TheCleanupTool]). Directories not pruned, and hence not cleaned, are named by the -path expressions. We prune directories because the cleanup scripts descends the directory that was passed to it.

Note that find does not add a superfluous ./ at the begin nor / to the end of directory path-names. In the above example, where libs is passed as the search-root, it finds libs/third-party and not ./libs/third-party or libs/third-party/.

find will always put the search-root in front of found path-names.

The basic command to delete any empty directory under from-dir is ([DelEmptyDir]):

When deleting empty directories we must ascend from the bottom directories, or empty directories are not considered empty because they contain empty directories. Hence -depth is mandatory.

There are pitfalls! With GNU find the + forms of -exec may disable -depth. The reason is that empty directories are found before the execute expression is evaluated. Consequently, a directory containing nothing but empty sub-director(ies) is not considered empty:

find removed the empty directories ./test/a/b and ./test/c/d, but we have to run find three times to delete ./test/a and ./test/c and ./test. With ; the command works as expected:

To omit directory names that begin with a dot simply negate the -path.

This operation is implemented in my shell script to remove unneeded files ([TheCleanupTool]). Note that ; must be escaped to prevent it from expansion by the shell. In shell scripts, however, use

This command from the GNU find manual is likely to do something unintended:

Because -delete turns on -depth, the -prune action has no effect and files in dirname/foo will be deleted too.

From the perspective of a shell a name either names an alias, keyword, function, builtin, or file. The type command prints how the shell would interpret a name if used as command-name.

Note that the semicolon must be placed outside of the quotes. Because type is not working for directory names the -type f expression is necessary.

Goal: to execute dos2unix on all text files (notes, source codes, shell scripts, HTML, makefiles etc.).

The UNIX file utility determines the file type by peeking into the file data. The type printed will contain one of the word "text" when the file contains only printing characters and a few common control characters and is probably safe to read on an ASCII terminal.

grep -w force pattern "text" to match as whole word. So far the output is like:

Next cut -f1 -d: splits each line at ":" and prints the first field (the filename).

find is not able to peek into files. To do this the following function combines find and egrep:

Calling find from Shell scripts requires special care, because find accepts file-glob-patterns and grep-patterns as arguments. Most UNIX programs leave filename generation to the shell, but since find expands wildcards internally, arguments for find have to be quoted.

Suppose that the current directory contains the file a.log. Then the find command produces:

But the following shell function will not find ./a.log:

Within a real world script the runfind function possibly has the general form:

With both variants all of the following attempts to call runfind will fail:

The first two will find nothing; the third only a.log in the current working directory, but not as a glob-pattern. When searching big directories trees it is not obvious that find has seen just some of the files! And, if the working directory contains two files, a.log and b.log, the script breaks with a syntactic error. The third runfind call will then be equivalent to:

Most confusingly, when the working directory contain no .log-files the third runfind call will be equivalent to what was actually intended:

It will then — and only then — work as expected. When a file-name generation is not successful, the shell keeps the pattern literally, instead of replacing it with the empty string.

What about:

Now the shell will pass the whole string as supposed option name:

Slowly but surely we’re by ourselves running out of options. The reason are the quoting rules for shell scripts ([ShellQuotingRules]). runfind hadn’t worked because

Quoting occurs in most languages, including the natural ones we speak. Quoting introduces different levels of interpretation. In the shell language quoting is implemented by the characters " ' ` (double/single quote and back tick). To solve the original problem it is necessary to quote all arguments, and force one more round through the shell quoting steps. by using the built-in shell function eval.

Finally this works. When the argument to runfind is itself a variable use:

With Bourne shell it is advisable to additionally disable file-name generation:

Both calls to runfind will yield the same set of .log-files. When file-name generation is disabled the call to eval becomes unnecessary:

The goal is to read all direct sub-directories of $PWD into Bash-array a. Let’s jump into the cold water again.

Why isn’t this working?

The fauxpas here is that the while-loop is executed in a sub-shell. The elements of a will be lost when the pipeline finished, and then ${#a[@]} and i will both be 0. A less important problem is that read does backslash escaping, but under UNIX backslashes are legal path-name characters. Finally read trims leading and trailing newlines and white-spaces, but these are legal path-name characters too.

But this code goes into the right direction. Most importantly, find and read use a zero character \0 as the path-name delimiter, instead of allowing filenames to contain newlines and white-spaces. These abilities are GNU/BSD extensions. $'…' is Bash’s syntax to denote ANSI-C-characters, so $'\0' is just a showy empty string.

Rather than sending find’s output down a pipe we have to put the cart before the horse:

IFS= (or IFS="") disables word-splitting, and -r disables backslash escaping. The default value for IFS is $' \t\n'. Note that IFS is not emptied here to disable word- or line-splitting! IFS= prevents read from trimming leading and trailing newlines and white-spaces in path-names. Incidentally… will IFS= clobber the value of IFS for the entire script? No.

Another Bash-homegrown used here is the process-substitution operator <(…). It forks a new simultaneous process and connects it to a FIFO. <(…) allows you to spawn pipes in each direction (like "multidimensional pipes"). Note, however, that <(…) is just syntax. It is easy to fake the operator in the traditional Bourne shell by calling mkfifo explicitly.

Another form:

For more information see [CaptureFind] and [Wheeler].

The previous example used process-substitution, which is a form of command-substitution. The traditional Bourne shell form of command-substitution uses backquotes (`…`). The new form that permits nesting is $(…). Both forms fork sub-processes, like each external command, whereas a Shell builtin like for or echo is not paralleled. In some cases therefore Shell wildcard expansion might be faster than using find.

To delay the processing of found files use a temporary file. The following line slurps the list of files printed by find into a shell variable:

The downside is that found may contain any number of blanks and control characters from filenames. For filenames UNIX prohibits only the null character, since it’s the C string terminator.

Maybe you’re concerned about find’s linewise output. This is not a problem. In the output of a substituted command the shell silently replaces newlines by single blanks.

Note that found=`<found` is identical to found=`cat found`, but is faster since no sub-shell is forked (see [ShellCmdExecution]).

CPAN provides the File::Find and File::Finder (sic) packages. File::Find is just a layer that introduces a new syntax. In particular the wanted function of File::Find is not practical. File::Finder (by Randal R. Schwartz) is the better choice, since it emulates find syntax in Perl syntax.

However, there is evidence (posted here) that running find in a Shell pipeline is faster, making it unnecessary to learn yet another software-layer’s syntax.

The way in which commands are executed by the shell can be summarized as follows [Ritchie84]:

Every program you run from the shell opens three files: standard input (0), standard output (1), and standard error (2). The files provide the primary means of communications between the programs, and exist for as long as the process runs.

As some defaults, standard input is connect to the terminal keyboard, and standard output and standard error to the terminal display screen.

UNIX allows you to connect processes, by letting the standard output of one process feed into the standard input of another process. That mechanism is called a pipe. Example:

All processes in the pipe are started and their standard inputs and outputs are chained together. In the above example the shell forks find and grep within a sub-shell.

When a program does not support standard I/O, or exits for any other reason, the processes that read from its standard output, or write to its standard input, are sent a SIGPIPE signal.

Note that some programs may complain when receiving SIGPIPE. The ls and find programs will not complain.

In 1963 7-bit ASCII eventually became the successor to the 5-Bit ITA2-code (International Telegraph Alphabet No. 2) used by teletype-printers since the 1930s. In the late 60s ASCII-teletype-printers still were the most common terminal-type. In the new video terminals (nicknamed "glass teletypes") a scrolling window replaced the paper roll. They provided random access to character cells and characters have no longer to be over-typed by a # or @ to erase them.

Until today the tty-functionality of the UNIX kernel canonicalizes input as lines of ASCII-text. The kernel echoes characters to the terminal as they’re typed, but it sends only complete lines from input buffers to programs. The line terminator is a linefeed (ASCII 10). Full-screen editors like vi and tools like tput actually have to tweak the tty: mainly disable cooked mode and stop the tty from echoing characters. Yet the X Window terminal xterm has to provide a pseudo-terminal (pty) to simulate a tty, which in turn mainly pretends to be a massive ASCII-printer.

Shell quoting rules have only marginally changed since the Bourne shell was published with UNIX System VII (1978). An exceptional guide is provided by [Waldmann]. To summarize, before the shell executes a command it performs the following steps:

These steps are repeatedly done for each script and function.

Process-substitution is only available on systems that can afford it.

Some operators are special. Consider

The regular syntax

runs a command once for each file. The command is executed in the directory where find was started. (Use -execdir to run the command in the sub-directory containing the file.) -exec and -execdir are true if the executed command returns zero.

Why the peculiar syntax? Because quoting the whole -exec argument doesn’t scale — the command-line immediately becomes unreadable.

{} is replaced by the full path-name relative to (and including) the search-root. GNU find will replace {} everywhere in a word, not in words equal to {}, as in some versions of find. Note also that some versions of find only replace the first occurrence of {}. GNU find will replace {} in all words.

; is expanded empty.

Generally is necessary to quote ; but not {}. The shell uses { and } to open sub-shells and for functions, but it will not open an empty sub-shell, and makes it hard to declare empty functions:

The shell may even interpret {} as command-name:

When + instead of the ; punctuator is used to terminate -exec the command-line is built by appending each selected file name at the end; the total number of shell invocations will be much less than the number of matched files. This syntax is available only with GNU find. + builds the command-line in much the same way that xargs builds its command-lines. In fact find will use the maximum number of file-names it can pass in a shell command on this system.

The -exec expression is suitable for executing relatively simple commands. For complex commands it is safer to open an additional context. Do this by feeding find's output to xargs. xargs reads file-names from the standard input, delimited by blanks or newlines, and executes the command one or more times (default is /bin/echo). File-names can be protected with double or single quotes or a backslash.

If any invocation of the command exits with a status of 255, xargs will stop immediately without reading any further input.

xargs is not generally applicable. xargs has no placeholder syntax, so it is only useful when the command expects the file-list at the end of the command-line. Otherwise you must use a form of -exec.

Note that the above example is not quite exact. To achieve the same behavior as with -exec {} + we must specify the --no-run-if-empty (alias -r) option to xargs — this is a GNU extension.

A unique feature of GNU xargs is the execution of parallel commands. See the -P switch on the man-page, and my post on the ExifTool forum.

This question raised while writing example 17.

In computing self-producing program is called a Quine. Quines are eventually named after the philosopher and logician Willard van Orman-Quine. A Quine is the sophisticated variant of a "Hello, World!" program: an exercise divorced from reality.

A Quine will reproduce it’s source — perpetuating knowledge about concepts and an exact implementation. A Quine that locates its source-code on disk, and prints it, is considered a cheating Quine. So echo $(<$0) or cat $0 scripts are not Quines. Less apparently, #!/bin/cat is cheating too, without even running a Shell.

find is a tool for tools that do not provide tree traversal, or cannot filter input files by file patterns. These are 99% of all UNIX tools. find's capabilities are superior to simple tree traversal. ls is easier to use on the command-line, but has no query language and does not provide access to all aspects of the file-system.

find regularly outputs all hidden files (dot-files), whereas ls does not print these files unless you specify -a or -A.

The output of find can be used as arguments to ls:

Note also the -ls option of find. Both of the following command-lines are equivalent:

GNU find additionally defines the list operator , (comma). All expressions in the list will always be executed, and the result is that of the last expression. This allows multiple independent operations on one traversal. The operations are not truly independent because one operation could have side-effects like deleting files, that effect other operations in the list.

Likewise, if an operation is -prune, subsequent operations are "pruned".

To prevent error messages pipe stderr to /dev/null.