The awk command explained for server work

What the awk command does

The awk command reads text one line at a time, splits each line into fields, and runs an action on every line that matches a pattern. That makes it the right tool for column-shaped server text, such as access logs, CSV exports, /etc/passwd and the output of ps or df. A whole awk program is often one line: awk '$9 == 404 {print $1}' access.log prints the client address of every request that got a 404.

Every example below works on files you create yourself, so you can check each result against the input by eye. Nothing here reads a live log. A live log changes while you work, so you could not tell a wrong program from new traffic.

Which awk is on this server?

"awk" is a name, not one program. POSIX (the Portable Operating System Interface standard) defines what every awk must do. Several implementations build on that standard, among them GNU awk (gawk) and mawk. They agree on the core language. They differ on extras, so check which one you have before you copy a script from another machine.

readlink -f "$(command -v awk)"

command -v awk prints the path the shell would run. readlink -f follows every symlink to the real file. The $(...) part is bash command substitution: the shell runs the inner command and pastes its output into the outer one. The double quotes keep the path in one piece.

The last part of the path names the implementation, for example a file name ending in mawk or gawk. Do not assume the answer from another server. Different images can ship a different awk, and installing a package can change where awk points.

Then ask that implementation for its version. The flag depends on which one you have:

gawk --version
mawk -W version

Run the one that matches the path readlink printed. gawk prints its version information on standard output. The mawk manual gives -W version as its version flag. On a fresh Ubuntu 24.04 server its first line reads mawk 1.3.4 20240123, followed by copyright lines and a list of compiled limits. As of 3 November 2026, the mawk package in Ubuntu 24.04 is version 1.3.4.20240123-1build1.

Why does it matter which awk you have?

The core language is the same everywhere. The function list is not. The gawk manual describes a set of functions that belong to gawk, among them gensub, strftime, mktime, systime, asort, asorti, patsplit and strtonum, plus bit functions such as and and lshift. The mawk manual says itself that mktime, strftime and systime are gawk extensions, and that mawk provides them only where the C library supports the matching functions. So a script that formats dates with strftime can behave differently on two servers, and the cause is the awk, not your script.

Two habits keep scripts portable. First, stay with the core features this guide teaches unless you know the target runs gawk. Second, when you do need a gawk function, install gawk and call it by name:

sudo apt update && sudo apt install -y gawk
gawk --version

Calling gawk by name means a server without gawk stops with command not found. That is a clear failure. The other choice is a script that quietly runs under a different awk.

One feature moved the other way. POSIX.1-2024 lets length() take an array and return the number of elements in it, and both the gawk and mawk manuals document this. It is no longer a gawk-only trick.

How does the awk command split a line into fields?

Create a small file shaped like a web server access log:

printf '%s\n' \
  '203.0.113.10 - - [03/Nov/2026:10:00:01 +0000] "GET / HTTP/1.1" 200 512' \
  '198.51.100.7 - - [03/Nov/2026:10:00:02 +0000] "GET /about HTTP/1.1" 200 1024' \
  '203.0.113.10 - - [03/Nov/2026:10:00:05 +0000] "GET /missing HTTP/1.1" 404 128' \
  '192.0.2.44 - - [03/Nov/2026:10:00:07 +0000] "POST /login HTTP/1.1" 200 256' \
  '203.0.113.10 - - [03/Nov/2026:10:00:09 +0000] "GET /api HTTP/1.1" 500 64' \
  '198.51.100.7 - - [03/Nov/2026:10:00:12 +0000] "GET /old HTTP/1.1" 404 128' \
  > access.log

awk calls each line a record and each piece of it a field. By default it splits on blanks. The mawk manual defines the default separator as runs of spaces, tabs and newlines, with leading and trailing space trimmed. This file has exactly one space between fields, so the split is not in doubt.

Count along the first line. $1 is the client address 203.0.113.10. $4 is [03/Nov/2026:10:00:01 and $6 is "GET, because awk splits on blanks and does not know what a bracket or a quote means. $7 is the path, $9 is the status code and $10 is the byte count. $0 is the whole record.

awk '{print $1, $9}' access.log

This prints the address and the status of each request. The comma between $1 and $9 makes awk put a space between the two values.

Two built-in variables tell you where you are. NF is the number of fields in the current record. NR is the number of the current record, counted from the start of the input. $NF therefore means "the field whose number is NF", which is always the last field. Here that is the byte count.

awk '{print NR, NF, $NF}' access.log

Every line in this file has 10 fields, so the middle column should read 10 on every line. When a column seems to "move" in a real file, run this check first. A line with a different field count has an extra or missing blank somewhere, and that line is the one that breaks your program.

Patterns and actions: pattern { action }

The gawk manual puts it in one sentence: a rule consists of a pattern followed by an action, and the action goes in braces. awk tests the pattern against each record. When the pattern matches, the action runs. Leave out the pattern and the action runs for every record. Leave out the action and awk prints the matching record.

awk '$9 == 404' access.log
awk '$9 >= 500 {print $1, $7}' access.log
awk '$6 ~ /POST/ {print NR, $1, $7}' access.log

The first line has a pattern and no action, so it prints whole records. The file holds two 404 lines, so you should see two. The second prints the address and the path of every server error. The third uses ~, which means "matches this regular expression", and tests only field 6. Write /POST/ without $6 ~ and the pattern matches the text anywhere in the line.

To bring a value in from outside, use -v. POSIX lists it in the awk synopsis as -v assignment, and it sets an awk variable before your program reads the input:

awk -v code=404 '$9 == code {print $1, $7}' access.log

This is the safe way to pass a shell value into awk. The last section shows the unsafe way, and why it breaks.

BEGIN and END: code before and after the input

BEGIN and END are special patterns. They are never tested against input. POSIX says each BEGIN action runs once before the first record is read. The gawk manual says the END rules run when all the input is exhausted. That makes END the place for totals.

awk '{total += $NF} END {print "requests:", NR, "bytes:", total}' access.log

The main rule adds the last field of every record to total. The END rule prints the result once. Inside END, NR still holds the number of the last record, which is the number of records read. Check it by hand: the file has six lines, and the six byte counts add up to 512 + 1024 + 128 + 256 + 64 + 128, which is 2112.

How do you split on commas or colons with -F?

-F sets the field separator. POSIX says -F sepstring is the same as -v FS=sepstring, so FS is the variable that holds it. /etc/passwd is a good first target. Its manual page says it has one line per account, with seven fields separated by colons. In order they are the login name, an optional encrypted password, the numeric user ID, the numeric group ID, a comment field, the home directory and the optional login shell.

awk -F: '{print $1, $6, $7}' /etc/passwd

This prints each login name with its home directory and shell. Now a CSV file with a header line:

printf '%s\n' \
  'host,mount,used_gb,size_gb' \
  'web1,/,18.5,40' \
  'web1,/data,120,200' \
  'db1,/,22,40' \
  'db1,/srv,310,400' \
  > disks.csv
awk -F, 'NR > 1 {print $1, $2, $3 / $4 * 100}' disks.csv

NR > 1 skips the header, because the header is record 1. The header holds column names rather than numbers, so it does not belong in the arithmetic.

One warning about CSV: -F, splits on every comma. A quoted field such as "Berlin, DE" becomes two fields, and every column after it shifts by one. Use -F, only on CSV files that never quote their fields.

Why use printf instead of print?

print ends each record with the output record separator, which is a newline by default. printf works like the C printf function. It writes exactly what the format string says, so the format must end with \n. Leave the \n out and nothing ends the line.

awk -F, 'BEGIN {printf "%-6s %-6s %7s\n", "host", "mount", "used %"} NR > 1 {printf "%-6s %-6s %6.1f%%\n", $1, $2, $3 / $4 * 100}' disks.csv

%-6s prints a string left-aligned in 6 characters. %6.1f prints a number right-aligned in 6 characters, with one digit after the decimal point. %% prints a literal percent sign. The BEGIN rule prints the header once, before awk reads the file, so the columns line up under it.

How do you count and sum with awk arrays?

awk arrays are associative: the index is a string, not a position. That is exactly what counting needs. The status code or the IP address becomes the index, and the count or the sum becomes the value.

Requests per status code:

awk '{count[$9]++} END {for (code in count) print code, count[code]}' access.log | sort

count[$9]++ adds one to the entry for this record's status code. The END rule walks the array with for (code in count) and prints each code with its count. The sort at the end gives the same order on every run and every awk. Check the result against the file by eye: it holds three 200 lines, two 404 lines and one 500 line.

Bytes per client address:

awk '{bytes[$1] += $NF} END {for (ip in bytes) print bytes[ip], ip}' access.log | sort -rn

The sum goes first on each line so that sort -rn can put the largest number at the top. By hand: 198.51.100.7 sent 1024 + 128, which is 1152. 203.0.113.10 sent 512 + 128 + 64, which is 704. 192.0.2.44 sent 256. So 198.51.100.7 should head the list.

Two more array tools. length() with an array returns its number of elements, and in tests whether an index exists:

awk '{seen[$1] = 1} END {print length(seen), "distinct clients"; if ("192.0.2.44" in seen) print "192.0.2.44 made a request"}' access.log

The file has three different client addresses, so the count should be 3. The same pattern counts accounts per login shell in /etc/passwd:

awk -F: '{n[$7]++} END {for (s in n) print n[s], s}' /etc/passwd | sort -rn

To keep a summary and still see it on screen, end the pipeline with tee, for example | sort -rn | tee bytes-per-ip.txt. The guide on sending one command's output to the screen and a file with tee covers the details.

Running awk over many files

awk reads every file you name on its command line as one input stream, so NR and the END totals cover all of them. A glob over thousands of files can fail before awk even starts, because the expanded list is too long for one command line. The guide on fixing the Argument list too long error explains that limit. For awk, the simple fix is to let find feed one awk through a pipe:

find . -name '*.log' -exec cat {} + | awk '{count[$9]++} END {for (code in count) print code, count[code]}' | sort

If the list is long, find can start cat more than once. That is fine here, because every cat writes into the same pipe and a single awk reads it, so there is still one END with one set of totals. Do not write -exec awk ... {} + for totals. When find splits the list, each awk run prints its own partial totals.

awk is good for a question you have right now. If you are asking the same questions of the same logs every day, across several servers, collect the logs in one place instead. A self-hosted log management stack on a VPS keeps history and lets you search it.

When are cut or sed simpler than awk?

Use cut when the separator is one fixed character and you only want columns. Its manual says -d sets the delimiter (TAB by default), -f selects fields, and you use exactly one of -b, -c or -f. A line with no delimiter is printed whole unless you add -s.

cut -d: -f1,7 /etc/passwd

This prints the same login names and shells as the awk version, with less to type. cut has no patterns, no arithmetic and no arrays, so the moment you need "only the lines where" or "add up", go back to awk. For blank-separated text such as the access log, also stay with awk, because its default splitting is defined for runs of blanks and those are what log columns are made of. On Ubuntu 26.10 the default core utilities run on the Rust-based uutils implementation, so cut --version tells you which cut you have.

Use sed when you want to change text rather than pick columns. sed is a stream editor. -n turns off its automatic printing, -E turns on extended regular expressions, and s/regexp/replacement/ replaces the first match on each line.

sed -n '/ 404 /p' access.log
sed -E 's/^[0-9]+\.[0-9]+\.[0-9]+\.[0-9]+/x.x.x.x/' access.log

The first prints only lines that contain 404 . The second hides each client address before you paste a log sample into a ticket. sed -i edits a file in place. Run the command without -i first and read the output, because -i overwrites the file.

Why does $1 vanish inside double quotes?

Always put an awk program in single quotes. The bash manual says single quotes keep the literal value of every character inside them, so $1 reaches awk unchanged. Inside double quotes, $ keeps its special meaning to the shell. The shell replaces $1 with its own first positional parameter before awk starts, which means awk never sees the $1 you typed.

You can watch the shell do it. echo prints exactly the program awk would receive:

echo "{print $1}"
echo '{print $1}'

The second line shows {print $1} intact. The first shows whatever your shell put in place of $1. In an interactive shell that is usually nothing at all. Inside a script it is worse, because $1 is the script's first argument. The caller's text is pasted straight into your awk program.

To use a shell variable, pass it with -v and keep the program in single quotes:

code=404
awk -v code="$code" '$9 == code {print $1}' access.log

The bash manual also says a single quote cannot occur between single quotes, even after a backslash. When a program needs one, or grows past one line, put it in a file and run it with -f:

cat > status.awk <<'EOF'
{count[$9]++}
END {for (code in count) print code, count[code]}
EOF
awk -f status.awk access.log | sort

The quotes around EOF stop the shell from expanding $9 inside the here-document, so the file holds the program exactly as written.

FAQ

Why does my awk command print the whole line instead of one column?

The program is almost certainly in double quotes. Inside double quotes the shell replaces $1 with its own first positional parameter before awk runs. In an interactive shell that is usually empty, so awk does not receive the $1 you typed. Run echo "{print $1}" to see what awk actually gets. Put the program in single quotes, and pass shell values in with -v name="$value".

Is mawk or gawk the default awk on Ubuntu?

Do not rely on an answer from another machine. Different images can ship a different awk, and installing the gawk package can change where awk points. Run readlink -f "$(command -v awk)" on the server itself. Then run gawk --version or mawk -W version, whichever matches the path. If a script needs gawk features, install gawk and call gawk by name.

Why do strftime or gensub not work in my awk script?

The gawk manual describes gensub, strftime, mktime, asort and several others as gawk functions. The mawk manual says itself that mktime, strftime and systime are gawk extensions, which mawk provides only where the C library supports them. Check which awk runs your script with readlink -f "$(command -v awk)". If you need these functions, run sudo apt install -y gawk and start the script with gawk instead of awk.

How do I skip the header line of a CSV file in awk?

Add NR > 1 as the pattern, for example awk -F, 'NR > 1 {print $1, $3}' file.csv. NR is the number of the current record from the start of the input, so the header is record 1 and the pattern skips it. Note that -F, splits on every comma, including commas inside quoted fields, so use it only on CSV that does not quote its fields.

How do I count requests per status code with awk?

Use an associative array indexed by the status field, and print it in an END rule: awk '{count[$9]++} END {for (code in count) print code, count[code]}' access.log | sort. In the common access log layout the status code is field 9. Confirm that on your own file first with awk '{print NR, NF, $9}'. The sort gives the output the same order on every run.