jq: The Swiss Army Knife for JSON Processing#

When processing JSON on the command line, many reach for grep with regex or write a quick Python one-liner. But jq, designed specifically for JSON, is the true efficiency booster.

Core Model: Pipes and Filters#

jq draws from functional programming—data flows through a series of filters, each doing one thing, combined to solve complex problems.

# The simplest example
echo '{"name": "Alice", "age": 28}' | jq '.name'
# Output: "Alice"

. is the identity filter, representing current data. .name accesses object property, equivalent to JavaScript’s obj.name.

Four Common Operations#

1. Array Operations#

# Array element access
echo '[1, 2, 3, 4, 5]' | jq '.[0]'      # First element: 1
echo '[1, 2, 3, 4, 5]' | jq '.[-1]'     # Last element: 5
echo '[1, 2, 3, 4, 5]' | jq '.[1:3]'    # Slice: [2, 3]

# Array iteration
echo '[{"id": 1}, {"id": 2}]' | jq '.[].id'  # Outputs: 1 and 2 (one per line)

# Array mapping
echo '[1, 2, 3]' | jq 'map(. * 2)'      # [2, 4, 6]

# Array filtering
echo '[1, 2, 3, 4, 5]' | jq 'map(select(. >= 3))'  # [3, 4, 5]

2. Object Operations#

# Construct new object
echo '{"name": "Alice", "age": 28}' | jq '{username: .name, years: .age}'
# Output: {"username": "Alice", "years": 28}

# Merge objects
echo '{"a": 1}' | jq '. + {"b": 2}'    # {"a": 1, "b": 2}

# Delete field
echo '{"a": 1, "b": 2}' | jq 'del(.b)' # {"a": 1}

# Recursive update
echo '{"a": {"b": 1}}' | jq '.a.b = 10'

3. String Operations#

# String interpolation
echo '{"name": "Alice"}' | jq '"Hello, \(.name)!"'  # "Hello, Alice!"

# Split and join
echo '"a,b,c"' | jq 'split(",")'        # ["a", "b", "c"]
echo '["a", "b", "c"]' | jq 'join("-")' # "a-b-c"

# Regex matching
echo '"hello123world"' | jq 'match("[0-9]+")'

4. Conditionals#

# if-then-else
echo '5' | jq 'if . > 3 then "big" else "small" end'

# Complex conditions
echo '{"score": 85}' | jq '
  if .score >= 90 then "A"
  elif .score >= 80 then "B"
  elif .score >= 70 then "C"
  else "D"
  end'

Real-World: Processing API Responses#

Fetching repos from GitHub API:

curl -s "https://api.github.com/users/torvalds/repos?per_page=100" | jq '
  map({
    name: .name,
    stars: .stargazers_count,
    lang: .language,
    url: .html_url
  }) |
  sort_by(-.stars) |
  .[:5]
'

This extracts key fields, constructs new objects, sorts by stars descending, and takes top 5. Much cleaner than equivalent grep/awk/sed combos.

Performance#

jq is written in C and parses fast. Testing a 50MB JSON file:

time jq '.' large.json > /dev/null
# Measured: ~0.3 seconds

Compare with Python:

time python3 -c "import json, sys; json.load(sys.stdin)" < large.json
# Measured: ~0.8 seconds

Memory-wise, jq loads the entire JSON into memory, not ideal for GB-scale files. For those, use jq’s streaming mode (jq -c) with head for batch processing.

Debugging Tips#

Complex jq expressions can be tricky. Some debugging approaches:

# 1. Print intermediate results
echo '{"data": [1, 2, 3]}' | jq '.data | debug | map(. * 2)'

# 2. Type checking
echo '[1, "a", null, true]' | jq 'map(type)'
# ["number", "string", "null", "boolean"]

# 3. Error handling
echo '[1, 2, "x"]' | jq 'map(. * 2 // "non-numeric")'
# [2, 4, "non-numeric"]

Integration with Other Tools#

jq shines as part of a pipeline:

# Parse logs (JSON Lines format)
cat app.log | jq 'select(.level == "error") | .message' | head -20

# Calculate average API response time
curl -s https://api.example.com/stats | jq '.requests | map(.latency) | add / length'

# Generate CSV
jq -r '.[] | [.name, .age, .city] | @csv' data.json > output.csv

-r outputs raw strings (no quotes), @csv is a built-in formatter.

Summary#

jq has a learning curve, but mastering it dramatically improves JSON processing efficiency. Remember: pipes + filters—each operation does one thing, combined to solve complex problems.

Try jq expressions online: JSON Path Finder

Related: JSON Formatter | JSON Diff