All demos link to jqterm - an alternative interactive jq web terminal with autocomplete (and faster querying with large datasets as they're cached online in private gists).

Here's a collection of jq recipes I've collected over the last few months.

READER DISCOUNTSave $50 on terminal.training

I've published 38 videos for new developers, designers, UX, UI, product owners and anyone who needs to conquer the command line today.

Push on to an existing array (where source is [1, 2, 3]):

. + [ 4 ] # result: [ 1, 2, 3, 4 ]

Demo


Convert object to array, moving the key into the array item under the property slug:

to_entries | map_values(.value + { slug: .key })

Demo


Convert an array to a keyed object (the inverse of the above example):

map({ (.slug): . }) | add

Demo


Swap the key/value pair to read as value/key object:

to_entries | map( {(.value) : .key } ) | add

Demo


Read a plain list of strings from a file into an array, specifically splitting into an array and removing the last empty \n:

echo "1\n2\n3" | jq --slurp --raw-input 'split("\n")[:-1]'

Demo


Convert a plain list of timestamps to an array of objects with date and time separated (using jq's --slurp and --raw-input options combined):

cat timestamps.txt | jq --slurp --raw-input 'split("\n")[:-1] | map({
	date: (. | strptime("%a, %d %b %Y %H:%M:%S") | todate[0:10]),
	time: (. | strptime("%a, %d %b %Y %H:%M:%S") | todate[11:19])
})'

Demo


From a plain list of timestamps, count the occurrences of unique days (the first part is from the example above):

split("\n")[:-1] | map({
  date: (. | strptime("%a, %d %b %Y %H:%M:%S") | todate[0:10]),
  time: (. | strptime("%a, %d %b %Y %H:%M:%S") | todate[11:19])
}) | reduce .[] as $item (
  {}; # initial value
  .[$item.date] += 1 # reducer
)

Demo


Take an object with two similar objects, but separated between team and formerly, and merge into a single object, adding a flag for all those from the formerly group:

[.team, (.formerly | map(. + {formerly: true }))] | flatten

Demo


Download and extract all the files from a gist:

eval "$(
  curl https://api.github.com/gists/968b8937a153127cfae4a173b6000c1e |
  jq -r '
    .files |
    to_entries |
    .[].value |
    @sh "echo \(.content) > \(.filename)"
  '
)"

Demo


Update all outdated npm dependencies to latest (possibly unsafe as it'll also update to major changes):

npm i $(echo $(npm outdated --json | jq -r 'to_entries | .[] | "\(.key)@\(.value.latest)"'))

Change the above from .latest to .wanted for a safe upgrade.

Demo


Install the dependencies from one node project to another:

npm i $(cat ../other-project/package.json| jq '.dependencies | keys[]' -r)

Demo


Add a new property to every object:

map(. + { "draft": true })

Or

[.[] | . + { "draft" : true }]

Demo


Add new property to every object in a nested object, i.e. source looks like:

{
 "offline-panel": {
    "title": "Offline Panel",
    "tags": [
      "web"
    ]
  },
  "rewrite-it": {
    "title": "Let's just rewrite it",
    "tags": [
      "business"
    ]
  }
}

Command:

with_entries(.value += { "draft": true})

Demo


Remove a property from a nested object (example as above):

with_entries(.value |= del(.title))

Demo


List all the dependencies in a package.json for use in other commands, like npm uninstall:

echo $(cat package.json | jq '.dependencies | keys | .[] | "\(.)"' -r)

Demo


Get mongodb data into jq compatible format:

mongo <host>/<db> --norc --username <user> --password <pwd> \
  --eval 'printjson(db.getCollection("users").find().toArray())' | \
  jq '.[]'

From Twitter's API, take all DM received and sent and transform into readable format sorted by date order:

[ .[] | {
  text,
  date: .created_at,
  from: { screen_name: .sender.screen_name },
  to: { screen_name: .recipient.screen_name}
} ] |
sort_by(.date)

Using Serverless and Next.js and working out which dependencies I need to force include (because they live in the .next directory):

$ depcheck --json |
  jq '
    .using |
    [
      to_entries[] |
      select(.value[] | contains("/.next/")) |
      .key
    ] |
    unique |
    sort[] | "- \(.)"
  ' -r

Note: also uses depcheck to resolve the npm dependencies.

Demo


From a nested tree of objects, find the object whose id matches X:

curl -sL https://git.io/vxPyi | \
  jq '.. | objects | select(.id == "0:16")'

Demo


Strip all occurrences of a property (email in this example):

walk(if type == "object" then . | del(.email) else . end)

Note that the walk function is missing from jq@1.5 and needs to be added (seen in demo).

Demo


Bulk insert into elastic search using a vanilla JSON array, i.e. [1,2,3,4] - zipping the array with the required elastic search metadata:

$ cat data.json | \
  jq 'reduce .[] as $n ([]; . + [{ "index" : { "_index": "my-index", "_type" : "my-type" } }, $n]) | .[]' -c | \
  curl -H "Content-Type: application/x-ndjson" -XPOST http://localhost:9200/_bulk --data-binary "@-"

Demo


Filter an array, similar to a JavaScript array filter:

def filter(cond): map(select(cond));

filter(. > 2)

Demo


Converting a text output of columns and converting to a JSON object. In this case, running Zeit's now ls | jq --raw-input --slurp to find out how many running instance I have:

split("\n")[1:-3] | # split into an array of strings, removing the 1st and last few blank lines
map([ split(" ")[] | select(. != "") ]) | # convert large spaces into individual colmns
map({ # map into a usable object
  app: .[0],
  url: .[1],
  number: (if (.[2] == "-") then .[2] else .[2] | tonumber end),
  type: .[3],
  state: .[4],
  age: .[5]
}) |
# now I can query the result - in this case: how many running and are npm
map(select(.number > 0 and .type == "NPM")) | length

Demo


Find duplicates in an array based on a key:

[
  reduce .[].id as $item (
    {}; # initial value
    .[$item] += 1
  ) | to_entries[] | select(.value > 1)
] | from_entries

Demo


Quickly convert a list of strings into an array (for JavaScript dev, etc):

$ pbpaste | jq -Rs 'split("\n")' | pbcopy

Demo


Strip empty strings from arrays (at any level deep):

walk(if type == "array" then map(select(length > 0)) else . end)

Note that this requires the walk method (that was removed in jq@1.5) but included in the demo below

Demo


Install my missing dependencies (determined by using depcheck):

.missing | to_entries | map(.key) | join(" ") | "npm i \(.)"

Demo


How to avoid non-existant keys when filtering for null. Where endpoint is sometimes missing, sometimes it's set to null and I want those objects. First ensure the key is present, then select if null:

map(select(has("endpoint") and .endpoint == null))

Demo


How to find null, ignoring non-existant keys - the invert of the above:

map(select(has("endpoint") | not))

Demo


CSV content transformed to a structured object, using rawInput and slurp. First the lines are split and the header is dropped, then each line is split by comma and mapped to a new object:

split("\n")[1:] | map(split(",") | { name: .[0 ], url: .[1], image: .[2], category: .[3] | tonumber })

Demo


Recursively find all the properties whose key is errors whether it exists or not. The .. unrolls the object, the ? checks for the value or returns null and the select(.) is like a filter on truthy values:

[.. | .errors?[0] | select(.) ]

Demo


A generic CSV to JSON in jq. Obviously overkill (see csvkit and specifically csvjson), but it's doable and a good example of variables and reduce:

split("\n") | # break into lines
  map(split(",")) | # comma sep
  .[0] as $header | # save the header
  .[1:] | # drop the header
  map(
      . as $o | # save the current object, then
      reduce .[] as $item( # reduce into a header keyed object
        {};
        ($o | index($item)) as $index |
        .[$header[$index]] = $item
    )
  )

Demo


Take an objects properties and use them as both the key and value, for instance with this source:

[
  {
    "label": "house",
    "value": "lloyds pharmacy"
  },
  {
    "label": "house_number",
    "value": "105"
  },
  {
    "label": "road",
    "value": "church road"
  },
  {
    "label": "postcode",
    "value": "bn3 2af"
  }
]

…to:

{
  "house": "lloyds pharmacy",
  "house_number": "105",
  "road": "church road",
  "postcode": "bn3 2af"
}

Using ( .<prop> ) as a dynamic key:

map({ (.label): .value }) | add

Demo


Getting the standard deviation (and variance and mean) for a series of numbers:

def mean: reduce .[] as $n (0; . + $n) / length;
def pow2: . * .;
def variance: . | mean as $mean | map_values(. - $mean | pow2) | mean;
def stdev: . | variance | sqrt;

# pick those scores who
[.[].score] | # convert numbers to array
stdev as $stdev | # store in a variable
mean as $mean | # also store numbers in mean variable
map(select(. - $stdev > $mean) | . - $stdev) # filter those > 1 x stdev (and show by how much)

Demo


Sorting an object by it's value. We have to unroll the object first, run the sort, then rebuild the object:

to_entries | sort_by(.value) | from_entries

Demo


Convert an irregular object to a CSV file:

map({ start, duration, title, speaker: .name, twitter }) | # generate consistent structure

map(to_entries | map(.value) | @csv)[] # turn into csv structure

Demo

Drafts may be incomplete or entirely abandoned, so please forgive me. If you find an issue with a draft, or would like to see me write about something specifically, please try raising an issue.