All demos link to jqterm - an alternative interactive jq web terminal with autocomplete (and faster querying with large datasets as they're cached online in private gists).

Here's a collection of jq recipes I've collected over the last few months.

READER DISCOUNTSave $50 on terminal.training

I've published 38 videos for new developers, designers, UX, UI, product owners and anyone who needs to conquer the command line today.

Push on to an existing array (where source is [1, 2, 3]):

. + [ 4 ] # result: [ 1, 2, 3, 4 ]

Demo


Convert object to array, moving the key into the array item under the property slug:

to_entries | map_values(.value + { slug: .key })

Demo


Convert an array to a keyed object (the inverse of the above example):

map({ (.slug): . }) | add

Demo


Swap the key/value pair to read as value/key object:

to_entries | map( {(.value) : .key } ) | add

Demo


Read a plain list of strings from a file into an array, specifically splitting into an array and removing the last empty \n:

echo "1\n2\n3" | jq --slurp --raw-input 'split("\n")[:-1]'

Demo


Convert a plain list of timestamps to an array of objects with date and time separated (using jq's --slurp and --raw-input options combined):

cat timestamps.txt | jq --slurp --raw-input 'split("\n")[:-1] | map({
	date: (. | strptime("%a, %d %b %Y %H:%M:%S") | todate[0:10]),
	time: (. | strptime("%a, %d %b %Y %H:%M:%S") | todate[11:19])
})'

Demo


From a plain list of timestamps, count the occurrences of unique days (the first part is from the example above):

split("\n")[:-1] | map({
  date: (. | strptime("%a, %d %b %Y %H:%M:%S") | todate[0:10]),
  time: (. | strptime("%a, %d %b %Y %H:%M:%S") | todate[11:19])
}) | reduce .[] as $item (
  {}; # initial value
  .[$item.date] += 1 # reducer
)

Demo


Take an object with two similar objects, but separated between team and formerly, and merge into a single object, adding a flag for all those from the formerly group:

[.team, (.formerly | map(. + {formerly: true }))] | flatten

Demo


Download and extract all the files from a gist:

eval "$(
  curl https://api.github.com/gists/968b8937a153127cfae4a173b6000c1e |
  jq -r '
    .files |
    to_entries |
    .[].value |
    @sh "echo \(.content) > \(.filename)"
  '
)"

Demo


Update all outdated npm dependencies to latest (possibly unsafe as it'll also update to major changes):

npm i $(echo $(npm outdated --json | jq -r 'to_entries | .[] | "\(.key)@\(.value.latest)"'))

Change the above from .latest to .wanted for a safe upgrade.

Demo


Install the dependencies from one node project to another:

npm i $(cat ../other-project/package.json| jq '.dependencies | keys[]' -r)

Demo


Add a new property to every object:

map(. + { "draft": true })

Or

[.[] | . + { "draft" : true }]

Demo


Add new property to every object in a nested object, i.e. source looks like:

{
  "offline-panel": {
    "title": "Offline Panel",
    "tags": ["web"]
  },
  "rewrite-it": {
    "title": "Let's just rewrite it",
    "tags": ["business"]
  }
}

Command:

with_entries(.value += { "draft": true})

Demo


Remove a property from a nested object (example as above):

with_entries(.value |= del(.title))

Demo


List all the dependencies in a package.json for use in other commands, like npm uninstall:

echo $(cat package.json | jq '.dependencies | keys | .[] | "\(.)"' -r)

Demo


Get mongodb data into jq compatible format:

mongo <host>/<db> --norc --username <user> --password <pwd> \
  --eval 'printjson(db.getCollection("users").find().toArray())' | \
  jq '.[]'

From Twitter's API, take all DM received and sent and transform into readable format sorted by date order:

[ .[] | {
  text,
  date: .created_at,
  from: { screen_name: .sender.screen_name },
  to: { screen_name: .recipient.screen_name}
} ] |
sort_by(.date)

Using Serverless and Next.js and working out which dependencies I need to force include (because they live in the .next directory):

$ depcheck --json |
  jq '
    .using |
    [
      to_entries[] |
      select(.value[] | contains("/.next/")) |
      .key
    ] |
    unique |
    sort[] | "- \(.)"
  ' -r

Note: also uses depcheck to resolve the npm dependencies.

Demo


From a nested tree of objects, find the object whose id matches X:

curl -sL https://git.io/vxPyi | \
  jq '.. | objects | select(.id == "0:16")'

Demo


Strip all occurrences of a property (email in this example):

walk(if type == "object" then . | del(.email) else . end)

Note that the walk function is missing from jq@1.5 and needs to be added (seen in demo).

Demo


Bulk insert into elastic search using a vanilla JSON array, i.e. [1,2,3,4] - zipping the array with the required elastic search metadata:

$ cat data.json | \
  jq 'reduce .[] as $n ([]; . + [{ "index" : { "_index": "my-index", "_type" : "my-type" } }, $n]) | .[]' -c | \
  curl -H "Content-Type: application/x-ndjson" -XPOST http://localhost:9200/_bulk --data-binary "@-"

Demo


Filter an array, similar to a JavaScript array filter:

def filter(cond): map(select(cond));

filter(. > 2)

Demo


Converting a text output of columns and converting to a JSON object. In this case, running Zeit's now ls | jq --raw-input --slurp to find out how many running instance I have:

split("\n")[1:-3] | # split into an array of strings, removing the 1st and last few blank lines
map([ split(" ")[] | select(. != "") ]) | # convert large spaces into individual colmns
map({ # map into a usable object
  app: .[0],
  url: .[1],
  number: (if (.[2] == "-") then .[2] else .[2] | tonumber end),
  type: .[3],
  state: .[4],
  age: .[5]
}) |
# now I can query the result - in this case: how many running and are npm
map(select(.number > 0 and .type == "NPM")) | length

Demo


Find duplicates in an array based on a key:

[
  reduce .[].id as $item (
    {}; # initial value
    .[$item] += 1
  ) | to_entries[] | select(.value > 1)
] | from_entries

Demo


Quickly convert a list of strings into an array (for JavaScript dev, etc):

$ pbpaste | jq -Rs 'split("\n")' | pbcopy

Demo


Strip empty strings from arrays (at any level deep):

walk(if type == "array" then map(select(length > 0)) else . end)

Note that this requires the walk method (that was removed in jq@1.5) but included in the demo below

Demo


Install my missing dependencies (determined by using depcheck):

.missing | to_entries | map(.key) | join(" ") | "npm i \(.)"

Demo


How to avoid non-existant keys when filtering for null. Where endpoint is sometimes missing, sometimes it's set to null and I want those objects. First ensure the key is present, then select if null:

map(select(has("endpoint") and .endpoint == null))

Demo


How to find null, ignoring non-existant keys - the invert of the above:

map(select(has("endpoint") | not))

Demo


CSV content transformed to a structured object, using rawInput and slurp. First the lines are split and the header is dropped, then each line is split by comma and mapped to a new object:

split("\n")[1:] | map(split(",") | { name: .[0 ], url: .[1], image: .[2], category: .[3] | tonumber })

Demo


Recursively find all the properties whose key is errors whether it exists or not. The .. unrolls the object, the ? checks for the value or returns null and the select(.) is like a filter on truthy values:

[.. | .errors?[0] | select(.) ]

Demo


A generic CSV to JSON in jq. Obviously overkill (see csvkit and specifically csvjson), but it's doable and a good example of variables and reduce:

split("\n") | # break into lines
  map(split(",")) | # comma sep
  .[0] as $header | # save the header
  .[1:] | # drop the header
  map(
      . as $o | # save the current object, then
      reduce .[] as $item( # reduce into a header keyed object
        {};
        ($o | index($item)) as $index |
        .[$header[$index]] = $item
    )
  )

Demo


Take an objects properties and use them as both the key and value, for instance with this source:

[
  {
    "label": "house",
    "value": "lloyds pharmacy"
  },
  {
    "label": "house_number",
    "value": "105"
  },
  {
    "label": "road",
    "value": "church road"
  },
  {
    "label": "postcode",
    "value": "bn3 2af"
  }
]

…to:

{
  "house": "lloyds pharmacy",
  "house_number": "105",
  "road": "church road",
  "postcode": "bn3 2af"
}

Using ( .<prop> ) as a dynamic key:

map({ (.label): .value }) | add

Demo


Getting the standard deviation (and variance and mean) for a series of numbers:

def mean: reduce .[] as $n (0; . + $n) / length;
def pow2: . * .;
def variance: . | mean as $mean | map_values(. - $mean | pow2) | mean;
def stdev: . | variance | sqrt;

# pick those scores who
[.[].score] | # convert numbers to array
stdev as $stdev | # store in a variable
mean as $mean | # also store numbers in mean variable
map(select(. - $stdev > $mean) | . - $stdev) # filter those > 1 x stdev (and show by how much)

Demo


Sorting an object by it's value. We have to unroll the object first, run the sort, then rebuild the object:

to_entries | sort_by(.value) | from_entries

Demo


Convert an irregular object to a CSV file:

map({ start, duration, title, speaker: .name, twitter }) | # generate consistent structure

map(to_entries | map(.value) | @csv)[] # turn into csv structure

Demo


Count the occurrences of an element in an array (note that I don't think is this the best way of doing this, but it works):

reduce to_entries[] as $_ (
  {};
  . + { ($_.value | tostring): (.[($_.value | tostring)] + 1) }
)

Demo


More of a word of warning: 0 is truthy:

if 0 then "true" else "false" end # "true"

Demo


List all the unique combinations without duplication in the list:

5 as $n | [reduce range(0; $n) as $i ([]; . + [[range(0;$n)]]) | combinations | select(unique | length == $n)]

Generates 120 long array of [0, 1, 2, 3, 4] in all the possible combinations.

Demo


Unique combinations of pairs of values - not allowing for repetition, i.e. [1,0] and [0,1] is duplication:

[1,2,3,4] | [combinations(2) | select(.[0] != .[1]) | sort] | unique

Generates all the combinations of [1,2], [1,3], [1,4], [2,3], [2,4], [3,4]

Demo


Chunk an array - so instead of an array containing 161 items, I have a two dimensional array 11 elements long holding 16 elements each except the last which holds one element:

# vars
. as $x | # capture the original data
(($x | length)/10) | floor as $by | # store size of 10%

[range(0;$x | length;$by)] | # generate a range jumping by 10%

map($x[.:.+$by]) # chunk the source data

Demo


Transform an object into jsdoc like property types and names:

to_entries |
# add the type and remove the value
map(. + { type: .value | type } | del(.value)) |
# transform to text in jsdoc fragment
map("* @property {\(.type)} \(.key)")[]

Demo


Convert a decimal value into a hex string:

def hexChr:
  floor | if . < 10 then . else ["A", "B", "C", "D", "E", "F"][. % 10] end
;

def toHex:
  def toHex:
      if . / 16 >= 1 then
          (. / 16 | toHex), (. % 16 | hexChr)
      else
          . % 16 | hexChr
      end
  ;
  [toHex] | join("")
;

toHex

Demo


Convert JSON object into a flat path based schema, i.e:

{ a: [ { b: 1 }, { b: 2} ], c: { d: true } }

// into
a.b
c.d
def flat($prefix):
	map(.key as $key | (.value | type) as $type |
      if $type == "array" then
      	.value[0] | to_entries | flat($prefix + $key + ".")
      elif $type == "object" then
		.value | to_entries | flat($prefix + $key + ".")
      else
      	"\($prefix)\(.key)"
      end
	)[]
;

def flat: flat("");

first | to_entries | flat

Demo


Select the arrays where all the items are the same value:

def allSame:
	first as $first | all(. == $first);

to_entries | map(select(.value | allSame)) | from_entries

Demo


Sum of values where numbers are new line separated and slurped:

reduce .[] as $_ (0; (. | tonumber) + $_)

Demo


Repeat an array:

[. as $_ | range($n) | $_] | flatten

Demo


Convert binary to decimal

def fromBinary:
	reverse | . as $_ |
	reduce range(length) as $i (
      0;
      if $_[$i] == 1 then
        . + pow(2; $i)
      else
        .
      end
	)
;

"1001" | split("") | map(tonumber) | fromBinary

Demo


All the unique domains requested by a web page. Open devtools, open the network tab and hit reload. Then select one entry and "Copy" - "Copy all as HAR":

[.log.entries[].request.url | split("/") | "\(.[0])//\(.[2])"] | unique

Demo

Drafts may be incomplete or entirely abandoned, so please forgive me. If you find an issue with a draft, or would like to see me write about something specifically, please try raising an issue.