jq recipes

All demos link to jqterm - an alternative interactive jq web terminal with autocomplete (and faster querying with large datasets as they're cached online in private gists).

Here's a collection of jq recipes I've collected over the last few months.

Push on to an existing array (where source is [1, 2, 3]):

. + [ 4 ] # result: [ 1, 2, 3, 4 ]

Demo

---

Convert object to array, moving the key into the array item under the property slug:

to_entries | map_values(.value + { slug: .key })

Demo

---

Convert an array to a keyed object (the inverse of the above example):

map({ (.slug): . }) | add

Demo

---

Swap the key/value pair to read as value/key object:

to_entries | map( {(.value) : .key } ) | add

Demo

---

Read a plain list of strings from a file into an array, specifically splitting into an array and removing the last empty \n:

echo "1\n2\n3" | jq --slurp --raw-input 'split("\n")[:-1]'

Demo

---

Convert a plain list of timestamps to an array of objects with date and time separated (using jq's --slurp and --raw-input options combined):

cat timestamps.txt | jq --slurp --raw-input 'split("\n")[:-1] | map({
	date: (. | strptime("%a, %d %b %Y %H:%M:%S") | todate[0:10]),
	time: (. | strptime("%a, %d %b %Y %H:%M:%S") | todate[11:19])
})'

Demo

---

From a plain list of timestamps, count the occurrences of unique days (the first part is from the example above):

split("\n")[:-1] | map({
  date: (. | strptime("%a, %d %b %Y %H:%M:%S") | todate[0:10]),
  time: (. | strptime("%a, %d %b %Y %H:%M:%S") | todate[11:19])
}) | reduce .[] as $item (
  {}; # initial value
  .[$item.date] += 1 # reducer
)

Demo

---

Take an object with two similar objects, but separated between team and formerly, and merge into a single object, adding a flag for all those from the formerly group:

[.team, (.formerly | map(. + {formerly: true }))] | flatten

Demo

---

Download and extract all the files from a gist:

eval "$(
  curl https://api.github.com/gists/968b8937a153127cfae4a173b6000c1e |
  jq -r '
    .files |
    to_entries |
    .[].value |
    @sh "echo \(.content) > \(.filename)"
  '
)"

Demo

---

Update all outdated npm dependencies to latest (possibly unsafe as it'll also update to major changes):

npm i $(echo $(npm outdated --json | jq -r 'to_entries | .[] | "\(.key)@\(.value.latest)"'))

Change the above from .latest to .wanted for a safe upgrade.

Demo

---

Install the dependencies from one node project to another:

npm i $(cat ../other-project/package.json| jq '.dependencies | keys[]' -r)

Demo

---

Add a new property to every object:

map(. + { "draft": true })

Or

[.[] | . + { "draft" : true }]

Demo

---

Add new property to every object in a nested object, i.e. source looks like:

{
  "offline-panel": {
    "title": "Offline Panel",
    "tags": ["web"]
  },
  "rewrite-it": {
    "title": "Let's just rewrite it",
    "tags": ["business"]
  }
}

Command:

with_entries(.value += { "draft": true})

Demo

---

Remove a property from a nested object (example as above):

with_entries(.value |= del(.title))

Demo

---

List all the dependencies in a package.json for use in other commands, like npm uninstall:

echo $(cat package.json | jq '.dependencies | keys | .[] | "\(.)"' -r)

Demo

---

Get mongodb data into jq compatible format:

mongo <host>/<db> --norc --username <user> --password <pwd> \
  --eval 'printjson(db.getCollection("users").find().toArray())' | \
  jq '.[]'

---

From Twitter's API, take all DM received and sent and transform into readable format sorted by date order:

[ .[] | {
  text,
  date: .created_at,
  from: { screen_name: .sender.screen_name },
  to: { screen_name: .recipient.screen_name}
} ] |
sort_by(.date)

---

Using Serverless and Next.js and working out which dependencies I need to force include (because they live in the .next directory):

$ depcheck --json |
  jq '
    .using |
    [
      to_entries[] |
      select(.value[] | contains("/.next/")) |
      .key
    ] |
    unique |
    sort[] | "- \(.)"
  ' -r

Note: also uses depcheck to resolve the npm dependencies.

Demo

---

From a nested tree of objects, find the object whose id matches X:

curl -sL https://git.io/vxPyi | \
  jq '.. | objects | select(.id == "0:16")'

Demo

---

Strip all occurrences of a property (email in this example):

walk(if type == "object" then . | del(.email) else . end)

Note that the walk function is missing from jq@1.5 and needs to be added (seen in demo).

Demo

---

Bulk insert into elastic search using a vanilla JSON array, i.e. [1,2,3,4] - zipping the array with the required elastic search metadata:

$ cat data.json | \
  jq 'reduce .[] as $n ([]; . + [{ "index" : { "_index": "my-index", "_type" : "my-type" } }, $n]) | .[]' -c | \
  curl -H "Content-Type: application/x-ndjson" -XPOST http://localhost:9200/_bulk --data-binary "@-"

Demo

---

Filter an array, similar to a JavaScript array filter:

def filter(cond): map(select(cond));

filter(. > 2)

Demo

---

Converting a text output of columns and converting to a JSON object. In this case, running Zeit's now ls | jq --raw-input --slurp to find out how many running instance I have:

split("\n")[1:-3] | # split into an array of strings, removing the 1st and last few blank lines
map([ split(" ")[] | select(. != "") ]) | # convert large spaces into individual colmns
map({ # map into a usable object
  app: .[0],
  url: .[1],
  number: (if (.[2] == "-") then .[2] else .[2] | tonumber end),
  type: .[3],
  state: .[4],
  age: .[5]
}) |
# now I can query the result - in this case: how many running and are npm
map(select(.number > 0 and .type == "NPM")) | length

Demo

---

Find duplicates in an array based on a key:

[
  reduce .[].id as $item (
    {}; # initial value
    .[$item] += 1
  ) | to_entries[] | select(.value > 1)
] | from_entries

Demo

---

Quickly convert a list of strings into an array (for JavaScript dev, etc):

$ pbpaste | jq -Rs 'split("\n")' | pbcopy

Demo

---

Strip empty strings from arrays (at any level deep):

walk(if type == "array" then map(select(length > 0)) else . end)

Note that this requires the walk method (that was removed in jq@1.5) but included in the demo below

Demo

---

Install my missing dependencies (determined by using depcheck):

.missing | to_entries | map(.key) | join(" ") | "npm i \(.)"

Demo

---

How to avoid non-existant keys when filtering for null. Where endpoint is sometimes missing, sometimes it's set to null and I want those objects. First ensure the key is present, then select if null:

map(select(has("endpoint") and .endpoint == null))

Demo

---

How to find null, ignoring non-existant keys - the invert of the above:

map(select(has("endpoint") | not))

Demo

---

CSV content transformed to a structured object, using rawInput and slurp. First the lines are split and the header is dropped, then each line is split by comma and mapped to a new object:

split("\n")[1:] | map(split(",") | { name: .[0 ], url: .[1], image: .[2], category: .[3] | tonumber })

Demo

---

Recursively find all the properties whose key is errors whether it exists or not. The .. unrolls the object, the ? checks for the value or returns null and the select(.) is like a filter on truthy values:

[.. | .errors?[0] | select(.) ]

Demo

---

A generic CSV to JSON in jq. Obviously overkill (see csvkit and specifically csvjson), but it's doable and a good example of variables and reduce:

split("\n") | # break into lines
  map(split(",")) | # comma sep
  .[0] as $header | # save the header
  .[1:] | # drop the header
  map(
      . as $o | # save the current object, then
      reduce .[] as $item( # reduce into a header keyed object
        {};
        ($o | index($item)) as $index |
        .[$header[$index]] = $item
    )
  )

Demo

---

Take an objects properties and use them as both the key and value, for instance with this source:

[
  {
    "label": "house",
    "value": "lloyds pharmacy"
  },
  {
    "label": "house_number",
    "value": "105"
  },
  {
    "label": "road",
    "value": "church road"
  },
  {
    "label": "postcode",
    "value": "bn3 2af"
  }
]

…to:

{
  "house": "lloyds pharmacy",
  "house_number": "105",
  "road": "church road",
  "postcode": "bn3 2af"
}

Using ( .<prop> ) as a dynamic key:

map({ (.label): .value }) | add

Demo

---

Getting the standard deviation (and variance and mean) for a series of numbers:

def mean: reduce .[] as $n (0; . + $n) / length;
def pow2: . * .;
def variance: . | mean as $mean | map_values(. - $mean | pow2) | mean;
def stdev: . | variance | sqrt;

# pick those scores who
[.[].score] | # convert numbers to array
stdev as $stdev | # store in a variable
mean as $mean | # also store numbers in mean variable
map(select(. - $stdev > $mean) | . - $stdev) # filter those > 1 x stdev (and show by how much)

Demo

---

Sorting an object by it's value. We have to unroll the object first, run the sort, then rebuild the object:

to_entries | sort_by(.value) | from_entries

Demo

---

Convert an irregular object to a CSV file:

map({ start, duration, title, speaker: .name, twitter }) | # generate consistent structure

map(to_entries | map(.value) | @csv)[] # turn into csv structure

Demo

---

Count the occurrences of an element in an array (note that I don't think is this the best way of doing this, but it works):

reduce to_entries[] as $_ (
  {};
  . + { ($_.value | tostring): (.[($_.value | tostring)] + 1) }
)

Demo

---

More of a word of warning: 0 is truthy:

if 0 then "true" else "false" end # "true"

Demo

---

List all the unique combinations without duplication in the list:

5 as $n | [reduce range(0; $n) as $i ([]; . + [[range(0;$n)]]) | combinations | select(unique | length == $n)]

Generates 120 long array of [0, 1, 2, 3, 4] in all the possible combinations.

Demo

---

Unique combinations of pairs of values - not allowing for repetition, i.e. [1,0] and [0,1] is duplication:

[1,2,3,4] | [combinations(2) | select(.[0] != .[1]) | sort] | unique

Generates all the combinations of [1,2], [1,3], [1,4], [2,3], [2,4], [3,4]

Demo

---

Chunk an array - so instead of an array containing 161 items, I have a two dimensional array 11 elements long holding 16 elements each except the last which holds one element:

# vars
. as $x | # capture the original data
(($x | length)/10) | floor as $by | # store size of 10%

[range(0;$x | length;$by)] | # generate a range jumping by 10%

map($x[.:.+$by]) # chunk the source data

Demo

---

Transform an object into jsdoc like property types and names:

to_entries |
# add the type and remove the value
map(. + { type: .value | type } | del(.value)) |
# transform to text in jsdoc fragment
map("* @property {\(.type)} \(.key)")[]

Demo

---

Convert a decimal value into a hex string:

def hexChr:
  floor | if . < 10 then . else ["A", "B", "C", "D", "E", "F"][. % 10] end
;

def toHex:
  def toHex:
      if . / 16 >= 1 then
          (. / 16 | toHex), (. % 16 | hexChr)
      else
          . % 16 | hexChr
      end
  ;
  [toHex] | join("")
;

toHex

Demo

---

Convert JSON object into a flat path based schema, i.e:

{ a: [ { b: 1 }, { b: 2} ], c: { d: true } }

// into
a.b
c.d

def flat($prefix):
	map(.key as $key | (.value | type) as $type |
      if $type == "array" then
      	.value[0] | to_entries | flat($prefix + $key + ".")
      elif $type == "object" then
		.value | to_entries | flat($prefix + $key + ".")
      else
      	"\($prefix)\(.key)"
      end
	)[]
;

def flat: flat("");

first | to_entries | flat

Demo

---

Select the arrays where all the items are the same value:

def allSame:
	first as $first | all(. == $first);

to_entries | map(select(.value | allSame)) | from_entries

Demo

---

Sum of values where numbers are new line separated and slurped:

reduce .[] as $_ (0; (. | tonumber) + $_)

Demo

---

Repeat an array:

[. as $_ | range($n) | $_] | flatten

Demo

---

Convert binary to decimal

def fromBinary:
	reverse | . as $_ |
	reduce range(length) as $i (
      0;
      if $_[$i] == 1 then
        . + pow(2; $i)
      else
        .
      end
	)
;

"1001" | split("") | map(tonumber) | fromBinary

Demo

---

All the unique domains requested by a web page. Open devtools, open the network tab and hit reload. Then select one entry and "Copy" - "Copy all as HAR":

[.log.entries[].request.url | split("/") | "\(.[0])//\(.[2])"] | unique

Demo

---

Pick every odd element in the array:

to_entries | map(select(.key % 2 == 0).value)

Demo

Drafts may be incomplete or entirely abandoned, so please forgive me. If you find an issue with a draft, or would like to see me write about something specifically, please try raising an issue.