CSV to JSON conversion is commonly used in the Monitoring part of the DevOps cycle, usually when we need to gather some data, parse it, and ship it via some monitoring agent, like Fluentd for example, to a centralized monitoring service. And, if we don’t have any plugin installed that handles and parses the data for us, this is the most “native” thing we can do.
In today’s blog post, the focus is going to be put on converting Linux commands output data from CSV to JSON.
Prerequisites
- Linux bash environment
Example solution
Let’s say we want to convert the df -h command output from CSV to JSON.
Step 1. Run the following command:
df -h
Output:
Filesystem Size Used Avail Use% Mounted on
udev 3.4G 0 3.4G 0% /dev
tmpfs 697M 1.2M 696M 1% /run
/dev/sda2 40G 8.9G 29G 24% /
tmpfs 3.5G 0 3.5G 0% /dev/shm
tmpfs 5.0M 0 5.0M 0% /run/lock
tmpfs 3.5G 0 3.5G 0% /sys/fs/cgroup
/dev/loop0 89M 89M 0 100% /snap/core/7270
tmpfs 697M 0 697M 0% /run/user/1012
Step 2. Let’s say we want to filter the column names and root block device only.
df -h | grep -E 'Filesystem|/dev/sda2'
Expected output:
Filesystem Size Used Avail Use% Mounted on
/dev/sda2 40G 8.9G 29G 24% /
Step 3. Let’s remove the repetitive spaces and replace them with commas.
df -h | grep -E 'Filesystem|/dev/sda2' | tr -s ' ' ','
Expected output:
Filesystem,Size,Used,Avail,Use%,Mounted,on
/dev/sda2,40G,8.9G,29G,24%,/
Step 4. Now, do the final transformation using the jq command-line tool.
df -h | grep -E 'Filesystem|/dev/sda2' | tr -s ' ' ',' | \
jq -cnR '( input | split(",") ) as $keys | ( inputs | split(",") ) as $vals | [ [$keys, $vals] | transpose[] | {key:.[0],value:.[1]} ] | from_entries'
Expected output:
{"Filesystem":"/dev/sda2","Size":"40G","Used":"8.9G","Avail":"29G","Use%":"24%","Mounted":"/","on":null}
Conclusion
If you can’t get things done with a simple jq command, manipulating command outputs in Linux, using grep, awk, sed, tr, cut, and many more command-line utilities and tools, will definitely do the job. As always, there’s more than one way to skin a cat.
Feel free to leave a comment below and if you find this tutorial useful, follow our official channel on Telegram.