Skip Navigation
Programming @programming.dev
BlackEco @lemmy.blackeco.com

DuckDB as the New jq

www.pgrs.net DuckDB as the New jq

Recently, I’ve been interested in the DuckDB project (like a SQLite geared towards data applications). And one of the amazing features is that it has many data importers included without requiring extra dependencies. This means it can natively read and parse JSON as a database table, among many othe...

8 comments

  • Love this. Always interesting to see novel ways of querying data in the terminal, and I agree that jq's syntax is difficult to remember.

    I actually prefer nu(shell) for this though. On the lobste.rs thread for this blog, a user shared this: 

     http get https://api.github.com/orgs/golang/repos
        
| get license.key -i
| uniq --count
| rename license

This outputs the following:

╭───┬──────────────┬───────╮
│ # │    license   │ count │
├───┼──────────────┼───────┤
│ 0 │ bsd-3-clause │    23 │
│ 1 │ apache-2.0   │     5 │
│ 2 │              │     2 │
╰───┴──────────────┴───────╯

    • Haha, I tried querying in Nushell before reading this and I was sure there was a better way. And indeed there is (especially since I was missing uniq). I'm still learning the available operators, but I enjoy the shell a lot, as well as its promised capabilities.

      • I really like Nushell. I would not run it as a daily driver currently, as it mostly doesn't win me over from Fish, feature-wise, but I love having it available for anything CLI date pipeline work I need to do.

  • While it doesn't provide an SQL interface, I've been using Nushell as my shell, which has native data operations.

    I tried querying the same, and I'm still not fluent (this was my third or fourth bigger/practical data querying), but it works well and fast too when you know the syntax: 

     nu
        
http get https://api.github.com/orgs/golang/repos | each {|x| get license} | get key | group-by --to-table | update items {|x| $x.items | length }

    I've used Nushell for reading en-mass json files, generating command json files for stuff saved in excel files (you can natively open those too), and most recently to query log files for specific information and usage analysis.

    /edit: This comment has the better nu solution.

8 comments