3

I'm currently working with some CSV datasets and was wondering whether there are any good (even better if standard) CLI ways of manipulating CSV data.

To be more precise, I mainly want to be able to:

  • vertically concatenate CSVs
  • SQL-style join CSVs on one or more columns
Kusalananda
  • 333,661
nikitautiu
  • 1,309

1 Answers1

12

Yes: CSVkit. http://csvkit.readthedocs.io/

CSV is not a standard that has anything to do with Unix, hence there are no "standard" (as in POSIX) utility for working with CSV files.

To vertically concatenate CSV files, use something like paste -d ',' file1.csv file2.csv (unless the CSV data contains newlines). CSVKit will handle most other kinds of operations (including queries and database loading, re-formatting etc.) on CSV files.

CSVKit is a collection of Python utilities. Among these is csvjoin, for example.

Kusalananda
  • 333,661
  • Note that ksh93 has built-in support to process csvs (not all flavours though). – Stéphane Chazelas Sep 12 '17 at 09:55
  • What about vertically concatenating columns within the same file? – ptrcao Dec 21 '19 at 09:10
  • 1
    @ptrcao That sounds like something you could do with a combination of cut and paste, but feel free to ask a new question about this. – Kusalananda Dec 21 '19 at 09:13
  • cut and paste are just shell commands or a part of CSVkit? – ptrcao Dec 21 '19 at 09:21
  • 1
    @ptrcao These are standard shell tools. As I don't know what your data is, I suggested using these. There is also a csvcut tool in CSVkit, and csvstack, but again I know nothing about your data and what it is you want to do. I suggest that you ask a brand new question about this rather than using the comment thread of other questions. – Kusalananda Dec 21 '19 at 09:43
  • @Kusalananda I have done as you advised: https://unix.stackexchange.com/questions/558347/concatenating-columns-of-the-same-csv-file-to-create-a-new-column-with-a-new-hea – ptrcao Dec 21 '19 at 09:57