Questions tagged [csv]

Files containing data arrange in a table, often with commas (hence Comma Separated Values), to separate columns. Rows are separated by newlines (but not all newlines are row separators as fields can be quoted to contain the separator newlines. Use this tag for full-fledged CSV data not the simpler case of one record per line or completely unquoted (use csv-simple for that kind of data).

Files in CSV (comma-separted values) format contain tabular data, with rows separated by newlines and columns—normally—separated by a comma (,). Not every newline has to be a row separator.

There are many CSV variants (partly caused by language settings of the generating spreadsheet programs), so the separator might differ (semi-colon is common in CSVs generated by German software, tabs in others) and cell content might not be quoted ("...")if not necessary (if not containing the separator character nor any newlines).

Tools like awk and grep are only suitable for simplified CSV files without quotes and cells with newlines. Use for those files not .

For normal CSV files (quotes around fields, newlines and separator character in fields), use a proper CSV parser and tell it what quoting rules the file uses—see Is there a robust command line tool for processing csv files?

936 questions
5
votes
3 answers

Selecting rows in a CSV file based on column value

I have a CSV file with 4 columns: Itemname,Value,Description and component which is quite huge. I have to generate a template from the above CSV file that displays only the rows of the specified component(say component='abc' which is the search…
Blessy
  • 69
3
votes
2 answers

How do I truncate a csv file?

I have a csv file that is 6 gigabytes, but I don't need that much data, I need like 100 rows or so. How can I truncate it?
2
votes
2 answers

how to convert multiple txt to CSV with field data separated by blank lines

I have some data in multiple text files where fields are separated by blank lines. There are only 4 fields but in the second field there are more subfields, could be three or more. The first field is always a number, 0 or 1. 0 name_surname 1 yellow…
saiko
  • 21
2
votes
4 answers

I need a shell script to convert a csv file into pipe (|) delimited file by keeping the commas inside the enclosing quotes

Sample File (test.csv): "PRCD-15234","CDOC","12","JUN-20-2016 17:00:00","title, with commas, ","Y!##!" "PRCD-99999","CDOC","1","Sep-26-2016 17:00:00","title without comma","Y!##!" Output file: PRCD-15234|CDOC|12|JUN-20-2016 17:00:00|title, with…
Shanthi
  • 23
2
votes
4 answers

Addition of values in second columns if the first column entry is same in UNIX

I am trying to aggregate a file containing the following data in UNIX. I need to add the amounts if the key is…
Kshitij
  • 21
  • 1
1
vote
2 answers

Given a CSV file, how do I delete the content between the 2nd and 3rd tabs of each row?

I am using a mac and I have a CSV file delimited by tabs. I want to just remove all the content between the 2nd and 3rd tabs or replace it with something like "XXXX". Is there a command for this?
bigpotato
  • 275
  • 1
  • 3
  • 10
1
vote
2 answers

Getting file as rows and columns data matrix

I have gene.csv file which is about 1.3 GB in size and has got 300 columns and more than a million rows. it looks like following id1 id2 id3 id4 count1 count2 S1001 450 GAF ARHGAP18 1.56E-05 1483 S1001 450 …
David
  • 13
  • 3
1
vote
1 answer

Switch Columns in .csv files so that they are all the same

I have 20 excel files having 6x6 rows and columns, where the first row and column are string headers. Each of these files has 4 same columns and row headers except 1 different. I want to know how I can modify them so that the the columns and rows…
Tak
  • 529
0
votes
2 answers

How to cut a string CSV variable with unknown values count in bash, and loop over its values?

It might seem very easy, but I'm stuck at this. I have a string that is CSV and the length of items is unknown. "item1,item2,item3,..." I want to cut it, and loop over its items. I tried: while IFS=, read item; do echo $item done <<<…
0
votes
1 answer

Calculate mean and stdev between similar formatted csv files and output to another file following format

I have a group of output csv files from replicates of a simulation. Each file line in the file follow the same format: generation, number, value1, value2, .... valueX. (the file also include a header, same order. I would like to calculate the mean…
0
votes
2 answers

How to split single column to multiple column in CSV file

I have below text in csv file and need to place FILE and TIMESTAMP into separate columns to a csv file. Could you please let me know how i can achieve this. FILE,…
0
votes
2 answers

What can I do if delimiter of csv is appear in value?

Suppose, my csv file contains "item_name","price","description" mobile,500$,It has many features (e.g., camera, big display, etc) I want to load this csv data file in mysql database using mysql command like load data local infile 'file.csv'…
alhelal
  • 1,301
0
votes
3 answers

How to merge lines broken by newlines inside a double quoted field?

Imagine input is: KY,On,Ind ,Yes,1J5Z,KYEEI9,1/1/2016 Contract Code KY,On,Ind ,Yes,"1GH8 ",KYEEID,1/1/2016 Contract Code KY,On,Ind ,Yes,1J5Y,KYEEIJ,1/1/2016 Contract Code I would like to have the 3 lines(with newline as #### for example): KY,On,Ind…
0
votes
2 answers

I want to output certain rows froms a spreadsheet to a different file, if they contain a number which is one higher than in the cell above

I have a large csv file looking a little like this: SomeData,SomeData,1,SomeData SomeData,SomeData,1,SomeData SomeData,SomeData,2,SomeData SomeData,SomeData,3,SomeData SomeData,SomeData,1,SomeData SomeData,SomeData,1,SomeData …
0
votes
1 answer

short `csvgrep` equivalent as bash drop-in function

Question: Nice "drop-in" replacements to grep some lines of csv files containing some keywords in some column. Requirement: support more complex csv files - that have double quotes, newlines. Usecase: "csvkit" is awesome, but it might be too much to…
1
2