cat and split a file and upload to hdfs?

Question

I am trying to achieve this scenario where I can split a gz file which is on local linux and upload to hdfs as part unzipped or zipped files without writing the split output to disk. I am having issues after trying this below command.

below command writes to local disk then I can upload to hdfs which I don't want :-

zcat ./file.txt.gz | tail  -n +2 | split -l 20 - file.part 


hdfs dfs -copyFromLocal ./*file.part* /folder/in/hdfs/

I want something like this is it achievable ?:-

zcat ./file.txt.gz | tail  -n +2 | split -l 20 | gzip -d | hdfs dfs -put - /folder/in/hdfs/file.part

See How do I break up a file like split to stdout for piping to a command? — don_crissti, Oct 07 '17 at 16:47

score 0 · Accepted Answer · answered Oct 07 '17 at 16:29

0

You can avoid split and do the splitting on your own:

number_of_files=5 # for you to determine
zcat ./file.txt.gz | for((i=0;i<5;i++)); do
    head -n 20 | hdfs dfs -put - /folder/in/hdfs/file.part_$i
done

answered Oct 07 '17 at 16:29

Hauke Laging

90,279

cat and split a file and upload to hdfs?

1 Answers1