Formatting ls -l Output into pipe delimited file

Question

I am fairly new to Unix and could use some assistance. When I execute the ls -l command (actually ls -alR) and pipe the output into a file, there are spaces used in separating the ls fields returned. What I am looking to achieve is essentially separating the ls fields by say a Tab or a pipe character (either should work). I began looking at the pr command and replaced the spaces with Tabs, but that did not work as there are varying number of spaces in the ls output, which resulted in erroneous fields when importing this file.

Answer 1

If you going to parse this output later on, I would discourage it. See this page for a good description of why. In any case it's going to be overly difficult to get the information you want using ls, especially if there are spaces or other special characters in filenames. A better option if you just want to build a csv or something is to use the -printf option to find. This command should get you started:

find . -printf "%M|%n|%u|%s|%P\n" > output

Note that this will recurse from the starting directory like your ls command. It dumps the permissions, link count, owner, group, size, and filename to a file with fields separated by pipes. Check out the man page for find for more info.

Answer 2

Instead of jumping through hoops to parse ls, you could instead emulate ls -l with the GNU coreutils stat command, using a format of your choosing. If you layer on a bash shell for the globstar option to recursively list files and the dotglob option to list dot-files, then you can get pretty reasonable results. For the file modification times, I've chosen to use %Y, which is "seconds since the epoch"; you could use %y, or even pass the %Y value to date to reformat it however you choose. Using seconds-since-the-epoch is a pretty flexible way to let the down-stream processor display it however it likes.

For a simple file:

$ stat -c "%A|%h|%U|%G|%s|%Y|%n" hosts.txt
-rw-r--r--|1|username|pgrp|135387|1202230643|hosts.txt

To emulate ls -alR:

$ shopt -s globstar dotglob
$ stat -c "%A|%h|%U|%G|%s|%Y|%n" **
drwxr-xr-x|2|username|pgrp|4096|1510073497|.dotdir
-rw-r--r--|1|username|pgrp|0|1510073497|.dotdir/.dotfile
-rw-r--r--|1|username|pgrp|0|1507213776|f/test case
-rw-r--r--|1|username|pgrp|0|1507213776|f/test case.csv
-rw-r--r--|1|username|pgrp|135387|1202230643|hosts.txt
drwxr-xr-x|2|username|pgrp|4096|1510073307|space dir
-rw-r--r--|1|username|pgrp|0|1510073307|space dir/space file
-rwxr-xr-x|1|username|pgrp|112|1510073163|stat.sh

While this may be less portable (to systems that don't have coreutils / stat), it does have the benefit of being flexible with printing only the fields that you're interested in, and doesn't risk replacing spaces in filenames.

Answer 3

simply add following:

ls -alR |  sed -r s'/\s+/|/g'

this should work for any command whose output you would like to modify. \s stands for "space" the "+" stands for one or more occurences (so it would match tab" then the "|" is the symbol which we'd use for the new separator. Please bare in mind that there are some special characters which should be escaped with "\". if you like tabs, here it is:

 ls -alR |  sed -r s'/\s+/\t/g'