2

I have to do a long task (convert image format) for every file of a folder. I achieved to use pv to write the estimation of duration with (used sleep here to simulated processing time):

pv -B 1 =(find . -iwholename '*.png') | xargs -l sh -c 'sleep 0.1'

I was thinking like there may be an internal buffer which can mess up the estimation, because the list of files itself is not very large (tens of KB) and there may be internal buffer somewhere like in pv, so I limited the buffer to 1 byte with -B 1. And I wonder if that could cause performance issues (because disk can only read by a minimum of 512 bytes per read, and discard 511 bytes), but I don't know if maybe it resides it memory because =() is used. I feel like -B 1 is a bad solution.

But more importantly, there are strange behaviour with others values for -B. For example :

pv =(yes | head -n 100) -B 1 | xargs -l sh -c 'sleep 1'

Will work as expected (although starting strangely around 17% and starting rendering only after a few seconds), but

pv =(yes | head -n 100) -B 10 | xargs -l sh -c 'sleep 1'

will print immediately 100% even if I thought it should take 100 seconds to process, updating at interval of 5%, because 200 bytes are printed, x100 characters y and \n. Why ?

Note : =() requires zsh. I tried with <() and the progress bar doesn't work. Maybe pv can't estimate the fifo size, but it can with a file?

rafoo
  • 121
  • The output from pv is often misleading, when writing via a buffer. When the process has finished writing to the buffer, it has finished according to pv, but there might be a slow background process to write from the buffer to the target, for example the memory cells of a USB drive. One way to manage that is to watch flushing the 'dirty' data from the buffer after the sync command. I use this method in mkusb, and you can grab the shellscript watch-flush from there. See also this link. – sudodus Aug 02 '22 at 10:36
  • The version of watch-flush in mkusb is updated (improved) compared to the version in the link from my previous comment. – sudodus Aug 02 '22 at 10:45

1 Answers1

1

To convert image format for a lot of files, you can let pv count the files converted. This is not perfect, because some files take longer time that other files, but it can provide a fair estimate.

Start by counting the total amount of files, something like the following command (my pictures from June this year),

$ find 2022/06 \( -name '*.jpg' -or -name '*.png' \)|wc -l
55

Then create a command line, that can take advantage of pv -l -s55, counting lines. If your converting process does not produce any output, you can add an echo statement (feh shows pictures but produces no output to the calling terminal window, so I let echo print only a line feed).

$ find 2022/06 -type f -exec bash -c "feh -r -g 1000x1000 --zoom max {} & sleep .5; pkill feh ;sleep .5;echo ''" \; | pv -l -s55 > /dev/null
22,0  0:00:23 [ 978m/s] [==============>                       ] 40% ETA 0:00:34

or if you want to see the output scroll, don't redirect the output,

$ find 2022/06 -type f -exec bash -c "feh -r -g 1000x1000 --zoom max {} & sleep .5; pkill feh ;sleep .5;echo ''" \; | pv -l -s55
0,00  0:00:01 [0,00 /s] [>                                     ]  0% ETA 0:00:00
1,00  0:00:02 [1,01 /s] [>                                     ]  1% ETA 0:01:48
2,00  0:00:03 [ 984m/s] [>                                     ]  3% ETA 0:01:19
3,00  0:00:04 [ 978m/s] [>                                     ]  5% ETA 0:01:09
4,00  0:00:05 [ 983m/s] [=>                                    ]  7% ETA 0:01:03
5,00  0:00:06 [ 984m/s] [==>                                   ]  9% ETA 0:01:00
6,00  0:00:07 [1,08 /s] [==>                                   ] 10% ETA 0:00:57
7,00  0:00:08 [ 980m/s] [===>                                  ] 12% ETA 0:00:54
8,00  0:00:09 [ 980m/s] [====>                                 ] 14% ETA 0:00:52
9,00  0:00:10 [ 979m/s] [=====>                                ] 16% ETA 0:00:51
10,0  0:00:11 [ 983m/s] [=====>                                ] 18% ETA 0:00:49
11,0  0:00:12 [1,08 /s] [======>                               ] 20% ETA 0:00:48
12,0  0:00:13 [ 982m/s] [======>                               ] 21% ETA 0:00:46
13,0  0:00:14 [ 982m/s] [=======>                              ] 23% ETA 0:00:45
14,0  0:00:15 [ 981m/s] [========>                             ] 25% ETA 0:00:43
15,0  0:00:16 [ 986m/s] [=========>                            ] 27% ETA 0:00:42
16,0  0:00:17 [1,08 /s] [==========>                           ] 29% ETA 0:00:41
17,0  0:00:18 [ 983m/s] [==========>                           ] 30% ETA 0:00:40
18,0  0:00:19 [ 984m/s] [===========>                          ] 32% ETA 0:00:39
19,0  0:00:20 [ 982m/s] [===========>                          ] 34% ETA 0:00:37
20,0  0:00:21 [ 981m/s] [============>                         ] 36% ETA 0:00:36
21,0  0:00:22 [1,08 /s] [=============>                        ] 38% ETA 0:00:35
22,0  0:00:23 [ 985m/s] [==============>                       ] 40% ETA 0:00:34
23,0  0:00:24 [ 983m/s] [==============>                       ] 41% ETA 0:00:33
24,0  0:00:25 [ 981m/s] [===============>                      ] 43% ETA 0:00:32
25,0  0:00:26 [ 986m/s] [================>                     ] 45% ETA 0:00:31
26,0  0:00:27 [ 985m/s] [================>                     ] 47% ETA 0:00:30
27,0  0:00:28 [1,08 /s] [=================>                    ] 49% ETA 0:00:29
28,0  0:00:29 [ 983m/s] [==================>                   ] 50% ETA 0:00:27
29,0  0:00:30 [ 983m/s] [==================>                   ] 52% ETA 0:00:26
30,0  0:00:31 [ 983m/s] [===================>                  ] 54% ETA 0:00:25
31,0  0:00:32 [ 979m/s] [====================>                 ] 56% ETA 0:00:24
32,0  0:00:33 [1,08 /s] [=====================>                ] 58% ETA 0:00:23
33,0  0:00:34 [ 982m/s] [=====================>                ] 60% ETA 0:00:22
34,0  0:00:35 [ 981m/s] [======================>               ] 61% ETA 0:00:21
35,0  0:00:36 [ 980m/s] [======================>               ] 63% ETA 0:00:20
36,0  0:00:37 [ 983m/s] [=======================>              ] 65% ETA 0:00:19
37,0  0:00:38 [1,08 /s] [========================>             ] 67% ETA 0:00:18
38,0  0:00:39 [ 984m/s] [=========================>            ] 69% ETA 0:00:17
39,0  0:00:40 [ 983m/s] [=========================>            ] 70% ETA 0:00:16
40,0  0:00:41 [ 978m/s] [==========================>           ] 72% ETA 0:00:15
41,0  0:00:42 [ 979m/s] [===========================>          ] 74% ETA 0:00:14
42,0  0:00:43 [1,08 /s] [===========================>          ] 76% ETA 0:00:13
43,0  0:00:44 [ 981m/s] [============================>         ] 78% ETA 0:00:12
44,0  0:00:45 [ 981m/s] [=============================>        ] 80% ETA 0:00:11
45,0  0:00:46 [ 985m/s] [=============================>        ] 81% ETA 0:00:10
46,0  0:00:47 [ 983m/s] [==============================>       ] 83% ETA 0:00:09
47,0  0:00:48 [1,08 /s] [===============================>      ] 85% ETA 0:00:08
48,0  0:00:49 [ 983m/s] [================================>     ] 87% ETA 0:00:07
49,0  0:00:50 [ 981m/s] [================================>     ] 89% ETA 0:00:06
50,0  0:00:51 [ 986m/s] [=================================>    ] 90% ETA 0:00:05
51,0  0:00:52 [ 981m/s] [=================================>    ] 92% ETA 0:00:04
52,0  0:00:53 [1,08 /s] [==================================>   ] 94% ETA 0:00:03
53,0  0:00:54 [ 982m/s] [===================================>  ] 96% ETA 0:00:02
54,0  0:00:55 [ 984m/s] [====================================> ] 98% ETA 0:00:01
55,0  0:00:55 [ 982m/s] [====================================>] 100%
sudodus
  • 6,421