I found a way in Windows to do such thing
echo "This is just a sample line appended to create a big file. " > dummy.txt
for /L %i in (1,1,21) do type dummy.txt >> dummy.txt
http://www.windows-commandline.com/how-to-create-large-dummy-file/
Is there a way in UNIX to copy a file, append and then repeat the process?
Something like for .. cat file1.txt > file1.txt?
yes "Some text" | head -n 100000 > large-file
With csh/tcsh:
repeat 10000 echo some test > large-file
With zsh:
{repeat 10000 echo some test} > large-file
On GNU systems, see also:
seq 100000 > large-file
Or:
truncate -s 10T large-file
(creates a 10TiB sparse file (very large but doesn't take any space on disk)) and the other alternatives discussed at "Create a test file with lots of zero bytes".
Doing cat file >> file would be a bad idea.
First, it doesn't work with some cat implementations that refuse to read files that are the same as their output file. But even if you work around it by doing cat file | cat >> file, if file is larger than cat's internal buffer, that would cause cat to run in an infinite loop as it would end up reading the data that it has written earlier.
On file systems backed by a rotational hard drive, it would be pretty inefficient as well (after reaching a size greater than would possibly be cached in memory) as the drive would need to go back and forth between the location where to read the data, and that where to write it.
dd if=/dev/zero is the least efficient.
– Stéphane Chazelas
Mar 12 '16 at 07:10
dd seek=xxx ones, so less lazy in terms of human effort and also a lot less lazy in terms of computer effort.
– Stéphane Chazelas
Mar 12 '16 at 13:47
dd, and it is an important tool to know if you find yourself doing cli admin tasks. /dev/zero might be relatively slow but you can always use a form like yes|dd of=... if you prefer. answers that do not use dd don't reinforce the memory of essential admin tools. — again, I want to reiterate, and this is visible in my initial comment, I don't think you are wrong, neither in your answers nor in the comment that dd if=/dev/zero can be slow. I was just talking about what is common
– roberto tomás
Mar 12 '16 at 15:03
dd, but I have never understood why. In fact, I think I've only ever used it to read an MBR or similar fringe tasks. In my experience, other tools are faster, simpler and safer for the vast majority of cases where people use dd. I think this is one of those cases where common != optimal, like sudo su or cat file | grep foo.
– terdon
Mar 12 '16 at 17:48
head -c 1048576 /dev/zero > file
– Random832
Mar 13 '16 at 05:19
You can create a large file on Solaris using:
mkfile 10g /path/to/file
Another way which works on Solaris (and Linux):
truncate -s 10g /path/to file
It is also possible to use:
dd if=/dev/zero of=/path/to/file bs=1048576 count=10240
The fastest way possible to create a big file in a Linux system is fallocate:
sudo fallocate -l 2G bigfile
fallocate manipulates the files system, and does not actually writes to the data sectors by default, and as such is extremely fast. The downside it is that it has to be run as root.
Running it successively in a loop, you can fill the biggest of filesystems in a matter of seconds.
From man fallocate
fallocate is used to manipulate the allocated disk space for a file, either to deallocate or preallocate it.
For filesystems which support the fallocate system call, preallocation is done quickly by allocating blocks and marking them as uninitialized, requiring no IO to the data blocks. This is much faster than creating a file by filling it with zeros.
Supported for XFS (since Linux 2.6.38), ext4 (since Linux 3.0), Btrfs (since Linux 3.7) and tmpfs (since Linux 3.5).
This will keep going until you CTRL-C:
yes This is stuff that I want to put into my file... >> dummy.txt
Be careful though, because you can get a hundreds of thousands of lines/second...
From man yes:
yes - output a string repeatedly until killed
yes $BIG_STRING | head -c $TARGET_SIZE >> dummy.txt would let you get precise amount. (-n $TARGET_NUMBER_OF_LINES). yes would automatically die as result of a 'broken pipe' when head terminates because the target number has been reached.
– PypeBros
Jul 18 '19 at 07:46
If I understand you correctly, you are looking for something like:
echo "test line" > file;
for i in {1..21}; do echo "test line" >> file; done
That will create a file with 22 repetitions of "test line". If you want a specific file size, you can use something like this (on Linux). 1024 is one kilobyte:
while [ $(stat -c "%s" file) -le 1024 ]; do echo "test line" >> file; done
Personally, when I want to create a large file, I use two files and cat one into the other. You can repeat the process until you reach the desired size (1MB here):
echo "test line" > file;
while [ $(stat -c "%s" file) -le 1048576 ]; do
cat file >> newfile
cat newfile >> file
done
Note that this solution will often exceed the desired size because if the file is under the limit, everything will be catted into it again.
Finally, if all you want is a file of the desired size and don't need it to actually contain anything, you ca use truncate:
truncate -s 1M file
cating the file actually have any advantage to just appending though ? It would seem as though it would take longer as it has to fork two processes every loop and also move the entire contents multiple times.
– 123
Mar 11 '16 at 15:14
cat approach is much, much faster. It only makes sense for creating huge files but that created a 545M file in 10 seconds on my machine. The same while loop with echo "test line" >> file created a 96K file in the same amount of time.
– terdon
Mar 11 '16 at 17:15
truncate around, you can truncate -s 1G to create the file in first place. https://unix.stackexchange.com/a/269184/85549. You could replace it by a head -c $DESIRED_SIZE, possibly within the while loop.
– PypeBros
Jul 18 '19 at 07:43
By piping the contents of /dev/urandom to head you can redirect the output to a file, so :
cat /dev/urandom | head --bytes=100 >> foo.bar
Will give you a file with 100 bytes of garbage.
echo "This is just a sample line appended to create a big file. " > dummy.txt
i=1
while [ $i -le 21 ]
do
cat dummy.txt >> bigfile
cat bigfile > dummy.txt
(( i++ ))
done
same effect of your windows script but in bash, you can not concatenate a file to itself, directly.
.txt extension, you're leaving 2 big files at the end.
– ott--
Mar 12 '16 at 13:49
for i in {1..1000000};do echo "string" >> file;donein bash. – 123 Mar 11 '16 at 15:00type file >> fileto run in an infinite loop (at least as soon as it's sufficiently large that it doesn't fit in a buffer). – Stéphane Chazelas Mar 11 '16 at 15:14typeis going to load the contents offileinto memory, then openfilefor writing and append the loaded contents, then exit. – cat Mar 11 '16 at 22:41typewould just run out of memory, because NTFS and the NT kernel lock any file that's opened for reading / writing, unlike POSIX where there are no implicit locks. – cat Mar 12 '16 at 13:41seek-esque syscalls in NT, so you have no choice but tomallocthe length of the file at once. – cat Mar 12 '16 at 13:44