Is It OK to use oflag=direct and conv=fsync together in dd?

Question

My target device is a hard drive with 4096 bs. I want to bypass caches and write to storage directly or as soon as possible. Speed is not my primary concern: 30 hours is too much, but the difference between 4 and 7 hours is insignificant.

---

What I understand (might not be 100% correct):

conv=fsync will be executed only once at the end of dd call.

But I don't want to postpone sync until the end. I want the data written to storage as soon as possible. There two other options: oflag=direct and oflag=sync. I don't like oflag=sync, because (1) it's extremely slow when I test with bs=4096, and (2) it still uses memory cache -- unnecessarily, I think.

oflag=direct bypasses the kernel's page cache (memory cache), writing directly to the storage. But the storage may itself store the data in a write-back (hdd) cache, so conv=fsync will still be required to write hdd cache to the actual storage.

Thus, I hope it is permissible -- I wonder whether it is ideal -- to use the two arguments together, maybe like this:

dd if=/dev/zero of=/dev/sdX bs=4096 status=progress oflag=direct conv=fsync

Answer 1

It sounds like your question is about speed. You are already using conv=fsync, so I see no question of safety or "correctness". Unless you have hardware with a specific bug, where you would need to tell us exactly what your bug looks like.

The general rule for performance is to test your own situation, and not worry too much about small differences. dd tells you the speed. You can test different options using a small count=. E.g. you can use bs=4k count=100k to test writing 400M.

---

You are asking about a relatively simple case. /dev/zero will be much faster than your device, so I ignore read performance.

Using oflag=sync will wait for all temporary caches to empty after submitting each block of data, losing the benefit of always keeping data in-flight. Therefore you would need to specify a larger block size to get the best speed, e.g. bs=16M.

In principle, you want to feed the device up to two requests at a time, so that it always had at least one request to work on. Specifically on mechanical drives, if you let the feed run dry, you will have to wait for a full rotation before your next request can be written in the right place. dd itself does not do anything to ensure this. It relies on writeback cache in the kernel, or in the device.

oflag=direct is a useful in-between option. If you have a problem with kernel cache (see below), it is a great way to bypass that cache. Many devices include a writeback cache of their own, so oflag=direct can be faster than oflag=sync for the same block size.

The kernel cache is intended to work well, without slowing down IO access to other devices. E.g. your system drive that you are using at the same time :-). But this problem does happen sometimes, and people complain about it. So it might depend whether you are expecting or worried about such a problem :-).

If you want to try both options at once, you specify this as oflag=direct,sync.

Even without oflag=sync, increasing the block size by a few more times might reduce CPU usage. E.g. bs=16k or bs=1M. But bs=4k / bs=4096 is already fairly good, much better than the ancient default bs=512.

Answer 2

I wouldn't bother - dd input is so slow that you should maximise any cache usage, since output is incredibly slow. Using a large blocksize (in an effort to use more cache) is the best you can do.

With regard to @sgon00's specific question, the dd manpage says:-

fdatasync     physically write output file data before finishing            
fsync         likewise, but also write metadata
oflag=FLAGS   write as per the comma separated symbol list...

oflag=nocache Request to drop cache. This option at least mentions cache. Doesn't necessarily mean that it will be granted though. This coild be combined with the 'direct' option.

dd is an old program and so its many possible parameters may not be interpreted exactly as they were originally intended.