8

I routinely need to completely erase SD cards.

This process is greatly speed up using SD Memory Card Formatter.

Unfortunately said program seems to be unavailable under Linux and alternatives (e.g.: dd or shred) take a LOOOONG time.

AFAIK SD/MMC cards have specific commands to mass-erase and I suspect that's what used by "SD Memory Card Formatter" (as opposed to trying to erase using their content by overwriting).

Is there some tool using the same?

ZioByte
  • 870

6 Answers6

10

Use blkdiscard

In short:

blkdiscard /dev/mmcblk0

Discussion

To quickly erase an entire SD card, you can use the blkdiscard(8) command. This calls the Linux BLKDISCARD ioctl, which in turn passes CMD38, the same as SD Memory Card Formatter.

Interesting options

  • -s, --secure securely discard blocks. As has been pointed out elsewhere, a normal CMD38 will make some blocks appear empty, but leave others plainly visible due to garbage collection. Use -s if you wish to be a little more secure and force garbage collected blocks to also be erased.

  • -z, --zero zero-fill blocks rather than discard.

Does not work over USB

Note that while this works on my laptop and Raspberry Pi, it would not work on a USB SD card reader. The difference is that kernel needs to have access to the low-level MMC subsystem, which USB abstracts away as a generic "mass storage" device. [I believe Microsoft Windows has the same limitation, but please correct me in the comments if I'm wrong.]

If you do not have a device that can speak directly to the low-level SD card, it is possible to build one out of an Arduino and run SDFormatter.ino.

Extraordinarily dangerous!

Since blkdiscard must be run as root, it can easily destroy all of your data. You better be sure that you are pointing it at the right device and then double-check again. Use lsblk and df.

The manpage claims that, to be safe, it will only work if the drive isn't already mounted. That would be a nice safety feature, but it is not true as of this writing. The current implementation requires the -f, --force option if the drive is already formatted, which is (almost) always going to be the case since SD cards come pre-formatted. Unfortunately, using --force means that all checks, including if the drive is in use, are disabled.

Hopefully someday Linux will allow blkdiscard to be run on removable devices without requiring root privileges which would make it much less dangerous.

Other uses

Besides quickly wiping out data for privacy, supposedly blkdiscard, similar to fstrim for an SSD, will improve wear-leveling and make some SD cards generally run a little bit faster.

hackerb9
  • 1,569
  • For SSDs, is there a difference between blkdiscard and hdparm --user-master u --security-set-pass password /dev/sdX; hdparm --user-master u --security-erase-enhanced password /dev/sdX? – anick Jul 08 '22 at 09:00
  • 2
    I don't know if hdparm uses BLKSECDISCARD for security erase on SSDs, but I suspect not since that is an ATA command. Of course internally in the drive it may be implemented as the same thing, but that's beyond the scope of this question. – hackerb9 Jul 08 '22 at 19:14
4

Since recently SD Memory Card Formatter (CLI version) is available for Linux (for x86_64 and arm64):
https://www.sdcard.org/downloads/sd-memory-card-formatter-for-linux/

I needed to use the official SD Memory Card Formatter some time ago, preferably under Linux (as part of a toolchain), so I emailed the SD Association about it. They are very responsive and helpful. And recently they emailed me back with the information that the formatter is now available for Linux.

Piotr Henryk Dabrowski
  • 141
  • Thanks for being the squeaky wheel that prompted them to release a tool! And thanks for taking the time to share their response with StackExchange. If you're up for it, I have a few suggestions for how you might make the answer even better. For example: How does SD Memory Card Formatter differ from LInux's blkdiscard? Is it under an open license? Also, you only mention x86_64 and arm64. If it doesn't work on arm32, you might want to mention that since most Raspberry Pis currently run in 32-bit mode even when they have 64-bit processors. – hackerb9 Apr 22 '23 at 17:37
3

Just to make the point of the reporter: https://forum.armbian.com/topic/3776-the-partition-is-not-resized-to-full-sd-card-size/?do=findComment&comment=27413

tkaiser

Posted March 13, 2017

Regarding 'reserve sectors' and overprovisioning: If a card claims it's n bytes in capacity it has internally a larger capacity. This is used as reserve (if the controller detects bad sectors, then reserve sectors are mapped in) and to allow somewhat ok-ish write performance when the card gets full. On flash media you can't overwrite directly, it's always a very time consuming read/delete/write cycle, the number a flash cell can be written to is determined by the count of program/erase (P/E) cycles it is designed for, the controller has to take care of this so that all flash cells wear out equally (wear leveling).

Since the controller has no idea which sectors contain real data and which not (there's no TRIM support for SD cards) as soon as you completely fill the card once (all space partitioned) from now on the controller considers every sector containing useful data (even if you deleted the data in the meantime -- since there's no TRIM support the controller doesn't know what's empty or not, from now on the whole capacity is considered in use). Now only the 'reserve sectors' are available to perform read/delete/write cycles and if this amount of sectors is small things slow down a lot on average SD cards (not those more recent Samsung).

Just check articles explaining SSDs and keep in mind that SD cards behave like crappy SSDs from a decade ago containing slow/primitive controllers and do not support TRIM.

BTW: This is the only great use case for SD Association's 'SD Formatter'. This tool is used to format SD cards appropriately (partitions it while choosing the 'correct' file system which is either FAT or exFAT) which obviously is pretty useless from Armbian's perspective since burning an OS image as next step both overwrites the partition table and the filesystems present before. So why using SD Formatter in the first place? Since this tool implements ERASE CMD38. It tells the card's controller that every sector/block of the card does not contain any real data any more and can be considered empty. On 'the average' SD card this also might restore horribly low performance back to 'factory default' performance. But more recent SD card controllers especially when paired with many reserve sectors aren't that much affected.

(...)

Edit: A small note regarding 'SD cards don't support TRIM'. The SD protocol defines a block erase command and tools like fstrim are supposed to do the job. Whether your kernel + SD card combination supports that or not a simple 'sudo fstrim -v /' might tell. Whether this has the desired effect or not is a different question though (see this attempt to test this -- I'm not sure whether the method is sufficient since the point of marking data segments as already erased should not involve overwriting them, it's just that the SD card's controller knows that specific sectors/pages can be added to the wear level pool since marked 'emtpy' now)

So there is no Linux equivalent to sdcard.org SD Formatter which is only available for windows and mac.

And fstrim or dd are not replacements. I do not know why we miss such a tool. Likely with improved overprovisioning and better controllers this will not be a huge problem in the long term. Maybe ask Tuxera which made SD Formatter if they plan for an Android or plain linux version.

Alban Browaeys
  • 61
2

If you use dd correctly it's extremely fast:

There is no reason to overwrite the whole card, that will always take a long time, no matter what software you use. You just have to remove the first sectors (partition table):

dd if=/dev/zero of=/dev/locationofcard bs=1024 count=1000

A way lower count will also do the job, but this should already be fast enough (seconds).

I am using 1000 as 'safe' value because i don't know by heart how low you can get.

Garo
  • 2,059
  • 11
  • 16
  • No. I need to be sure the WHOLE card is zero-filled. – ZioByte Jul 23 '20 at 13:18
  • In that case there is nothing faster then dd if=/dev/zero of=/dev/locationofcard (without options), if there is software that looks faster then it's because it is skipping parts. But why would you need to zero-fill the whole card ? (The only thing i can think of is security and in that case you should do multiple overwrites with random data instead of 1 overwrite with zero's) – Garo Jul 23 '20 at 19:22
  • 4
    I beg to differ. SDcards are "intelligent" devices and many have "erase yourself" (or, to be more precise, "discard all sectors") commands that work essentially in parallel on all sectors at once. I would even know how to do it, but that would mean to go to DD level, which I would rather avoid, if possible. Reason why I need this erasure is we use SD for special purpose and software will "manage" all non-zero sectors, even if they do not contain meaningful data, do "empty" cards should be all zeroed (strange things happen in the embedded world ;) ) – ZioByte Jul 23 '20 at 21:15
  • 2
    @Garo: "nothing faster than dd without options"? By default it uses a tiny block size of only 512 bytes. That's extremely slow for most if not all media, in particular SD Cards. Nowadays one should always use at least 4K as a bare minimum, preferably 1M. – MestreLion May 12 '22 at 06:41
  • If the partition table is GPT, there's a second one at the end of the block device. – anick Jul 08 '22 at 08:56
1

SD cards do not need to be formatted. "Formatting" is an old term specific for magnetic disks, where you prepare the tracks and sectors for actual data (they are not defined by disk, but by the head when you format the disk. So formatting is very slow.

Now "formatting" is not more such slow process, so it is not really formatting.

Often programs will just do a mkfs style formatting, so just creating the structure of file systems. Where to put data is already defined by card and internal controller.

Unfortunately I'm not find the command (given by controller) just to tell disk that sectors are not used. This may help, but possibly it is part of mkfs (hidden on kernel side). But do not do unnecessarily dd but on real magnetic harddisk: on SD, SSD, etc. they just consume the device.

Giacomo Catenazzi
  • 2,846
  • 1
    Again: I am aware of how the whole system works. I need to be SURE the WHOLE card is zero-filled, even "not used" sectors. – ZioByte Jul 23 '20 at 13:19
  • 2
    The "great" speed it seems that they use exactly the same trick.They told the card to DISCARD sectors. so you will read them as zero. But the sectors may be restored in appropriate facilities. BTW also if you write zero the usual facilities to restore data can get most of original data. [so: never gives other people cards where you put confidential information] – Giacomo Catenazzi Jul 23 '20 at 13:27
  • That is ok with me. I do not need it for security reasons, but because some tools (i have no way to change) will process "all non zero sectors", regardless of they being actually linked in a filesystem. So I don't care if content could be restored "in appropriate facilities", it's enough that raw read return zeroed sectors. Do You have news of such a tool running on Linux? ... and, if you can, could you tell me where You gleamed info about that closed tool? – ZioByte Jul 23 '20 at 15:07
  • It is not a closed tool. It is in protocol. https://wiki.archlinux.org/index.php/Solid_state_drive has a lot of information. Many SD cards are similar (especially the newest/fast ones) – Giacomo Catenazzi Jul 23 '20 at 15:15
  • 1
    Uhm... I wouldn't be so sure. Their EULA states clearly You are not supposed to try to reverse engineer from binary to source. Thanks for the pointer! ... but it seems to deal with SSD (Solid State Disk), which is a bit different from SD (Secure Digital) – ZioByte Jul 23 '20 at 16:05
  • It is not reverse engineer. We are not looking the internal of a card. It is just the standard protocol used to talk with the cards. It is standard (with many extensions). Write here, trim data, fetch data, get capabilities, etc: card must works on many systems, so there is a standard. As I wrote, modern SD are similar to old SSD, but look: I found something interesting for you: https://superuser.com/questions/18906/can-i-use-something-like-the-ssd-trim-command-on-a-sd-card – Giacomo Catenazzi Jul 24 '20 at 05:22
0

A note about SD standard compliance: "SD Specifications Part 2: File System Specification" v3.00 ("SD.2"; try Google or libgen) has special restrictions on the SD card partitioning and formatting structure. The official formatter will obey it, but you might not. Some important points are:

  • The standard is MBR, one partition of type "F8" only. There is no telling what happens if you use anything else. The worst-case scenario is a Samsung SSD-like GPT corruption issue. Best-case? Some speed reduction for the second partition's FAT.
  • The standard wants (Informative Appendix C; "should" elsewhere) you to position the partition to align the FAT area with a number of Boundary Units (~physical page), so that this special page can be optimized for frequent small I/O but also doesn't get filled by files.
    • For FAT12/16/32, you need to partition correctly since there's no FAT offset mechanism. For 12/16, SD.2 wants you to squeeze the FATs into the first BU with with the MBR and everything. For 32, SD.2 wants you to put the partition start at the beginning of the second BU.

      mkfs.fat has no knowledge of a BU. It does align to cluster (~fs block) size, which makes sense for FAT32 but not necessarily the smaller formats. Manual alignment is possible with reserve sectors, but very tedious.

    • For ExFAT, you technically have the same requirement at BU, but the filesystem itself lets you shift the FAT around, so getting it wrong is less severe of an issue.

      mkfs.exfat reads the position of the partition relative to disk start and makes use of this information to calculate a good offset.

The easiest to format in a standard-recommended way is ExFAT, because the formatting tool is smart enough: if you get the BU size and cluster size right (the default differs from SD.2 recommendation, but you can read its manpage for SD.2 recs), you are on the right path. Partition to BU offset for pedantry, or don't, because it won't matter too much anyways.

As for the legacy FAT formats, good luck: you will need to calculate the FAT size and then translate that into either the reserve sector count (FAT32) or the partition offset (FAT12/16). This could be a feature request to the dosfstools/fatprogs people, with very dubious usefulness.

Mingye Wang
  • 1,181