34

I have a 22TB disk on /dev/sdb. How do I create a 22TB partition? Don't really care about the file system - ext4 or zfs is fine.

Running CentOS 6.2 - Partition will be used as a data dump. Only a single stream of data so being picky/choosy over what file system isn't really a concern right now. The disk is formed from 12x2TB nearline SAS drives and a Dell Perc controller.

I just want a 22TB partition.

rav_kr
  • 111
LVLAaron
  • 1,735
  • 1
    a few more pieces of information would be helpful. what OS are you using? what kind of performance do you expect or need to get from it? what's the underlying hardware? it is already a raid set? does it have redundancy? do you need the extra data protection features of zfs? – Tim Kennedy Jan 14 '12 at 01:46
  • 3
    Which tries have failed so far? fdisk, cfdisk? – Nils Jan 14 '12 at 20:39

7 Answers7

45

The simplest solution is to use GPT partitioning, a 64-bit version of Linux, and XFS:

  • GPT is necessary because the MS-DOS-style MBR partition table created by fdisk is limited to 2 TiB disks. So, you need to use parted or another GPT-aware partitioning program instead of fdisk. (gdisk, gparted, etc.)

  • A 64-bit kernel is necessary because 32-bit kernels limit you to filesystems smaller than you're asking for. You either hit a size limit based on 32-bit integers or end up not being able to address enough RAM to support the filesystem properly.

  • XFS is not the only solution, but in my opinion it is the easiest one for RHEL systems.

    You cannot use ext4 for this in RHEL 6. Although the filesystem was designed to support 1 EiB filesystems, there is an artificial 16 TiB volume size limit in the version of e2fsprogs included in RHEL 6 and its derivatives. Both Red Hat and CentOS call this out in their docs. (The ext4 16 TiB limit was raised considerably in RHEL 7 to 50 TiB.)

    ZFS may not be practical in your situation. Because of its several legal and technical restrictions, I can't outright recommend it unless you need something only ZFS gives you.

    Having ruled out your two chosen filesystems, I suggest XFS. It is the default filesystem in RHEL 7, it was available as a supported filesystem in all RHEL 6 versions, and was backported to the later RHEL 5 releases after RHEL 6 came out.

Here's the process:

  1. Check whether you have mkfs.xfs installed by running it without arguments. If it's not present, install the userland XFS tools:

    # yum install xfsprogs

    If that failed, it's probably because you're on an older OS that doesn't have this in its default package repository. You really should upgrade, but if that is impossible, you can get this from CentOSPlus or EPEL. You may also need to install the kmod_xfs package.

  2. Create the partition:

    Since you say your 22 TiB volume is on /dev/sdb, the commands for parted are:

    # parted /dev/sdb mklabel gpt
# parted -a optimal -- /dev/sdb mkpart primary xfs 1 -1

    That causes it to take over the entire volume with a single partition. Actually, it ignores the first 1 MiB of the volume, to achieve the 4 KiB alignment required to get the full performance from Advanced Format HDDs and SSDs.

    You could skip this step and format the entire volume with XFS. That is, you would use /dev/sdb in the example below instead of /dev/sdb1. This avoids the problem of sector alignment. In the case of a volume that only your Linux-based OS will see, there are no downsides worth speaking about, but I'd caution against doing this on a removable volume or on an internal volume in a multi-booting computer, since some OSes (Windows and macOS, for instance) will offer to format a partitionless hard drive for you every time it appears. Putting the filesystem on a partition solves this.

  3. Format the partition:

    # mkfs.xfs -L somelabel /dev/sdb1

  4. Add the /etc/fstab entry:

    LABEL=somelabel    /some/mount/point    xfs     defaults   0 0

  5. Mount up!

     # mount /some/mount/point

If you want to go down the LVM path, the above steps are basically just a more detailed version of the second set of commands in user bsd's answer below. You have to do his first set of commands before the ones above.

LVM offers certain advantages at a complexity cost. For instance, you can later "grow" an LVM volume group by adding more physical volumes to it, thereby making space to grow the logical volume ("partition" kinda, sorta), which in turn lets you grow the filesystem living on the logical volume. (See what I mean about complexity? :))

Warren Young
  • 72,032
  • 4
    The ZFS part is debatable. The Linux port has come a long way since 2010 and compared to XFS it has many advantages – TheLQ Jan 18 '12 at 18:03
  • Regarding GPT/MBR, doesn't that only really apply if /boot is part of /? The MBR shouldn't care how big / is if it only has to mount a small /boot right? I could be wrong. – jonescb Jan 20 '12 at 04:39
  • 1
    @jonescb: The location of /boot doesn't have any bearing on the limitations of MBR. If you need a partition over 2 TB, you can't use MBR partitioning. It is true, however, that it is possible to work around lack of BIOS support for booting from GPT by putting /boot on a smaller MBR-partitioned disk. Once the kernel is up, you don't have to worry about BIOS limitations because it knows how to interpret the GPT partition table. If your machine is EFI-based, you don't need to do this dance, because EFI understands GPT. – Warren Young Jan 20 '12 at 06:32
16

Just as an alternative to the other suggestions.
You don't have to partition a disk at all.
You could simple create a Volume Group, with one or more Logical Volumes.

pvcreate /dev/sdb
vgcreate data /dev/sdb
lvcreate --name dump -L '100%VG' data

Now you have a logical volume that you can format with any filesystem type you wish.

mkfs.XXXX /dev/mapper/data-dump #<- XXXX can be ext4, xfs, btrfs, reiser
mount /dev/mapper/data-dump /mntpt
bsd
  • 11,036
  • LVM is basically an advanced form of partitioning. It is pointless to use LVM if you are just going to create a single LV using all of the space, so you may as well just mkfs directly on the whole disk device. – psusi Jan 17 '12 at 18:42
  • 4
    no LV's add tools for snapshots of live data, resizable, multiple copies of metadata; far more flexible than a simple fs on a device. – bsd Jan 18 '12 at 03:03
  • 1
    You don't need multiple copies of metadata when you have no metadata ( partition table ) in the first place. Snapshots require free space, as does adding/expanding volumes, hence why it is rendered pointless if you just create a single logical volume using all of the space right off the bat. If you want the features of LVM, then you should start with a smaller volume so you have plenty of free space to use later. – psusi Jan 18 '12 at 16:39
  • I wasn't recommending LVM in this individual's case, simply listing it as an alternative to "other" partitioning, non-partitioning answers/solutions. It's up to her to read all the responses, follow up with her own research and then decide which course of action best suits her needs. – bsd Jan 18 '12 at 17:57
  • 3
    One advantage of LVM even if you only have one LV is it lets you easily add storage later. – plugwash Dec 11 '15 at 12:01
6

Question to the question: You asked 'how to partition 22TB disk' and then in the question again, you said, you just wanted a 22TB partition. So this is ambiguos in first place.

If you already have a single block device which can support 22TB of space on it, then you already posses whole 22TB partition. All you need is a filesystem on top of it, which will make the device mountable and usable for reading/writing by system processes. More ever, you need to have a Linux kernel running in 64-bit mode with a filesystem module/driver that supports and scales to 22TB of data growth, can handle the ins and outs of managing the data on the (single) block device with ease. Performance is altogether another dimension to it. In such case, I would opt to choose XFS as my filesystem, for the reason that it is a 64-bit filesystem and capable of handling filesystems as large as a million terabytes. It supports upto 9 EXABYTES.

2^63  = 9 x 1018 = 9 exabytes

For more details on XFS: http://oss.sgi.com/projects/xfs/

If you are looking for further partitioning the huge 22TB block device, then use gparted to split the device into usable partitions and then format them with the filesystems to make them mountable.

It seems that you have got hardware RAID controller, since you mention that you have got DELL perc RAID controller -- which will mean that, you have to tell which RAID configuration (precisely which RAID level are you using?) and in most cases, you are not going to get complete 22TB of space for use, I could be wrong though.

Nikhil Mulley
  • 8,315
  • I was going to suggest xfs as well, but xfs_check will take a large amount of memory, and a long time to run w/22G fs. Between physical and swap one would need at least 32G to check, that is if he ever cares to check the fs on a 'datadump' (whatever that is ;) – bsd Jan 14 '12 at 11:43
  • @bdowning, you might want to correct it to 22T fs :) – Nikhil Mulley Jan 14 '12 at 11:44
  • It is 22TB Raid 5. I have a lot of disks :) – LVLAaron Jan 14 '12 at 13:20
  • That was a typo(brain fart). I have the same device, 10TB, with LSI Megaraid (same card as Dell Perc) – bsd Jan 14 '12 at 13:27
  • @bdowning: xfs_check does indeed use a lot of memory, but the manual page(8) mentions: "Note that using xfs_check is NOT recommended. Please use xfs_repair -n instead, for better scalability and speed.". – Cristian Ciupitu Nov 29 '12 at 06:10
4

You shouldn't need any partitioning when using ZFS, just create a ZFS pool on your 22 TB device and a file system in it if you don't want to use the default one and that's it. If for some reason, zpool doesn't support using the whole disk, first create an EFI label and a partition using the whole space available inside then use that partition to create the pool.

For several reasons, I wouldn't recommend using anything but ZFS for such a large file system. The most obvious is if you have a brutal power off (eg: kernel panic or power shortage), fsck can take a painful length of time to recover traditional file systems. ZFS on the other hand doesn't need fsck so will import the pool instantly.

Note that you'd better break the hardware RAID configuration and use the twelve devices as a JBOD to build a ZFS pool with taking advantage of its software RAID capabilities. If your goal is performance, you might mirror pairs of disks and if your goal is maximized space, you might use a RAIDZ, RAIDZ2 or RAIDZ3 configuration. Doing it will largely improve the reliability of your data and the fault tolerance of the solution.

jlliagre
  • 61,204
  • 1
    I know this is an old answer to an old question, but....if using ZFS, the best option is to destroy the RAID array, configure the RAID controller for JBOD so that linux sees each individual disk, and then create mirrored pairs (less capacity, great performance) or RAIDZ/Z2/Z3 (more capacity, lackluster performance) from the individual disks. You lose much of the benefit of ZFS if you layer it on top of existing RAID rather than let it handle the disks itself. The same applies to btrfs. – cas Jun 15 '16 at 04:05
  • @cas You are definitely right, answer updated. I was focused to the OP question though "how to partition a 22TB disk" and not "what would you recommend to do with my disk configuration". – jlliagre Jun 15 '16 at 05:45
3

For your partition table, as mentioned elsewhere, GPT is an excellent option, as it supports partitions up to 9.4 ZiB in size (9.4 × 1021 bytes), which is well beyond anything you'd need with 22 TiB.

For your filesystem, on Linux BTRFS is an excellent copy-on-write file-system:

  1. Its copy-on-write attribute means that no duplicate file is stored twice.
  2. It features on-the-fly compression so your data is run through LZO or GZip before being written to and read from disk, saving physical disk space.
  3. It supports redundancy in RAID-1, RAID-10, RAID-5, and RAID-6 configurations without any overhead.
  4. It also supports RAID-0, if speed is of the essence.
  5. It also features subvolumes, snapshots, and more.
  6. Almost all filesystem tasks are done online, so you won't usually ever have to unmount the filesystem to fix things.

It's similar to ZFS in features, but is a part of the mainline Linux kernel.

Naftuli Kay
  • 39,676
3

I'm not sure this is currently possible using the standard partition table. In the standard partition table scheme, volumes are limited to 232 sectors. With 512 bytes per sector, you'd simply run out numbers to assign to sectors around 2TB.

However, you ought to be able to do this if you use a GUID Partition Table instead of a standard one. GUID partition tables allow for volumes to extend into the zettabyte range. Most Linux distros are bootable from a GUID volume, however no version of Windows (except for Windows 7 on EFI) currently is.

Some tools like fdisk can't work with GUID volumes, however other tools like GParted can. Once you create your GUID partition table, you ought to be able to create a volume using one of the several common filesystems which support a volume of that size (e.g. EXT4.)

Andrew Lambert
  • 2,366
  • On a 32-bit system may be. I have been able to create a 3.5TB partition on a 64-bit Ubuntu server supported under version 8.04 and higher. Speaking of http://www.cyberciti.biz/tips/fdisk-unable-to-create-partition-greater-2tb.html – Karlson Jan 14 '12 at 04:20
  • To my knowledge you may not even need a partition table to put a filesystem on a harddrive. Plus according to the OP it's not a boot drive just a large dump space. – Karlson Jan 14 '12 at 05:47
1

If you are not looking for the redundancy or ability to back it up you can probably do:

mkfs -t ext4 /dev/sdb
Karlson
  • 5,875
  • 4
    Have you tried it? I changed the mkfs to specify the filesystem – Karlson Jan 14 '12 at 04:05
  • 1
    @AaronJAnderson an explanation would help. – nopcorn Jan 14 '12 at 05:26
  • That command will not work if the device is over 2TB – LVLAaron Jan 14 '12 at 13:26
  • 1
    @AaronJAnderson I have created 3.5TB volume using reiserfs and ext3 so if the ext4 claims to have maximum size of the volume of 16 exabytes I don't see any reason that 22TB won't work. – Karlson Jan 14 '12 at 20:04
  • 1
    the version of e2fsutils that ships w/ this version of linux (and most others at the moment do not support over 16TB) – LVLAaron Jan 17 '12 at 08:00
  • What does redundancy or backups have to do with anything? Presumably the raid array already has redundancy and your backup options are not affected by this. – psusi Jan 17 '12 at 18:45
  • @psusi Raid will take care of redundancy but attempting to back up 22TB to tape may prove problematic depending on the solution... – Karlson Jan 17 '12 at 18:52
  • I suppose, but what does that have to do with how you format the fs? The way your answer is written implies that formatting in that way will reduce your ability to back it up. – psusi Jan 17 '12 at 19:49
  • @psusi That was referring to the volume size but I see your point. – Karlson Jan 17 '12 at 19:56