TCB Data Storage Facility

The TCB's server infrastructure is primarily dedicated to providing a centralized data storage for all of the Resource's needs, protecting the valuable data from loss by hardware failure and, to a lesser extent, human error. Since it was first developed in 1999, the system has proved both reliable and readily expandable, offering remarkably few compromises as our data needs have grown exponentially. We expect that we will be able to continue this growth, and hope that our experiences may help other groups that are struggling with the same needs.

Scrtach Storage

Cluster GPU

Cluster GPU

Cluster GPU

Intel Xeon CPU E5-2623 v3
4 Supermicro Storage Servers
(240TB Storage each server) 		Intel Xeon E5-2637 -
24 TB SSD Internal Storage 		Intel Xeon CPU E5-2623 v3
4 Supermicro Storage Servers
(360TB Storage each server) 		Intel Xeon CPU E5-2623 v3
5 Supermicro Storage Servers
(360TB Storage each server)

System Specifications

  • File Servers
    • Intel Xeon E5-2637 - 24 TB SSD x2
    • Intel Xeon E5-2623 V3 - 240 TB HGST x4
    • Intel Xeon E5-2623 V3 - 360 TB HGST x13

    • File Servers

    We use redundant file servers running Linux, FreeBSD to share our files across our network. These systems offer three major benefits:

    1. An enterprise-class - ie stable and powerful - NFS server package.
    2. Support for the ufs and zfs file system and filesystem snapshots.
    3. Security and reliability.

    The disks are divided up into the following categories:

    /home and /Projects

    These disks are backed up regularly. /home partitions are made for every user of the system; each disk is shared across many (40+) users with quotas.

    /Projects partitions are allocated to users that show need for them. In general, they are allocated on a per-person basis; exceptions to this rule are permitted only after discussion with the sysadmin team. To this end, partitions are generally either 150GB or 300GB.

    /Scr

    These disks are not backed up, but are otherwise treated like /Projects spaces. They are also on ZFS pools with quota.

    Disk Arrays

    Unlike many groups, we use ZFS RAID and the ZFS file system, a feature with Linux and FreeBSD Operating System. ZFS allows for greatly simplified storage management and snapshot based backups.

    Backups

    Backups for home and project spaces are sent nightly via ZFS snapshot to multiple backup servers in a different machine room. Three nightly snapshots are retained on the live server to allow users to recover accidentally deleted files without administrator assistance.