The majority of open source bioinformatics applications in use today have been written for the Linux OS on x86_64 hardware, and this is a standard platform for the majority of Linux servers and HPC clusters. A commodity server available from most major vendors will typically support a maximum of 256GB or 512GB of RAM on the motherboard, which is adequate for the majority of applications. However there are algorithms in modern bioinformatics which require a very much larger memory footprint and where a standard server (or node in a cluster) is not a practical solution.
The SGI Altix UV range of system offers a large contiguous memory footprint within a single system image (SSI), which is of particular importance for memory intensive applications such as the ‘assembly’ of genome data output from high throughput genome sequencing instruments.
TGAC has three SGI UV systems. Two UV100 systems with 768 cores and 6TB of memory, and one UV2000 system with 2560 cores, 20TB of memory and 32 Xeon Phi modules. The UV2000 has been divided into three hardware partitions, each of which runs a separate instance of Linux, and so they can be regarded as three separate systems.
System Model | UV100 | UV100 | UV2000 | (UV2000) | (UV2000) | UV300 | UV300 |
---|---|---|---|---|---|---|---|
Partitions | Partition 1 | Partition 1 | Partition 1 | Partition 2 | Partition 3 | Partition 1 | Partition 1 |
System Name | uv1 | uv2 | uv2k1 | uv2k2 | uv2k3 | Ripley | |
Designation | Decommissioned | Production | Production | Production | Development | Production | Production |
CPU cores | 768 | 768 | 1024 | 1024 | 512 | 256 | 256 |
Total RAM | 6TB | 6TB | 8TB | 8TB | 4TB | 12TB | 12TB |
CPU model | Intel Xeon X7560, 2.27 GHz | Intel Xeon E7-8837, 2.66 GHz | Intel SandyBridge E5-4650L, 2.6 GHz | Intel SandyBridge E5-4650L, 2.6 GHz | Intel SandyBridge E5-4650L, 2.6 GHz | Intel Haswell E7-8867V3, 2.5GHz | Intel Haswell E7-8867V3, 2.5GHz |
CPU cache | 24MB L3 | 24MB Smart Cache | 20MB Smart Cache | 20MB Smart Cache | 20MB Smart Cache | 45MB Smart Cache | 45MB Smart Cache |
Offload processors | none | none | none | none | 32 x Xeon Phi 5110P (Knights Corner 2250). 8GB RAM and 60 cores per Phi, 1.1GHz | none | none |
Blades | 48 | 48 | 64 | 64 | 48 | 4 | 4 |
Sockets per blade | 2 | 2 | 2 | 2 | 16 x 2 skt , 32 x 1 skt *See Note 1 | 4 | 4 |
Cores per socket | 8 | 8 | 8 | 8 | 8 | 16 | 16 |
RAM per socket/blade | 64GB / 128GB | 64GB / 128GB | 64GB / 128GB | 64GB / 128GB | 64GB / (32x64GB , 16x128GB) *See Note 1 | 768GB / 3TB | 768GB / 3TB |
RAM per core | 8GB | 8GB | 8GB | 8GB | 8GB | 48GB | 48GB |
Operating System | RHEL 6.2 | RHEL 6.4 | RHEL 6.4 | RHEL 6.4 | RHEL 6.4 | RHEL 7.1 | RHEL 7.1 |
Load Scheduler | PBS Pro version 11.3 | PBS Pro version 11.2 | PBS Pro version 11.3 | PBS Pro version 11.3 | PBS Pro version 12.0 | SLURM v14.11 | SLURM v14.11 |
Scratch Storage | 20TB XFS | 88TB CXFS | 110TB CXFS shared across all UV2000 partitions | 100TB S100 disk XFS 32TB NVMe FLASH (16x 2TB P3700) | 100TB S100 disk XFS 32TB NVMe FLASH (16x 2TB P3700) |
Note 1: 48 blades in total, 32 blades with a Phi installed, 16 without