Reading Petabyte-Scale Science Archives in Terabytes
A research-computing architect lives between two vocabularies. The instrument team, the data-management plan and the funding proposal all talk in petabytes. The systems that actually hold the data speak terabytes — filesystem allocations, tape pool sizes, the per-project quota on a facility ticket, the line item on a purchase. Converting between the two is the first arithmetic of every capacity conversation.
Instruments Report in PB
Facilities Allocate in TB
Tiers Are Sized Separately
Storage at this scale is sold, quoted and reported with decimal SI prefixes, so the conversion stays a clean factor of a thousand. Anything messy in a petabyte-scale plan comes from replication, parity and metadata, not from the unit change.
Working Through an Archive Allocation Request
Take the figure from the data-management plan
Type the petabyte number as written — 2.4, 0.75, 18 — into the left field. Fractions are normal here, and the input accepts a comma or a dot as the separator.
Read the terabyte total as it updates
The right-hand field converts live as you type, so you can walk the estimate up and down — raw against reduced, one copy against two — and watch the TB requirement move.
Copy the bare number into the request
The copy button on each field puts the plain figure on the clipboard with no unit and no spaces, which is what an allocation form or a spreadsheet cell wants. Ctrl+C in a field does the same.
Run it the other way for usage reports
Both fields are editable, so a terabyte figure typed on the right gives petabytes on the left. The swap button flips the direction outright when a facility report lands in TB and the steering committee wants PB.
The unit menus on both sides are searchable and list every storage unit, so neighbouring questions — a per-file average in gibibytes, a nightly ingest in gigabytes — are answered on the same page.
Real Science Archives Sized in Petabytes and Terabytes
Published archive sizes are useful anchors when you are sanity-checking an estimate. These are the commonly cited orders of magnitude for well-known collections; every one grows, so treat them as scale markers rather than live inventory.
| Scientific archive or data stream | Petabytes | Terabytes |
|---|---|---|
| Typical per-project allocation on a shared archive facility | 0.25 PB | 250 TB |
| Rubin Observatory / LSST camera raw images, per month | 0.5 PB | 500 TB |
| Mid-size institutional HPC scratch filesystem | 5 PB | 5 000 TB |
| Data recorded by a large LHC experiment in one year | 40 PB | 40 000 TB |
| NCBI Sequence Read Archive, public sequence data | 42 PB | 42 000 TB |
| One copy of the full ten-year LSST survey data set | 85 PB | 85 000 TB |
A two-digit petabyte headline becomes a five-digit terabyte number, and that is where the plan stops being abstract. Eighty-five petabytes is a sentence; eighty-five thousand terabytes is a tape library floor plan and a decade of migrations.
Every archive unit in one searchable list
Type-to-filter dropdowns on both sides carry decimal PB and TB for procurement, binary TiB and GiB for what the filesystem reports, and bit units for transfer rates.
Readable output at survey scale
Results carry up to eight decimals with thousands separated by spaces, and switch to scientific notation once a value gets very large, so byte-level projections stay legible.
Allocation figures stay on your machine
All arithmetic runs in the page once it has loaded, which matters when the number you are typing is an unpublished collaboration figure.
Petabyte Archive Questions from Research Computing
Why are scientific data sets quoted in petabytes rather than terabytes?
Because the terabyte figure would be five digits long and hard to compare. Modern instruments are continuous producers — a wide-field survey camera, a sequencing core and a collider detector all write tens of terabytes a day — so annual and lifetime totals land naturally in the petabyte range. Quoting 85 PB rather than 85 000 TB keeps proposals and review documents readable. The terabyte restatement comes back the moment somebody has to buy or allocate the space.
How is a petabyte-scale research archive split between tape and disk?
Most keep a disk tier sized as a working cache — a few hundred terabytes to a couple of petabytes — in front of a much larger tape tier holding the bulk of the collection. The split follows access pattern: reprocessing campaigns and recent data stay on disk where random reads are cheap, older raw data migrates down. Convert per tier, not in one lump. A 10 PB archive is not a 10 000 TB purchase; it is perhaps 1 000 TB of disk and 9 000 TB of tape media, at very different unit costs.
How long does it take to move a petabyte over Globus or a dedicated transfer node?
It depends on the pipe and on how many transfer nodes sit behind it. A single 10 Gb/s link delivers about 108 TB per day fully utilised, so one petabyte takes a little over nine days with nothing going wrong. Facilities that have engineered this properly do far better: demonstrations across well-tuned DTNs have sustained roughly 93 Gb/s for a full day, about one petabyte in 24 hours. The real limits are usually file count, per-file size distribution and metadata rates rather than the wire.
How much time does end-to-end checksum verification add at this scale?
Enough to change the schedule if you plan it as a separate pass. Hashing a petabyte at 500 MB/s on one core takes roughly 23 days; at an aggregate 2 GB/s across several workers it is still about six days. That is why serious transfer tooling overlaps verification with movement at block level instead of running it afterwards. Expect a small percentage of transfers to fail verification and need a retry — corruption at the source, on the wire or at the destination is a normal event at this volume.
Why do shared facilities hand out project allocations in terabytes?
Because the allocation has to be enforceable and comparable. A facility distributing a few petabytes among dozens of groups needs granularity fine enough to say yes to 40 TB and no to 400 TB, and to sum those grants without every line being a fraction. It also keeps reviews honest: a group writing "we need 1.5 PB" is asking for 1 500 TB, which against a 6 PB shared archive is a quarter of the whole facility — a claim that reads very differently in everyone else's unit.
No comments yet. Be the first to comment!