We live in perilous times, and increasingly depend on digital technology to preserve our access to knowledge, as our human read/write memories inevitably fade. Yet the digital storage technologies on which datacentres rely have notorious weaknesses. Software environments and file formats quickly become obsolete. Hard drives fail and flash loses data; LTO tapes have a 30 year life span; even archive discs only last 50.
What if part of the answer is in rethinking basic building blocks of enterprise storage? Computer Weekly spoke to US non-profit the Arch Mission Foundation (AMF) about the ultra-long term storage in its Billion-Year Archive project.
Billed as civilisation’s backup plan, the initiative ultimately seeks to build an interplanetary cloud for apocalypse-level disaster recovery, with distributed data repositories on Earth, the moon, near-Earth asteroids, orbiting the sun, and other places in the solar system.
Nova Spivack, AMF co-founder (with Nick Slavin), says the vision is to ensure that human history, culture, science and technology is backed up and can be retrieved, whether by humans or another species, even in an extinction-level event.
“If you’re creating a digital preservation strategy, key principles include redundancy and multiple off-site backups. You cannot just print everything out, so I started looking at all the media typically used. None had a real shelf-life of longer than 10 years, or nobody will certify them for longer than 10 years, Even M-DISC – meant to last 1000 years – doesn’t, because the plastic housing the ceramic layer falls apart,” he says.
Spivack’s original idea was to put a backup of Wikipedia on the moon. Any Earth civilisation, he reasons, is going to look at the moon and eventually want to go there. The questions for years were how to do it, and what storage media to use. Anything on the moon must survive diurnal cycles of boiling to sub-zero temperatures as well as massive bursts of radiation: a USB key would be destroyed in a month.
Eventually, Spivack teamed up with Professor Peter Kazansky at the University of Southampton’s Optoelectronics Research Centre, who had invented 5D optical data storage or “Superman memory crystals”. This uses femtosecond laser writing on nanostructured quartz glass, delivering 360TB of capacity per data disc capable of withstanding 1000ºC and with a lifetime of 13.8 billion years at 190ºC.
Micro-storage
First demonstrated using a 300kB text file in 2013, layers of voxel micro-storage can be written into such crystals, which can be read with an optical microscope and polariser, using machine learning algorithms that interpret polarised light as it is shone through the glass.
It’s 5D because information is encoded with the size and orientation as well as the position of the nanostructures in three dimensions. When the 2013 research was presented in San Jose that year, it was hailed as opening the door to “unlimited lifetime data storage”.
“But then, years ago, even writing a MB took like a month of work. We tested something smaller but symbolic: Isaac Asimov’s sci-fi classic, the Foundation trilogy, which is actually about people doing exactly what we’re trying to do, to preserve civilisation, helping the empire be more resilient to a future dark age,” says Spivack.
Today, writing to 5D optical can be done much faster – for example, using a new process tested by Microsoft. But at that time the AMF created just five copies of Foundation on the quartz, each worth about $1m. After many months of “maximum networking”, the team managed to get one copy sent to Mars in 2018 in the glovebox of Elon Musk’s cherry-red Tesla. Then the SpaceX mission veered off course, ending up orbiting the sun instead of crashing on Mars.
“By missing, they actually extended the shelf life of our first ‘Arch Library’ to 30-50 million years at least, before it hits anything,” says Spivack...