2x HCP S30 with 801TB Capacity Licence
2x HCP VM Software
Hitachi Data Ingestor for VMware
Object storage and data management as-a-service, white-labelled as Cascade and delivered by Revera
Collect, share and preserve New Zealand’s digital heritage, cost effectively and to archival standards.
End-to-End, 900TB integrated object storage/HCP solution, delivered through Revera as-a-service
Innovation and technology enables national library to preserve and manage digital heritage collections, ensuring data provenance and authenticity.
The National Library of New Zealand builds collections of social, cultural, creative, economic, scientific and other outputs and outcomes of New Zealanders to preserve the history and identity of the people who live and work in New Zealand.
Its job is to keep these collections in perpetuity, to make them available to researchers, historians, teachers, students and others now and in the future, and to make sure they are preserved, accessible and meaningful in 100 – 200 years’ time.
The National Library of New Zealand's data centre was ageing and the city of Wellington had been hit by a serious earthquake. In addition to drivers such as geographic redundancy and disaster recovery, the Library undertook an Optimising Digital Storage project to address increasing storage needs, reduce the cost of storage and find a future-proofed solution for its significant data management requirements.
"As a collecting institution, our storage requirements will only ever grow. We need to make the process more affordable so we can collect even more digital treasures to share with New Zealanders in the future," said Bill Macnaught, National Librarian.
The challenge of making sure a digital collection can be accessed and understood on an ongoing basis means grappling with hardware and software obsolescence and researching what is required to maintain meaning in a digital object over time.
Digital record keeping and archives transcend generations and the decisions made today can influence an unknown future, especially for documents of cultural significance. The National Digital Heritage Archive, which houses the National Library's born digital and digitised collections, is not just a database, but an environment of about a petabyte with 170 discrete formats and their variants, some of which are dating back as far as the late 1700s.
"We take physical objects, digitise them and preserve those digital versions. And we must also collect and preserve what is called 'born digital' material, material that has only ever been in a digital form," said Steve Knight, Programme Director Preservation Research.
He and his team identify what is being published in New Zealand in digital form, collect it and build the necessary digital processes that conceptually mirror the archiving of physical objects, to make sure that born digital material can be kept in perpetuity.
"We collect the whole of the .nz web domain, harvesting it on an annual basis, at a rate of 20 to 30 terabytes per crawl. We have over five million pages of historical newspapers we make available online, and much more. The content we need to collect is only going to grow both in terms of volume and complexity," said Knight.
When it comes to data management, the Library identified that archival concepts such as context and provenance are critical. Knight explained that when preserving the authenticity of a digital collection, every change to or impact on every object must become part of the bundle of information that is stored with that object so its provenance can be verified and trusted.
"Our job is to make sure that in 100 years' time the person using our digital archive has the confidence and trust in the National Library of New Zealand to accept that each object is what it is supposed to be. And if it has been changed – even if it is just from a PDF to a JPEG – they can see the record of every change stored with that digital object," said Knight.
The New Zealand Government mandated that its departments use one of three sovereign data centres, which included Revera. By 2015, the Library had completely exited its own, failing data centre and transferred all of its permanent collections to the Revera data centre under an Infrastructure as a Service model. Looking to reduce its cost profile for digital storage and gain control of its unstructured data, the Library commissioned an options paper and decided to undertake a greenfield project with Revera and Hitachi Vantara. The result is a solution for digital preservation and provenance that is the first of its kind around the world.
Knight and his team believed that object storage would be a scalable, cost effective solution, but needed a robust data management layer so the Library could ensure that the authenticity and integrity of its digital objects are maintained over time. While they could see the value of the cloud, they wanted a private cloud option that would service their particular needs as a government department.
"We knew Revera had a history with Hitachi Vantara," said Knight. " At the beginning of the project we didn't realise that would be where the core technologies would come from, but it soon became clear that was the right way to go and that Hitachi were really interested in working with Revera and the National Library of New Zealand to co-create a solution."
That solution is end-to-end, 900TB integrated object storage underpinned by the Hitachi Content Platform solution, which has been white labeled as Cascade and delivered through Revera as-a-service.
"This collection belongs to all New Zealanders and technology makes it possible, at last, to make it very easy for any New Zealander to get access to the wealth of knowledge that we have collected on their behalf," – Bill Macnaught, National Librarian
When the building next door had to be demolished following a magnitude 7.8 earthquake in 2016, the need for geographic redundancy became very real for the National Library of New Zealand. They now have real-time replication from their home base in Wellington to Auckland and have plans for tertiary copies elsewhere as a next step.
But the benefits go well beyond fulfilling disaster recovery requirements according to Knight, who had been tasked with finding cost savings for the organisation – and New Zealand taxpayers.
"It is certainly not an easy or common thing to have a greenfield technology project that is actually going to save you money," said Knight. "We have a few months to go but the expectation is that the cost savings we envisioned in the beginning will be realised."
The solution offers the Library the ability to store, archive and access unstructured data, optimise storage with automated compression, rebuild large data sets quickly when they need to share data with others, and reduce costs overall.
"Hitachi Vantara and Revera have co-created a solution at the cutting edge of digital archiving and preservation, so we can ensure the digital heritage of New Zealand will be available, accessible, meaningful and authentic for generations to come."
Pairing intelligent archiving capabilities with advanced metadata features and tools that allow for tagging digital objects with custom metadata, means that the Library can fulfill its responsibility to preserve the digital provenance of its collections. The chain of custody for each digital object can be proven, and the Write Once Read Many (WORM) storage is undeletable and guarantees authenticity.
"I'm really pleased that we can demonstrate that we are capable in the digital space; that we are a digital business with digital services. Customers in New Zealand and around the world can expect digital interactions with us. On behalf of New Zealanders, we are setting ourselves up to be a local node in a global network of uniquely trusted institutions and countries," said Knight.
Accelerate innovation and build your DataOps practice with data management products for analytics, AI and data integration.LEARN MORE
Hitachi Vantara and Revera have co-created a solution at the cutting edge of digital archiving and preservation, so we can ensure the digital heritage of New Zealand will be available, accessible, meaningful and authentic for generations to come.
– Steve Knight, Programme Director Preservation Research, National Library of New Zealand.