Xinhua News Agency Supports Object-Based Cloud Platform With Hitachi File and Content Technologies

Due to the limitations of the system’s processing capability, the audio-visual library provided services only to the Audio-Visual Department, Xinhua 08 Video Terminal, International Department and Foreign Affair Department. With the audio-visual library system being overloaded with data and in order to ensure the data is captured in time, the amount of data being exported was restricted.

Additional challenges came from an increasing number of users from the headquarters and branches in Asia Pacific, Latin America and Europe who have requested access the system. Limited by the system’s operation pressure and architecture design, the existing database system was unable to be available to users outside of the headquarters. As a result, it was unable to meet the resources requirements of the business.

Xinhua News Agency also needed a way to realize self-description and self-management of video files with advanced object storage technology and customized metadata management function. These capabilities separate the video files from the database to become independent, queriable and accessible media asset data. They eliminate read failure of the tape data due to the database system failure.

Solution

Xinhua News Agency opted to build an audio-visual sub-library for the globally distributed multimedia database. The sub-library allows the agency to support unified management of from their headquarters and the foreign and domestic branches, consolidate resources and reduce the management cost of each node. It also allows the agency to provide a solid foundation for development of TV and new media business.

This project required Xinhua News Agency to process, import and export 120 hours of video data on average per day, and meet video data requirements of the TV business for 3-5 years. It also specified support for the audio-visual data archive requirements of branch offices, and unified management and sharing of video data across the agency.

Hitachi proposed a cloud media resource management solution based on Hitachi Content Platform 500XL (HCP 500XL) object storage technology, to build the online and archive storage systems required by the business. The online storage is used to store hotspot videos, critical frame images and low bit rate videos that require quick access. And the archive storage is used to replace the tape library to store video files requiring long-term archive.

The online storage system incorporated the high-performance Hitachi NAS Platform 4100 (HNAS 4100). It is available to millions of intranet users of Xinhua News Agency to browse and edit the videos concurrently.

The archive storage is built on private cloud, which allows it to be accessed directly by intranet and extranet users of Xinhua News Agency, eliminating the waiting time of tape export. In addition, the private cloud solution addressed the issue of centralized collection of the data from dozens of branches.

Hitachi implemented a high-availability cluster with two high-end HNAS 4100 gateways. By combining the gateways with one Hitachi Unified Storage 150 (HUS 150) system, a highly reliable NAS storage system with up to 4GB/sec bandwidth is established. In addition, the sufficient Fibre Channel interfaces of HUS 150 provide SAN-based storage service for the database server.

Further, six HCP 500XL gateways and one HUS 150 create a scale-out cloud storage platform. HCP utilizes disks to store older data, and data can be accessed online via S3, HTTP, CIFS or NFS file sharing protocols. To meet the high bandwidth requirement of the media resource management system, HCP 500XL provides a proven 250MB/sec throughput per gateway node; it can meet the read/write bandwidth requirement on the archived data and still serve as “online” storage. This cloud storage platform implemented with HCP object-based system was designed to meet the data archive demands, as well as the intranet and extranet data sharing requirements.

Before this project, journalists in branches of Xinhua News Agency had to upload the videos to the headquarters via FTP, resulting in low efficiency. For this challenge, Hitachi deployed Hitachi Data Ingestor (HDI) virtual machine (VM) software in the branch offices. Combined with the HCP cloud storage platform deployed in the headquarters, HDI software can automatically collect and protect data from branches. HDI VM software deployed on VMware ESX server in each branch can be used as elastic-scale, backup-free NAS in the LAN of the branch offices.

Benefits

Completely digital and always-on media resource. Being completely digital means that the storage media of the original media resource data is changed from tapes to disks, and from offline to online. In addition, the data reliability is improved and can be accessed anytime, eliminating the need to wait and maintain.

Reduced downtime. HCP provides extremely high system reliability. The underlying RAID data protection prevents data loss caused by disk failures, and the replication mechanism of HCP can avoid business downtime.

Data security. HCP provides a comprehensive data protection mechanism. In addition to three levels of user passwords, HCP can even lock down the data with the write once, read many (WORM) feature. Before the specified time, no one can delete or alter the file. This approach ensures file authenticity and accuracy.

Data protection. Furthermore, HCP can also provide file health check and self-healing to enhance data reliability. It creates multiple copies of the file automatically with its multicopy feature. And the system regularly performs “fingerprint” (hash value) matching for the file. If it finds out that the file doesn’t match the “fingerprint,” then the file is considered corrupted. In this case, HCP automatically replaces the corrupted one with another intact copy to ensure that the file copy is always in a good state.

Advanced metadata management. With advanced object storage technology, HCP cloud storage system provides customized metadata management. HCP can describe the content of file objects automatically, without relying on any databases or application systems. Therefore, Hitachi recommends adding an additional copy of the metadata for the file objects written to HCP through the APIs of HCP while replacing the IBM® Content Manager system. This can provide two layers of protection for the metadata.

Video files are accessible over extranet. S3 and HTTP protocols of HCP cloud storage system can support routing. As a result, HCP deployed in the headquarters can be accessed in real time by intranet and extranet users. The HTTP protocol of HCP can work within separation devices, such as firewalls and air gaps, so the video files can be accessible over extranet.

Automated collection of data from branches. Journalists in branches can upload the videos to HDI quickly via LAN, so they don’t need to wait for a long time in the offices. HDI will copy the video files to HCP cloud storage platform in the headquarters automatically. Once authorized, the HDI VM can also support file sharing between branches across regions.

Elastic scale. Automated archiving means that the HDI VM can delete local outdated data automatically according to policies. This approach can release storage space and create an unlimited storage resource pool. The deleted data will be replaced by a “stub,” which points to the data replica copied to HCP. So the user will not notice that the file has been archived.

“With the HNAS and HCP cloud storage solution, Xinhua News Agency will become the first news agency to adopt disks for streaming media asset storage in China,” explains Jiao Lu, leader of Cloud Media Asset Project Team, Audio-visual Data Division, Technology Department at Xinhua News Agency. “The object storage technology and customized metadata function of HCP provides self-description capability independent of the database and application system for streaming media files. At the same time, the standard HTTP-based access protocol offered by HCP can not only eliminate the waiting time of tape export, but also provide service for users from external networks. This can improve the read and write response speed of streaming files significantly,” he notes. “Upon the completion of the system implementation, HNAS and HCP cloud storage has become the core data platform for the TV service of Xinhua News Agency.”