Shared Storage for Media Workflows… Part 2

In this guest blog post, Quantum Senior Product Marketing Manager Janet Lafleur shares in-depth insights on storage technologies as well as general usage recommendations.
Read part one of this two-part series here, written by Dalet Director of Marketing Ben Davenport, which details the key challenges for storage in today’s media workflows.

Storage Technologies for Media Workflows

Video editing has always placed higher demands on storage than any other file-based applications, and with today’s higher resolution formats, streaming video content demands even more performance from storage systems, with 4K raw requiring 1210 MB/sec per stream—7.3 times more throughput than raw HD. In the early days of non-linear editing, this level of performance could only be achieved with direct attached storage (DAS).

As technology progressed, we were able to add shared collaboration even with many HD streams. Unfortunately, with the extreme demands of 4K and beyond, many workflows are resorting to DAS again, despite its drawbacks. With DAS, sharing large media files between editors and moving the content through the workflow means copying the files across the network or on reusable media such as individual USB and Thunderbolt-attached hard drives. That’s not only expensive because it duplicates the storage capacity required; it also diminishes user productivity and can break version control protocols.

NAS vs. SAN for media workflows

For media workflows, the most common shared storage systems are scale-out Network Attached Storage (NAS), which delivers files over Ethernet, and shared SAN, which deliver content over Fibre Channel.

Scale-out NAS aggregates I/O across a cluster of nodes, each with its own network connection, for far better performance than traditional NAS. However, even the industry-leading NAS solutions running on 10 Gb Ethernet struggle to deliver more than 400MB for a single data stream.

In contrast, shared Storage Area Network (SAN) solutions can provide the 1.6 GB/sec performance required for editing streaming video files at resolutions at or greater than 2K uncompressed. In a shared SAN, access to shared volumes is carefully controlled by a server that manages file locking, space allocation and access authorization. By placing this server outside the data path – between the client and the storage – shared SAN eliminates the NAS bottleneck and improves the overall storage performance. Fortunately, there are media storage solutions that provide both NAS and SAN access from a shared storage infrastructure, giving the choice of IP or Fibre Channel protocols depending on user or application requirements.

Object storage for large-scale digital libraries

Regardless of whether it’s SAN or NAS, most disk storage systems are built with RAID. Using today’s multi-terabyte drives and RAID 6, it’s possible to manage a single RAID array up to 12 drives with a total usable capacity of about 38 terabytes. However, even a modestly sized online asset collection requires an array larger than 12 disks, putting it at higher risk of data loss from hardware failure. The alternative is dividing data across multiple RAID arrays, which increases the cost as well as management complexity. Also, failure of a 4TB or larger drive can result in increased risk and degraded performance for 24-48 hours or more while the RAID array rebuilds depending on the load of work being done.

Object storage offers a fundamentally different, more flexible approach to disk storage. Object storage uses a flat namespace and abstracts the data addressing from the physical storage, allowing digital libraries to scale indefinitely. Unlike RAID, object storage can be dispersed geographically to protect from disk, node, rack, or even site failures without replication. When a drive fails, the object storage redistributes the erasure code data without degrading user performance.

Because object storage is scalable, secure and cost-effective, and enables content to be accessible at disk access speeds from multiple locations, it’s ideal for content repositories. Object storage can be deployed with a file system layer using Fibre Channel or IP connectivity, or can be integrated directly into a media asset manager or other workflow application through HTTP REST. The best object storage implementations allow both.

Choosing the right storage for every step in the workflow

An ideal storage solution allows a single content repository to be shared throughout the workflow, but stored and accessed according to the performance and cost requirements for each workflow application.

• Shared SAN for editing, ingest and delivery. To meet the high-performance storage demands of full-resolution video content, a SAN with Fibre Channel connections should be deployed for video editing workstations, ingest and delivery servers, and any other workflow operation that requires the 700 MB/sec per user read or write performance needed to stream files at 2K resolution or above.
• Object storage or scale-out NAS for transcoding, rendering and delivery. Transcoding and rendering servers should be connected storage that can deliver 70-110 MB/sec over Ethernet with high IOPS (Input/Output Operations Per Second) performance for much smaller files, often only 4-8K in size. While scale-out NAS and object storage can both fulfill this requirement, solutions that can be managed seamlessly alongside SAN-based online storage greatly simplify management and can reduce costs.
• Object storage or LTO/LTFS tape for archiving. For large-scale asset libraries, durability and lower costs are paramount. Both object storage and LTO/LTFS tape libraries meet these requirements. But for facilities doing content monetization, object storage offers the advantage of supporting transcode and delivery operations while also offering economical, scalable long-term data protection.
• Policy-based automation to migrate and manage all storage types. No workflow storage solution with multiple storage types is truly complete without automation. With intelligent automation, content can be easily migrated between and managed across different types of storage based on workflow-specific policies.

At a time where the digital footprint of content is growing exponentially due to higher-resolution formats, additional distribution formats, and more cameras capturing more footage, the opportunities for content creators and owners have never been greater. The trick is keeping that content readily available and easily accessible for users and workflow applications to do their magic. By choosing the right storage solutions and carefully planning, facilities can move forward with new technologies to meet new demands, without disrupting their workflow.