I'd like to introduce to you, a part of filmmaking that no-one likes to really think about, because it's (i'll admit) one of the least interesting parts of the process. Data Back up. I can't begin to describe how many times I've been working and been given either a slower card reader, or to save on money, been bought cheaper drives.
What many don't realise that all this adds up, and to those producers out there where overtime is a very realistic possibility, it might actually be worth investing in better hard drives to avoid overtime. Overtime for the DIT, overtime for production and if on location, overtime for the sparks and location costs. A small investment of a couple hundred £££ might save double, or triple that in the long run...
So in this article, we'll be taking a look at how we can ease the pain of those 5 cards at 128GB each, and how best to manage the workflow...
As a humble DIT / Data Manager, the case of dealing with data has become (I feel) more and more challenging over the past few years.
With the RED KOMODO happily making it's way into consumers hands, and with cameras of the likes of the Panasonic S1H / EVA1, Sony FX6 & A7S.iii, Canon C70 & C300.iii and BLACKMAGIC POCKET CINEMA CAMERAS all shooting LongOP, All-Intra or in some cases RAW, data management has become more crucial than ever, and how we can manage the sheer volume of cards that comes into the data station and where to put it can be quite stressful.
Especially when you need clearance to clear cards, and you have a limited amount of cards...slowly....going....down...
But even though as mundane as data transfer is, having a nicely structured system with reports makes the editors life a lot easier, and with this, we're going to take a look at my particular workflow, and why I use it.
Firstly, let's take a look at our source, input and destination.
There's 4 things you want to start with when it comes to DATA backup :
Source media
Bus
Software
Destination
So lets take a look at each one :
SOURCE MEDIA
This is your camera media. This is your SD card, CFast card, CFExpress, RED Mini mag, Arri Codex Capture Drive, Sony AXSM.
Manufacturers use different media to tailor what kind of content the camera is creating. For example, RED uses SSD's because it needs to haul huge amounts of data into the mini mag, whereas your humble GH5 or S1H uses V90(fast) SD cards - they're not recording RAW internally, and it saves on space and heat - hence why they're usually smaller / lighter.
I won't go into the speeds of each media here, because there's so many variables on each and every camera, and thats not why we're here.
BUS
This will be your USB and Thunderbolt ports. In order of speed we go :
Thunderbolt 3
USB 4*
Thunderbolt 2
USB 3.2
USB 3.1
USB 3.0
USB has gotten super confusing due to it's naming conventions of Gen 1 and Gen 2, and Type-C connections, but all you need to know really is that higher number means faster. This goes for USB version, and generation version.
*USB4 matches the Thunderbolt 3 protocol, which is a 40Gbps data throughput, however, I don't think anything uses USB 4 yet apart from those 2020 M1 Macbook Pros.
and with that, everything becomes more expensive the higher up the chain you go. It's the difference between a £20 card reader, and a £100 card reader (....Or a £3000 card reader)
Let's look at my personal set-up:
As well documented on my website, I shoot with a Panasonic EVA1 cinema camera. This camera uses SD cards for LongOP or Intra-Frame recording. I also on occasion output to a Blackmagic 12G View assist monitor, and record Prores.
My card reader is a PROGRADE SD + CFAST card reader, and usually my backup drives are SANDISK 1050mbps SSD.
I have 2 of these, one red, one blue - Master and backup. It's super important to have 2 hard copies of the footage. Yes, it's an additional expense, but if something happens to one of the copies and you've already formatted your cards, it's game over.
I use ANGELBIRD and PROGRADE SD Cards, neither one has caused me any problems in post, despite the prograde cards being V60 vs V90.
This is a generic USB-C hub I use with my Macbook pro, it reads SD cards, allows me to plug in my keyboard and grading panel and outputs HDMI.
Built even though it can read SD cards, it's not built for it.
SOFTWARE
There's a few decent options out there now for software to back up your footage, the ones I know of are :
- Shotput Pro
- Hedge
- Yoyotta
- Silverstack
Silverstack being by far the most featured, but it has a timed license, which can add up if you're not on a project with that budget in mind.
Shotput Pro, Hedge and Yoyotta all hit the same price-points there abouts around £150, and all effectively do the same thing.
Clone your cards, Perform Verification and Generate reports.
The only one I've used extensively thus far is Yoyotta, so thats what I'll focus on.
You can find your input on the left, in the middle it gives you some of the information that you'd need for a quick glance, and your output destination on the right.
You can add as many destinations as you like, just be mindful of having increased simultaneous destinations, as this could impact how quickly the backups will go.
If we delve a little into the project settings in Yoyotta, you can see that there's options for saving an ALE, CSV, MD5*, MHL and a PDF. There's also an option to send that PDF as an email - which I do every single time. For peace of mind.
*MD5 is a checksum type, Yoyotta is also utilises xxHASH and there's even a checkbox in there to ensure Yoyotta produces what it needs to for Netflix shows.
It's worth noting that using these tools can not only yield a better cloning speed, but also a guarantee that every single bit of data is identical to the source media.
Da Vinci Resolve also has a cloning tool. It's pretty basic and doesn't generate a comprehensive amount of reports, but it's there, and a great free option. I'm not sure how fast it is, however.
During a backup Yoyotta also lets you know how fast everything is moving along, meaning if you start a backup and you're getting 38MBps - you know something's wrong from the start.
And as of writing, Yoyotta has yet to un-indentify any type of media I've thrown at it. It's had everything from my EVA, Red mini mags, Sony AXSM cards & Cfast, all displaying the properties mentioned above.
SPEED
Speed is all important, and it's important to make sure you iron out any of the bottlenecks. This is source media, destination drive/s and cables.
First, let's talk card readers. Up above there's an image of a Prograde USB-C reader, and a £20-30 do-it-all dongle. Take a look below :
On the left, we have the Prograde reader. On the right, we have our regular adapter.
My making a simple change, we can see we've shaved a 12 minute backup, to 4.
That's 3x faster.
Take a look at the copy speed in the destination window, we're looking at 89ish MBs vs. 242MBs
If 3 cards with the same scene all came in with similar data sizes the total amount of time spend cloning would be :
3*12 = 36 minutes with normal reader.
3*4 = 12 minutes with prograde reader.
That's an extra 24 minutes, and that's only for 35GB!
Destination DATA
But then there's also the destination speed, the drive also impacts how quickly you can transfer data across.
Lets see the difference between a that SSD disk I pictured earlier, my G-RAID (2 Spinning disks, 7200rpm) and a standard £60 1TB hard drive
On the left, the Sandisk SSD, in the middle, a G-Raid and on the right a typical-standard USB spinning disk hard drive.
I mean, The results speak for themselves. Think about the throttling of data from one side of the other, at each point there's an opportunity to throttle.
WARNING. ALL CABLES ARE NOT MADE EQUALLY. RE: THUNDERBOLT 3.
You wouldn't think Thunderbolt has any cons, after all, it's an all in one port that does everything, and does it quickly, but be warned, the USB-C connection relies on the chain being Thunderbolt 3.
Let me explain,
Let's take that 'snow' drive, put it in my Prograde card reader, but maybe all I have is a Thunderbolt 3 cable - it's USB-C after all - so speeds should match right?
Wrong.
Thunderbolt uses the USB-C system, but how it works is all based on the chain being Thunderbolt. This means source to cloning, and cloning to destination.
Let's examine a backup, using the Prograde reader, a fast SD card and the SSD as a destination.
As we can see above, using Thunderbolt 3 cables in a Non-Thunderbolt 3 workflow absolutely hammers speeds. So be prepared with a variety of cables incase you're caught out.
Does That All Make Sense?
It's difficult to explain the expansive reasons why something isn't performing as it should. Maybe the card is old, maybe it's a bad card, maybe the connectors are dirty or maybe there's a case of the drives being old - it can and will happen!
But hopefully, this little article will explain to you a handful of those problems, and in some cases I hope, improve your workflow.
Thanks for reading folks, and as always, happy shooting!
Andrew McGovern is a Video Content Producer and DIT based in South West London. From big-budget feature films to commercial videos, 'Elvis' as he's known to his colleagues has experience in many types of filmmaking.