Blackmagic Design’s top new AI tools for DaVinci Resolve Studio & Fusion Studio.
At Blackmagic Design, our company philosophy includes helping true creativity blossom, and our approach to artificial intelligence (AI) is simply that. We view AI as a tool that can help creators, whether they be VFX artists, editors, colorists, sound engineers or all of the above, save valuable time so they can invest themselves elsewhere in their creative process.
While AI has been a hot topic this year, it’s been part of the foundation of DaVinci Resolve Studio since 2019 when we announced the DaVinci Neural Engine. This advanced system uses state-of-the-art deep neural networks and machine learning, along with AI, to power features such as SpeedWarp motion estimation for retiming, SuperScale for up-scaling footage, and facial recognition for automatically organizing media into bins.
The DaVinci Neural Engine provides simple yet powerful tools to solve complex, repetitive, and time-consuming problems. With tight deadlines, time constraints can often be the enemy of creativity and ingenuity, so the role of AI is to help vanquish that foe. By automating some of these more tedious post processes, we’re able to free up time that can then be used to further develop the creative, and hopefully fun, parts of the job. After all, no one got into the VFX industry because they wanted to spend all day sorting footage. When artists are unencumbered, new and unique ideas are allowed to flourish and drive innovation.
Empowering, Not Replacing Creatives
The goal of AI within both DaVinci Resolve Studio, our editing, color grading, VFX and audio post production software, and Fusion Studio, our VFX and motion graphics software, is to allow filmmakers to work not only faster and more efficiently, but also more creatively. We want AI to empower their work, not replace it.
With that in mind, we recently introduced several new AI-powered features into the software.
For example, both the latest versions of Fusion Studio and DaVinci Resolve Studio include a native AI-based depth map tool that lets users instantly generate a 3D depth matte of a scene to quickly grade the foreground separately from the background and vice versa. This allows them to quickly bring attention to action in the foreground, help interview subjects stand out, or add atmosphere in the background of a scene.
There is a new Relight FX that allows users to add virtual light sources into scenes to creatively adjust environmental lighting, fill dark shadows or change the mood. Light sources can be directional to cast a broad light, a point source, or a spotlight, and be adjusted for surface softness and specularity control.
We also introduced new AI-powered text-based editing with automated transcripts that are generated by the DaVinci Neural Engine. The AI speech-to-text transcribe feature automatically transcribes video and audio clips so users can mark in and out points to edit or delete text, generate sub clips, add markers, and even remove silence. Users need to simply select the clips in the media pool and use the menus to select audio transcription and then transcribe. Moreover, users can also quickly generate subtitles with the new auto subtitle feature.
For AI audio classification, the DaVinci Neural Engine can categorize audio clips based on their content, which makes editing choices faster when reviewing unfamiliar materials. After analysis, audio clips appear in collections bins for dialogue, music, and effects with detailed sub categories, such as explosions, water, or insects.
Additionally, the new AI-based voice isolation track FX lets users remove loud, undesirable sounds from voice recordings, which is perfect for interviews and dialogue recordings from noisy locations. By isolating dialogue from background sounds in a recording, users can eliminate everything else from moderate noise to loud aircraft sounds and explosions, leaving only the voice.
These new AI-powered features are built on the strong foundation of advanced machine learning that has been around since DaVinci Resolve Studio 16 and Fusion Studio 16.
For example, the magic mask palette uses the DaVinci Neural Engine to detect people and track their movement in a shot, producing clean traveling mattes for secondary grades or effects. Creators can make characters stand out in an under-lit shot or invert the person mask and stylize the background.
Object mask, located in the magic mask palette, is able to recognize and track the movement of thousands of unique objects. The DaVinci Neural Engine intuitively isolates animals, vehicles, people and food, plus countless other elements for advanced secondary grading and effects applications.
The object removal tool is for covering up moving objects in an otherwise static shot. It analyzes a scene and uses the background information to paint out unwanted data.
The Open FX library features more than a dozen stylize effects that can artistically modify the content of the image. Ranging from the creative, such as pencil sketch, halation, and abstraction, to the practical, such as blanking fill, these effects analyze the content of a shot to produce striking imagery.
The patch replacer tool samples an area of a scene to quickly cover up unwanted elements, such as boom mics or even blemishes on a face. Its auto grading feature seamlessly blends the covered area with the surrounding color data even when there is movement or changes in the light.
The face refinement filter uses facial feature recognition to automatically detect and track faces. Creators can use its many parameters for general beauty work, such as skin smoothing, correcting unflattering color casts, and retouching individual features like eyes, lips, and cheeks.
The SuperScale upscaling algorithm creates new pixels when increasing the resolution of an image, which is ideal for scaling archival footage into modern resolutions, or upscaling HD footage to match a 4K or 8K project. The newly generated data has more detail in upscaled images than standard resizing controls. Additionally, the dead pixel fixer will use surrounding pixel data to cover up dead pixels, rescuing footage and preventing reshoots.
The Smart Reframe tool for social media lets creators use 16:9 HD or Ultra HD and quickly create square or vertical versions for posting to Instagram and other apps. The DaVinci Neural Engine automatically identifies action and repositions the image inside a new frame, so users don’t have to do it manually.
The Future of AI
Since first introducing AI in 2019, we’ve seen the impact it can have on speeding up workflows and supporting, not replacing, creatives. That is where we think the future of AI will be, and where its true power lies: in helping true creativity blossom.
As we continue to develop Fusion Studio and DaVinci Resolve Studio, we’ll also continue to look to our customers for new ways we can automate tedious parts of the post production process with AI so we can free up time and further empower their artistry.