As sales pitches go, Nvidia’s deep learning supersampling (DLSS) is as revelatory, and preposterously sci-fi, as it gets. Specific cores dedicated to machine learning, harnessing bleeding-edge algorithms, take resource-friendly 1440p resolution images and blow them up to super-sharp 4K proportions—and all without the performance hit of actually running at that higher resolution in the first place.
Along with that quietly revolutionary real-time ray tracing capability, DLSS was the big selling point of Turing cards. But while the advantages are real, and available to you in a generous swathe of games, its uneven execution from title to title is confusing. In its worst moments, it’s been capable of actually making your game look worse when enabled than in the original native resolution.
But the teething problems might prove trivial when we look at deep learning’s potential to turn graphics rendering on its head. Just as real-time ray tracing’s proved to be what some might call a soft launch, offering tantalizing glimpses of its promise amid the noise of patchy game support and performance issues, so it goes with DLSS. For owners, it feels like being in on the ground floor on an important new advancement, and having to deal with adjustment inertia from software developers and driver writers.
Before we can really assess what deep learning might do for our gaming rigs in five or ten years’ time, it’s important to pin down what exactly’s happening in those Tensor cores, why it differs from temporal antialiasing methods or downsampling, and how it makes the games of 2020 look and perform. And to do that, you have to weigh the marketing talk against the look and feel of games such as Control and Metro Exodus, frontrunners in DLSS implementation. First, the fundamental premise: What’s going on inside Nvidia’s Tensor cores?
This much is clear: Nvidia’s been working on deep learning for many years now. Current technical director of deep learning, Andrew Edelstein, has been operating in that capacity since 2010, and for years the company’s CEO, Jensen Huang, has been demonstrating a deep-learning neural network demo on the trade show circuit in which a computer identified pictures of cats and dogs. The more pictures it examined, the more accurate it became.
Despite its interest and investment in this area for over a decade, it wasn’t until 2018 that deep learning was successfully applied to rendering a 3D environment. That breakthrough came via a conditional generative neural network that could interpret video footage of real locations and turn them into 3D environments. VP of applied deep learning Bryan Catanzaro’s research team captured dashcam-style video of cars driving around several cities, then identified the “high-level semantics” of each scene—using Unreal Engine 4 to colorize the objects and scenery elements, and teaching the network what a car is, what street lights are, what kerbs look like. Once that was established, the network could correctly identify them independently in future video footage, just like the AI that picked out cats and dogs.
The end result isn’t a simplistic environment, either. The busy downtown streets are packed with recognizable cars, each one a different real-world model, just as you’d see if you were unfortunate enough to find yourself in a real traffic jam. Beyond them are trees and skyscrapers, and it’s possible to drive through the environment as if it were just another urban circuit in a racing game. (Naturally, the authors of this tech were quick to put it to other tasks. One fed the learning AI footage of people dancing “Gangnam Style” to render a 3D scene of their colleague doing a very convincing rendition.) It’s undeniably impressive to see such a vivid 3D environment rendered intelligently by a machine that only had flat video data to build it from—but it bears repeating that this only happened in 2018. This is still very new technology.
THE BIG PICTURE
You can read up to 3 premium stories before you subscribe to Magzter GOLD
Log in, if you are already a subscriber
Get unlimited access to thousands of curated premium stories, newspapers and 5,000+ magazines
READ THE ENTIRE ISSUE