The Face of Computing; How Digital User Interfaces in Fiction Come Together

Iron Man HUD

Two characters in a sci-fi movie are arguing in hushed tones over a computer screen showing data about the invading alien force, the fleet of next-gen fighter planes in the air or the insertion of a new sequence into a DNA strand to create a super soldier.

‘Right there’, the gruff commanding officer says, pointing animatedly at the screen. ‘No,’ says the devil-may-care hero who plays by his own rules, pointing elsewhere, ‘over here we’ll extract more of the reverse quantum polarity quotient.’

We cut to a close-up of the sleek, flat, silver-backed computer monitor they’re arguing over and see… a blank screen.

Without user interface (UI) designers – the men and women who work in a specialised subset of the VFX field – that’s how it would look when the good guys figure out the time travel/weapons recalibration/cross-species splicing that solves the narrative problem.

UIs for the screen are an unsung art. Historically they were done using animation or the visuals native to actual computers by special effects people or editors (Steven Spielberg was never terribly interested in title sequences, usually leaving them to his longtime editor Michael Kahn).

In the CGI era they’re usually animated by the same VFX vendors that construct the scenes of Thanos’s attack on the Avengers HQ, Godzilla laying waste to a city or the realistic animal characters and backdrops of The Lion King.

If you can see yourself designing the heads up display in a futuristic warplane or the live schematics for an epic battle, there’s certainly plenty of work around. But how do you forge a path?

Master of the craft

As mentioned, UI design tends to be a subset of the larger offering from the world’s VFX vendors. But if you’re not interested in superheroes destroying cities or dinosaurs fighting, don’t despair. It’s entirely possible to make a name as a UI specialist.

“Most previous clients and frankly even new ones come to us specifically because that’s what they know we do,” says Steve Lawes, creative director and co-owner of VFX provider Cantina Creative. Cantina has a great reputation after working on Avengers: Infinity War and Endgame, Hotel Artemis, Captain Marvel, Blade Runner 2049 and countless other hits, but they’ve also carved out a name for great UI design.

Bradley G Munkowitz is a short filmmaker and photographer as well as a UI designer working under the name GMUNK, and since his well-received contribution to director Joseph Kosinski’s Oblivion (2013), he’s carved out a unique niche. He previously worked with Kosinski on 2010’s Tron: Legacy and has just completed work on the director’s sequel to 1986’s Top Gun, Top Gun: Maverick.

Munkowitz’s talent has allowed him to pursue work in only the area that interests him, so he’s living proof that you don’t have to become a VFX jack of all trades to prove yourself. VFX as a whole has never really interested him, and he uses a lot of terms from the worlds of graphic design and print layout like ‘grid’ and ‘typography’ – a very different starting point from the motion graphics principles that underlie most CGI work. “I’m interested by the directing in VFX but not the execution,” he says. “I’m inspired by user interfaces because I love graphic design and grid based design.”

But the trick to keep in mind is to remember that as a designer, you’re also a storyteller. You’re simply using UI design tools instead of Final Draft or a RED camera.

Matt Checkowski’s background is a varied as Munkowitz’s – he’s a writer and director as well as an acclaimed UI designer, so his approach to every project is how it best tells a story. His company, Department of the Fourth Dimension, was set up as his filmmaking career was taking off so he could continue to focus on design in advertising and marketing.

He’s also somewhat famous in VFX circles for coming up with the UI design for Steven Spielberg’s Minority Report (2002), which didn’t only have some of the most ‘story-embedded’ VFX ever, it actually came true – only a handful of years after the sequence of Tom Cruise pointing and swiping to control the dream images from the minds of precogs Arthur, Dashiell and Agatha, we were all doing the same thing after the arrival of the iPhone.

“We played with the VFX,” is how Checkowski describes the process of designing and executing that iconic scene, where he and his team bought their own filmmaking skills to bear. “It wasn’t the traditional process of locking the edit, sending it to VFX to do their stuff and getting it back to give notes. Everything happened in concert. The best advice I can offer is think like a filmmaker, not like a VFX artist.”

And Checkowski’s team talked the talk, submitting a pitch for the precog dream sequence as a fully realised short film you can view online (see sidebar ‘showreels’).

The DNA of the story

But just like in other VFX, there’s a strange kind tension in UI design. You want to do your best work and you want it to be seen and noticed – your work on screen is the best advertisement for your services, after all.

But if you wait for a director to hire you because they really want to showcase your stuff, you’ll soon starve. He or she wants you to design UI that fits seamlessly in the story they’re telling. In fact the axiom from way back in the pre-CGI special effects days very much applies – if you’ve done your job properly, it’ll be invisible.

Take Rogue One: A Star Wars Story. When the doomed Rogue One crew transmit the Death Star plans to the Tantive IV in orbit over the Imperial data facility on the planet Scarif, the screen showing the data transfer as the panicked rebel soldiers look on with impatience perfectly matches the style and aesthetic we saw almost 45 years ago in the original film.

Today audiences expect far more bells and whistles from UI design in science fiction because the rudimentary graphics technology from 1977 seem quaint by comparison. But in the Star Wars canon the events of A New Hope take place immediately after those of Rogue One, so it would have made no sense for the Death Star graphics to look decades more advanced.

Or take the example of the HUDs from Tony Stark’s Iron Man suits. When Cantina first worked in the MCU there was a lot of past work to draw upon. “The suits influence what we do and the [established] design language of it sort of transcends what we do,” says Lawes.

But how do you strike that balance? Everyone we spoke to who’s successful in the field had one thing in common – they consider their job is to serve the needs of the script. And as commendable as that is, it’s not always easy. Lawes says Cantina can only do it when they have all the necessary information and context. “A bit of backstory is always the most helpful scenario because we understand what we’re designing for. Things don’t feel quite so successful when we don’t have enough context or don’t know the entire story.”

In fact Lawes says that when Cantina doesn’t get that context (it’s more common than you think in today’s world of scenes split among so many providers and the overall plots on some movies being so secretive) they’ll come up with their own, just so they have a story-based basis to build UI design from.

Often, the only reference material Cantina will get is an edit and some postviz treatment, and then the challenge is to overcome what he calls ‘post love’ “They’ve lived with it for so long in the edit they can’t get out of that,” he says. “So we’re often going ‘no, there’s a different way of looking at this.’ That’s a pretty constant battle we fight.”

Despite Minority Report being Checkowski’s first onscreen UI design project (he’d mainly done main titles prior), he had what sounds like a fairly charmed time. Realistic UI design was baked into the project from the get go, when director Steven Spielberg assembled sixteen future technologies and behaviours experts and locked them in a room until they emerged with a carefully thought out picture of what the year 2054 would look like.

As soon as that was done and acclaimed production designer Alex McDowell came on board to sketch it all out, he recruited Checkowski and his team to help bring it all to life as storytellers, not just an effects crew. “It’s a testament to everyone involved in the early concept phases,” he says, “It really wasn’t just snapped on and I think that’s part of the reason why it still sings, they were there to support the amazing visual executions.”

But not every project is so easy. Having become one of the preferred providers of services to the ultra-secretive Marvel universe, Lawes and his colleagues often win assignments that are very light on story details. “We treat every project exactly the same, but it doesn’t take long before you realise ‘okay, it’s going to be one of those…’,” he says (we should hasten to add that he’s not singling Marvel out by name).

A lack of contextual detail about a project can also reveal deeper systemic problems with the whole project. As Lawes’ colleague Alan Torres told VICE online; ‘that’s when you start hearing, “Oh, yeah, make the text bigger, bigger, bigger, BIGGER!” Maybe that’s when they realize the writing isn’t so strong.’

And all the while, you have to be thinking about how the UI you’re building would work for real users in the world of your story. But you have to be thinking of the audience even more. ‘We do think of human psychology when we create these projects and consider where things need to be mapped and what makes sense in terms of visual hierarchy,’ Torres told VICE. ‘But sometimes we just go “screw that, this is what the story needs.” In real life it’s the complete opposite and a lot harder to create a working product. Development takes years as opposed to weeks.’

Tools and methods

In most cases, the workflow to concept, design and deliver UI design to a movie or TV show is much the same as any VFX gig. To Lawes, the best scenario is when you’re bouncing ideas back and forth with the production rather than just getting an assignment, toiling in isolation and reacting to notes.

Munkowitz got to do so for Oblivion in a way that would make most UI designers jealous because director Kosinski wanted the UI graphics played right on set with the actors, rather than the usual method of transposing/animating them in post production. It meant working with DP Claudio Miranda way back in preproduction, figuring out how to best display and shoot the UIs in situ on set.

As far as the critical tools and techniques, Lawes says he and his artists spend 80 percent of their time in AfterEffects and Cinema4D and a little less time in utilities like PF Track or specialist tools like Houdini where the project warrants it. Prior to that, almost everyone we spoke to said it all begins with doodling and pencil sketches, moves onto to basic wireframes or animatics for client approvals, and then goes into full-render production.

Munkowitz, again referring more to the rich legacy of print layout than motion graphics, says he works mostly in Illustrator, creating elements to then bring into Photoshop or AfterEffects to give them the optical qualities needed.

“We always start with sketches,” he says. “You know, information, hierarchy, usability, user experience. We go from ‘how does it work?’ to ‘what does it look like?’ What kind of typography are we looking at? What kind of dividers and colour palettes and structure? Then, once I have a rough idea of what we’re going to do I design the grid. The grid does the work that the interface will exist within and once you have that, you just kind of pick the time and populate the grid in beautiful ways.”

Of course, every production and director (as well as every other element) is different, and that can make the workflow a bit different for each project, according to Lawes. “Some designers love to start with little thumbnails, others start straight in Photoshop or Illustrator, some take a bunch of photographs of textures or reflections for source material for inspiration.”

And above all, remember that you can end up too deep down a small rabbit hole. Design inspiration comes from everywhere. Munkowitz and Checkowski run studios where they do far more than UI design for movies, and all that knowledge of filmmaking, art installations, advertising technologies and more cross pollinate and augment the artistry in every other area.

In fact Checkowski goes even further. He studied design at university but took classes in everything from meteorology to psychology. “I learned a lot about design through those, and then I went through a software class,” he says. “Learn how to be a storyteller and not just use the software.”

How not to do it

In the age of fast cuts, virtual cameras whirling at physically impossible speeds and orientations and the at-times dizzying overuse of CGI, how can you make your fictional UIs clear enough while making sure they’re suited to the visual style of the storytelling?

Bradley G Munkowitz refers back to 2013’s Pacific Rim. “It was interesting how they did it,” he says, “it was just so messy it was hard to follow. I think that’s kind of the default, right, just spinning spheres, layers upon layers upon layers of holograms, just to feel busy and high tech.

“But it just muddies the usability of it. As I get older as a experiential designer and artist I think more about how things are used. Even though the usability was very rudimentary in Oblivion, it was clear enough how things would be used to make it feel real. That’s just not always the case in science fiction. Sometimes it looks awesome but you don’t really understand how people use it.”

Featured speakers



The best onscreen graphics

1. Death Star destruction plan (Star Wars, 1977)

Computer animation artist Larry Cuba got the assignment to create the briefing room Death Star attack plan for Star Wars: A New Hope (1977/1997).

Given the era you might assume it was hand-animated, but Cuba worked off matte paintings and model pieces of the Death Star surface to create a digital model on a system called GRASS (GRAphics Symbiosis System), a programming language created to script 2D vector graphics animations. The sequences were then back projected onto a black screen on the briefing room set.

Prior to the LucasFilm gig, Cuba had produced several computer-animated short films, beating Pixar to the punch by decades. Search ‘Making of the Computer Graphics for Star Wars (Episode IV)’ on YouTube for the full story or go to Cuba’s Wikipedia page.

Nerd trivia; credited under ‘Computer Animation and Graphic Displays’ alongside Cuba’s name is one Dan O’Bannon, a titanic name in sci-fi as a writer and sometime actor.

2. Nostromo bootup sequence (Alien, 1979)

After a long time in cryostasis, the crew and ship of the mineral ore carrier USS Nostromo are bought back to full alertness with simple white line art on a red screen, displaying the name and serial number of the ship as the AI operating system ‘mother’ wakes up. It switches to a classic, DOS-style green text on black, a cursor scanning lines of code to the sound of an old pin printer.

It was all done by a computer graphic artist named Bernard Lodge who began his career with the BBC. Lodge and former colleague Martin Lambie-Nairn went on to herald a new era in computer-generated station idents and titles design in the UK television industry.

3. Point-and-swipe (Minority Report, 2002)

UI designer Matt Checkowski and his team went much deeper than just coming up with something that looked cool for the iconic sequence of John Anderton (Tom Cruise) organising the dream imagery from the precogs. They consulted criminal psychologists to learn about how differently multiple witnesses can remember an event to figure out how the brain synthesises and presents a picture to its owner.

Then they shot a series of short films, clips from which were incorporated into the sequence, the same scene often being slightly different between memories (degraded, off-centre, fading in and out, overlapping, etc) just like real memories behave in the brain.

4. Digital rain (The Matrix, 1999)

With shooting underway in Sydney, local VFX house Animal Logic (The Lego Movie, The Great Gatsby, Captain Marvel) were tasked with the groundbreaking visuals. In the main titles, concept artist and designer Simon Whitely gave the world something that’s still as recognisable as bullet time in the digital rain.

Comprising flipped, rotated and reflected Japanese and Latin characters and numbers, the effect evokes the green text/black background of 90s-era mainframe systems, especially the way the characters leave a fluorescent trace on the screen after they fall.

Revealing in a 2017 interview that the characters came from a Japanese cookbook, Whiteley’s design formed the basis of the quintessential main title and the underlying code behind the entire simulation Neo (Keanu Reeves) finally sees when he reaches his digital enlightenment.

5. The Iron Man suit HUD (Iron Man, 2008 – Avengers: Endgame, 2019)

The control systems, readouts and performance reports that surround Tony Stark’s (Robert Downey Jr) face are deceptively simple but a more effective storytelling device than you think.

Whether it’s receiving a mobile call, a warning about fatal ice build-up at altitude or searching CCTV footage (complete with little windows containing thumbnail clips), Downey Jr is shot face-on in extreme close-up in a dark environment to represent the inside of the suit, the overlaid animated elements not only assembled relative to each other but looking like the whole visual field is anchored to Downey Jrs POV by moving as a whole with his head or eye movements.