GhostFrame at NAB: Virtual Production with 3+ Realities

Let's take an inside look at GhostFrame's brand-new technology, which creates 3+ realities simultaneously on an LED wall, expanding the possibilities for virtual production.

📬 Get the VP Land newsletter in your inbox: https://ntm.link/vpland

📺 This episode is from our NAB 2023 video series. Watch the video version of this interview here: https://youtu.be/hDCHxS8pl1M

🛠 TOOLS & RESOURCES

GhostFrame
https://www.ghostframe.com

*************

GhostFrame: The Revolutionary Virtual Production Toolset

GhostFrame is a brand new production toolset that's changing the game in virtual production, live broadcasts, esports, and corporate videos. Able to hide video content in an LED wall and maintain a perfect naked-eye image, GhostFrame enables you to shoot multiple background scenes simultaneously without the need for re-blocking, resetting, or re-lighting. Using GhostFrame, you can save time and achieve greater efficiency in your production process.

How GhostFrame Works

Peter from GhostFrame explains that the technology enables you to have multiple images hidden in the wall, which are visible through the camera, but invisible to the naked eye. The system works by shuttering the camera at a higher rate than normal, carving out and chopping out the subframe that is needed to give a perfect image. The lighting is synchronized with the background, and the lights cycle to match whatever is on the background that GhostFrame is recording.

Applications of GhostFrame

Fox Sports and Valve are already using GhostFrame to power their analysis and studio analysis, respectively. Fox Sports is using it to put their analysts inside a virtual world, while Valve is using it to be inside their game world. Applications also include extended reality and AR shoots, where multiple cameras can be used to see different backgrounds on the wall. With GhostFrame, you can achieve multi-camera 3D Unreal production, and the realities are always on the wall, making it easy to preview and cut.

Benefits of GhostFrame

GhostFrame can help you achieve greater efficiency and creativity in your production process. With the ability to shoot multiple background scenes simultaneously, you can save time and money on re-blocking, resetting, and re-lighting. The lighting is synchronized with the background, and the lights cycle to match whatever is on the background that GhostFrame is recording. It's perfect for car commercials, where the reflections are real, and it enables you to change from day to night quickly and easily. GhostFrame is also great for extended reality and AR shoots, where multiple cameras can be used to see different backgrounds on the wall.

Transcript

Joey - Shure Mic: 0:05

Welcome to VP land, where we explore all things, virtual production and future video tech. I'm your host, Joey Dowd. For the next few episodes, as we gear up towards launching the new podcast, we're revisiting some of our interviews that we did at an AB in 2023. Uh, these interviews were primarily they're shot for video. Uh, so we're just gonna pull the audio here. But if you want to see the video, and there's obviously a lot of visuals that go along with it, head over to our YouTube channel. Uh, quickest way is head to the link ntm.link/youtube. But you can still get a lot, uh, of understanding from the audio and if you're driving or commuting or chilling, and you just want to listen to the audio of the interview. This is still very useful. Um, and I'll jump in to clarify the visuals. and anything. Uh, that. Might be confusing if you're just hearing the audio. Also make sure that you get our newsletter, which is sent out. Uh, two to three times a week with the latest news and video tech, and you can get that@mtmand.link slash VP land. So in this first interview, we are revisiting ghost frame. Uh, it goes frame. This one took me a second to wrap my head around what was actually happening. Uh, so it goes from is, uh, basically a technology that can display three or more images on an led wall simultaneously. And then the camera can record three separate feeds with three different backgrounds on that led wall. And get three completely different looks, simultaneously at the same time from one production. So let's just jump into the interview and I'll come back in to clarify and add some more context.

Peter - GhostFrame: 1:30

All right. I'm here with Peter from GhostFrame, and we've got some really incredible technology here. So Peter, can you explain, kind of walk us through what we're looking at here? Yeah, absolutely. So what we're looking at with GhostFrame here, it's a brand new production tool set and it can be used across a whole range of different sectors in the industry. We're sort of particularly focused on virtual production but it also works in live broadcasts, live events. E-sports is a really popular application, I should say, that we're doing a lot of work in, as well as corporate. So there's lots of different use cases for it. GhostFrame toolkit is basically about having the ability to hide video content in an LED wall, which is invisible to the naked eye but is visible through a camera. And doing all of that at the same time as maintaining a perfect naked eye image on the video wall. So you can look at a video wall, and it just looks like a normal wall. But it's got hidden in it multiple images.

Joey - Shure Mic: 2:24

All right, Joey, here again. Uh, post production, Joey. Uh, so guestrooms demo was in a private room. It was off the, uh, the show floor. Uh, at the Las Vegas convention center, um, it was a bit kind of a hush locked door kind of thing. So you walked in this room and in this room, We've got like a pretty small row led wall set up. And then they had four different cameras set up, pointing at a demo subject to sitting down. in front of the led wall. Uh, but don't let the number of cameras confused you. That was kind of what threw me off. at first. This is really just about one camera seeing three different things. So let's just take that as Tony Bennett set up, that was connected to a monitor. And so when we walk in, we see on the led wall, they've got like a looping video plane of a daytime shot, a driving plate, driving down the Las Vegas strip. Uh, but when we look at the monitor, that's connected to the Venice, we see a quad view, and it has three different feeds. So we see the one V fee that we see with our naked eye, which is the driving down the Las Vegas strip. Uh, but then we also see two different feeds and one of the other feeds has a, uh, nighttime shot of the plate, same plate, but it's night time driving down the strip. And then the third feet is just a straight up green screen background. Uh, but we can only see to our naked eye, this one. Daytime shot. Uh, driving down, but we could see that the camera is picking up three completely separate feeds. And it looked each independent feed looked completely normal, like no flickering and nothing. Nothing was happening. So yeah, let's just jump back in and let Peter out reiterate what we just covered. here.

Peter - GhostFrame: 3:50

We're sort of particularly focused on virtual production but it also works in live broadcasts, live events. E-sports is a really popular application, I should say, that we're doing a lot of work in, as well as corporate. So there's lots of different use cases for it. GhostFrame toolkit is basically about having the ability to hide video content in an LED wall, which is invisible to the naked eye but is visible through a camera. And doing all of that at the same time as maintaining a perfect naked eye image on the video wall. So you can look at a video wall, and it just looks like a normal wall. But it's got hidden in it multiple images. So if we take a look here at the screen behind me, this first image is the naked eye view. So this is what Joey and I are seeing. Yeah, behind us we see the screen right there, yeah. So no problem. But we also have in the wall another image. In this case, it's another driving plate. This is a nighttime driving plate. And we have a third image as well, which is also invisible hidden in the wall. And in this particular case, we've decided we want it to be a green screen. So all of these images, are actually on the wall at the same time. We in the room here can only see this one. The camera can see this one as well, so that's great. But it can also see the camera. this image and this image. So now we're getting from one camera, three outputs at the same time on this wall that we've got in front of us. And that's like the magic of GhostFrame because now you have three fully broadcast quality in this case because we're running at a 59.94 hertz, fully HD in this case because we're running HD 1080. And they can all be used at the same time. So you can use them as they are. You can use them in posts. So if you wanted to go and post some new green screen content behind, you can do that right now. One of the beauties of GhostFrame is that you can you have anything. We're content agnostic so we just happen to be showing for effect a daytime and a nighttime version of the same shot so it's kind of obvious with the green screen. We can also make that a tracking pattern so you can now do tracking off an LED volume where you don't have any physical markers. Many tracking systems rely on having physical markers to be able to work out where the camera is when you track it in a fully enclosed LED volume. That's impossible because there's no physical asset to track or to mark. So we can put a tracking pattern, hide it in the wall, and then the camera tracking system can use that for its marks, which means you can now track inside a 360-degree or a 270-degree volume. So you've got all of this that you can use in post. What's its really meaning though in terms of a production process? Well now you can do two things at once, right? I mean, the reason we have a car plate here, a driving plate, is because we think car process. So car commercials could be a really big user of this. It's just this massive time saving where you can do two things at the same time. So you can shoot a daytime and a nighttime scene. When you get your perfect take, you've already got everything in the can. You don't need to re-block and reset and re-light. Because if you notice, we're lighting for each individual scene. So on this side, we've got the daylight scene, we've got daylight lighting. So we've got the correct scene- the correct lighting for this scene. But in the parallel shot here, night time, we've got the night time light. And the daytime light isn't putting any light on the scene at all. And that's the beauty of GhostFrame, that we can work on a subframe by subframe level, where this is one subframe and this is another. And we can light them separately, we can treat them separately, we can color correct them separately. We can do all the things you would normally do in a regular production, but we can do them now in multiple layers because with GhostFrame, we have different layers of content on the screen the same time.

Joey - Shure Mic: 7:36

All right. Post production, Joey here again. So to give a math breakdown of what's happening here and to simplify things, let's say you wanted to shoot two separate feeds at 30 frames per second each. So the camera would be running at 60 frames per second. And then as it's capturing every other frame, it is. Half the frames it's capturing are for the one background, the one feed, and then the other half of the frames are for the other background or the other feed. And it is recording two separate feeds and it's just. Alternating between these two different backgrounds and it's synchronized together. So it was just capturing. What is needed. Uh, for the each appropriate. Camera feed. And now that's just some simple math, but you can multiply that math out, uh, w with the frame rate, uh, and to add more video feeds. nOw recording multiple images at the same time is sort of the use case that was demoed, but the other obvious use, uh, is a multi-camera shoot. And so this was actually on display at disguises booth. And they were using goats frame. Uh, so they had a spaceship stage, uh, on their led wall and they had a multi-camera setup. And so using ghost frame, each camera could have its correct perspective on the led wall so that the virtual background is properly lined up and displayed. And so one camera is only filming one version of the background that's cycling through the led display and then the other camera is cycling and getting its correct. Uh, camera angle. From the, uh, the virtual set, off the led wall. Uh, and then the other advantage of this too, is that if you're in the mixing room of it, the technical director. Uh, you're able to just have a. final pixel, a preview of what the image. What each camera sees. And now lastly, one other question that came up a lot on the YouTube video. Uh, it was, what about the lighting? Um, so in the display and the, in the demo room, they had Keno flows, new mimic lights set up. And those were synchronized with the background. So when we saw the daytime scene, the lights are set to lighting for the daytime scene. And then when it's the nighttime scene, the lights switched and reset for the nighttime scene. Is there the recycling and sinks? Everything was in sync. So not only the background, but the lighting and everything wasn't sync. And so it was all captured. Uh, correctly inside the camera. Okay. And now back to the demo.

Peter - GhostFrame: 9:42

If we take a look on the other side, this other large monitor is seeing a single output from a different camera. In this case, it's an LDX 150, which is a Grass Valley camera. We're just seeing one output from that, actually. This camera happens to be outputting multiple outputs at the same time. But we're just choosing to see one. And in this case, it's the nighttime version. So this same scene that we're seeing here on the Sony camera, we're also seeing through the Grass Valley camera. So what does that mean? You can either have one camera taking or seeing all of the images, all of the GhostFrame images in the wall at the same time, or you can have multiple cameras and you can tune each camera in to see a different image. That's pretty cool for things like extended reality and AR shoots because now you can do multi-camera shoots with extended reality, where each camera sees a different background on the wall. And if you're tracking it, because you've got a 3D Unreal scene, for example, each scene moves individually with its camera perspective. And you can now cut between the real perspectives, which match each other camera by camera, in your Unreal scene. So now you can do multi-camera 3D Unreal production.

Joey - Shure Mic: 10:49

And how is it- are you recording this one camera, getting your different outputs? Are you recording it into a separate system to get your different- is it recording multiple files, or is it something you kind of decode later?

Peter - GhostFrame: 11:01

So basically, once the light leaves the LED and goes into the camera, then we're not touching it anymore. So there's no post-processing at all here. Everything is real. These realities are really on the wall. And these realities are really going through the camera, and they're really coming out the other side of the camera. So it's just content. We don't touch anything after the light leaves the LED wall. So you just treat your content as normal. You get three feeds out of this camera. You just record them as you normally would on a media server. So there's no extra step. Once you've actually acquired the image from the LED wall, then you're back into your normal workflow.

Joey - Shure Mic: 11:38

And in order to filter out the light that you don't want or whatever you're looking at, is it an attachment or modification to the camera?

Peter - GhostFrame: 11:45

No, we just use the shutter capability of the camera. So what we're doing is shuttering the camera at a higher rate than otherwise you might do. And that just helps us divide optically or from a light point of view, gives a nice hard edge between this subframe and this subframe so you don't have any bleed between the two frames. So we're using the shutter to kind of carve out, chop out just exactly the frame that we want here, the subframe that we want, to give you this perfect image. And that's the magic of GhostFrame.

Joey - Shure Mic: 12:15

Awesome. And I know this is still a little very early stuff, but do you have a timeline or when kind of stuff like this will be more available or rolling out?

Peter - GhostFrame: 12:21

Yeah, so from a GhostFrame point of view, we're ready to go. So the NAB really is our market launch. We had a soft launch about six months ago at the IBC show in Amsterdam, where we've been rolling out GhostFrame to a few clients. I'm really delighted to say that GhostFrame is powering the NFL on Sunday Studio for Fox Sports. And we've also been doing some e-sports work. So we did the Dota International for Valve at the back end of last year as well. And we got the Emmy nomination for both of those. So that's amazing.

Joey - Shure Mic: 12:53

What other uses- just quick overview, how are they using- what was their use case, like how are they using the different-

Peter - GhostFrame: 13:01

So two slightly different use cases, still effectively in broadcast. So for Fox, what they're doing is using GhostFrame to power their analysis. So what they have, they've built basically a virtual world, and they wanted to be able to put their analyst inside that world as they analyze more or less play by play and be able to kind of move around and have the background, the scene that they're in move around with them. So that was the multicam kind of example I mentioned earlier on. So they're using GhostFrame so that you can see a different perspective from each camera, an accurate perspective. But the brilliant thing with GhostFrame is because the reality is always there, you can preview it. So you don't need to fake that. The director just sees everything and they can cut to it straight away. So they're super happy with that, and we're already looking at some season 2 upgrades for Fox as well. It's a similar idea for Valve. They wanted as well to do studio analysis of the in-game of Dota. But there, they wanted to really be kind of in their game world. So we did exactly the same thing, tracked the cameras, and then enabled them to cut between their game engine when they were doing the analysis of their kind of play-by-play as well. So two similar applications but for very different environments.

Joey - Shure Mic: 14:19

Awesome. Cool. It's exciting stuff. Thank you so much. Appreciate it.

Peter - GhostFrame: 14:23

We're happy. And yeah, we're really pleased with the response here at NAB. It's been crazy. We've been full here in our demo room since we started on Sunday. So it's been a great reaction, and we're really looking forward to seeing a lot more GhostFrame out there pretty shortly.

Joey - Shure Mic: 14:38

Yeah, it's exciting to see. Cool, thank you.

Peter - GhostFrame: 14:39

Awesome, thank you very much.

Joey - Shure Mic: 14:42

And thanks for listening. If you want to see the full demo, of the full video, check out the link in the show notes. Also make sure to subscribe to our newsletter. That is ntm.link/vp land. And make sure you subscribe to this podcast to get our future interviews. I promise that those will sound a lot better and be more designed. Uh, for audio and not the audio from a video interview. Thanks for listening, and I'll catch you in the next episode.