Augmented Reality - Part 2 - Vision Pro and Killer Apps

In the second of their two part series on AR, Rob and PJ discuss the Apple Vision Pro, the use case displayed thus far, and whether this thing has legs in the marketplace.

They also explore the social (or more precisely, anti-social) aspects of the AR, VR, XR space and question whether this is the right direction to be going in.

Finally, they also ask the question...how big a resolution do you actually need before it really looks "real world"?

Transcript

PJ: 0:01

Hi folks. This is the second of a two-part series on augmented reality. If you haven't listened to the first, you might want to go back and listen to that one. Just to get a little groundwork. On the other hand, maybe you just want to jump to the vision pro and talking about augmented reality on the apps and the product side of things. That's okay too. But regardless, we wanted to give you a fair warning. And so now the conclusion. Of augmented reality.

0:34

Ierengaym. com ierengaym. com

PJ: 0:46

It's really interesting for you to have connected wanting to take movie special effects, which as you put it, come across perfectly. But you know, one of the hitches with that is that all movies are basically all 2D images, right? So we do this, this sort of perfect combination, but it ultimately slams down to a 2D image. And I felt that was an interesting connection point to take us right back to mixed reality at this point in time. So we've done vr, we've done ar. Now let's talk about the magic that is the Vision Pro or what it's actually doing.

Rob: 1:31

The Vision Pro, it was announced for developers last Friday, the 19th of January, I believe it was. And it'll be for sale on, or at least for order on the 3rd of February. And then ultimately people will get these in the hands and we'll start to get reviews of them and all the issues potentially exist start to come up. It's gonna be very interesting to see how Apple, with their fully self-contained ecosystem. And how is this gonna interact with everything else? And I've seen a lot of this, I haven't seen it in a couple years, but, it's definitely gonna integrate with the ecosystem. So. That brings us to the question of what is the vision pro? Where did it come from? And there's always been rumors about Apple making glasses, and they're just gonna be a pair of super thin framed, ultra stylistic glasses that you can wear. And as we've talked about, it's not actually possible even for Apple because there's so much hardware that has to be in the head space and it will be tethered. You did make a thin pair of glasses'cause it would need to be powered. There'd be no room for batteries or things like that but the Vision Pro isn't a pair of glasses as we've all seen. It's, it's a headset. It's technically a VR headset it has no visible light from the outside world into the actual device and in ultimately into your eyes. In fact, it has a very good light seal around the device to stop outside light getting in. So why are we calling it a mixed reality device when all in intent and purpose, it's, it's a VR device and the reason is cameras to be specific. 14 cameras on that device. That's a awful amount of bandwidth to handle all the time. That's why it has custom silicon. Those cameras are dedicated to all different jobs. Some of them face out, some of them face down to look at your hands, some face looking at your eyes. Uh, some are infrared. There's a depth camera. There's many forms of cameras looking all directions, different frameworks, different resolutions. So. What they're calling mixed reality or XR or whatever other name they want to give. It is effectively going back to what I said about video and video playback on the Oculus in a standard VR device in real time. So we're taking live feeds from cameras, projecting it onto the displays, and so it looks like you're not wearing a headset. You could see the real world and. It's more than that because that's just pass through camera. Oculus played with this. There's various things that they played with years ago that people 3D printed things and put cameras on. There's a whole bunch of problems to solve here with projection and making sure that it looks the same as it would look if you were to. Not have the headset on. So you've gotta undo the distortions on the lenses, on the cameras and processor can put it back into your eyes. Now obviously there is latency there, there is a little bit of latency in that process. How fast can you do that? In reality? Very fast.'cause you really taking a frame from one thing, adding it to a display and that can be very quick. In fact, apple have a pattern mode. If it crashes, it will automatically, instantly Hardware. Pass through so you're not stuck with this device on your head that you can't actually see out of. So it's one thing that no other device has. Like Oculus, if it crashes, you end up with a crash still image on your face and you've gotta take the headset off. And in the process you might fall over. Whereas Apple have this watchdog system where if you don't respond to the timer quick enough, it will automatically in hardware go at this point. Instantaneous pass through mode. So it just looks like it's a pair of glasses at that point.'cause you're looking at the real world, but you're looking through a pair of cameras, you're not looking at the real world. Like I said, there's no way for light to get in this device.

PJ: 5:46

Earlier, Rob, we talked about 10 seconds as being that, you know, 10 seconds are under as being kind of that space you want it to be. Maybe 16, sorry, not seconds. 16. 10. 16 milliseconds to avoid the seasickness. So I take it that with the pass through mode, we're well underneath that at that point in

Rob: 6:05

Oh yeah. It's, it's instantaneous. it's it's scanned one and scan the other. It's just like boom, like super low agency. And so that's one thing it has and. In all intended purpose, it does all the things a VR device does. It's very integrated with the Apple ecosystem. Uh, it'll have iMessage support, it plays iPad apps in 2D space and all that. It has a head start over all competing devices, and it has Apple's name behind it, but it still has all the same problems. For example, if you open a web browser, how high resolution does the display have to be before you can read text on a window that's six feet away from you? And that's two 4K displays very high resolution, but nowhere near high enough resolution. To be able to read text at a distance, just put your laptop screen and walk across the other side of the room and see if you can read it. And that's actually a high risk screen at a distance. Now, make that a low risk screen effectively and you can't read the text. And text is incredibly difficult to handle in a VR device. All of the devices, up until now, magic Leap and Oculus all had this same problem we used to having a. Laptop screen, which is potentially higher than 4K, right? In a phase, it's a foot away. Now you've got that entire resolution. Rather than being a tiny little solid angle of your actual vision, it's now the whole thing. And which pro has a very wide field of view. So if you factor out like how many pixels per degree, that display needs to be like five times the resolution before you can reliably handle text at. Where you'd expect to be able to read it. Obviously, if you put a a letter piece of paper a hundred feet away, you're not gonna be able to read it in the real world either. So resolution at that scale becomes less relevant, but five, 10 feet away with a a big virtual desktop, you, you're gonna be wanting to be able to read the text and Apple do a very good job of it, but it's still gonna have some of these classic artifacts. That the other VR devices have when it comes to small objects door effect is minimized, but it's still there if you look for it. That passthrough mode that we just talked about is how that, in the keynote day, said that someone, if you are watching a, a movie in full immersive mode, someone walked in the room. It'll blend into the real world so you can kind of see what's going on in the real world without taking the headset off. That same pastoral system is how it's doing that. It's just blending Canberra images with virtual images and giving you that, uh, a a viewing to the real world. But, but again, that's still kind of vr. What's mixed reality? Well, mixed reality is, well, now we have this. Synthetic vision. Basically we have these cameras that are feeding in. We can render effectively augmented reality style on top of those camera images and display that in your face. And visibly it looks just kinda like ar. But in a lot of ways it's a lot easier because first of all, you can manage latency a lot easier. In true ar. The real world is the real world. It's gonna do what it does. In mixed reality, you can delay the frames from the camera up until the latency limits that are toleratable for VR basically. So if you take a frame from a camera that can be timestamped, this is exactly the time that frame was captured. You can render at exactly that timestamp, so there's no shimmying around in the scene. It's not like the virtual objects are gonna move with respect to the captured frame because you know the exact timestamps and you can render all that, process it all, and then display it 10 milliseconds, 20 milliseconds later, and the frame looks complete. So the whole virtual objects skied around due to the real world not having latency and the virtual object having latency. It's fixed. The whole thing now has latency in the same way that VR does and

PJ: 10:28

so to, put a fine point on this, we are cheating in the sense that we're delaying the real world, you know, 10, 20 milliseconds. And that's okay because that's still within the tolerance of seasickness, but we get to use that to synchronize the virtual and the real feed together.

Rob: 10:50

yeah. Basically we have a stream of frames coming in from the real world. None of them are visible in real time. They're all delayed slightly, and that delay allows us to render on top of them. And we still have to do everything we have to do for real. Ar We have to render, take two camera images, render stereo on top of those. They get projected in the glass. And now we, you've got this camera based reality with virtual objects in it. So the latency is a lot easier to manage. You still have to do the head tracking, head prediction and all that just to get the number down low enough. But, uh, the rendering side on the, just the purely technical side of this, generating an image is significantly better because you can have, I know this frame needs to have a 3D image at this timestamp, and I can blend them perfectly. You still have all the existing problems that we talked about with true ar. Just because you are in camera space instead of real world light space doesn't mean the occlusion problem goes away, depth clipping and all things like that still needs to occur

PJ: 12:00

The shadow problem's easier though.

Rob: 12:03

the shadow problem is now fixable because. The image is now virtual. The camera image is just a pixel based image. It's not the real world. So we can subtract light from it, we can replace pixels with black, and if we have a fine enough understanding, then lighting becomes much easier because we can do blended light, we can do subtractive light. It's not just additive like it was in the true AR sense. So visual effects get better. Dynamic range gets better'cause. We're not just adding light towards, it's already there and you still have all the same problems of if a light's flashing in the distance and you've got a shiny object, then that shiny object needs to reflect the real world. And at least when I was Apple, they had a lot of research going on into things like this as to like how to make this thing interact with the real world

PJ: 12:59

the scene understanding problem remains.

Rob: 13:02

understanding remains, it'll always remain. It'll just get better over time. It's like AR without scene understanding is useless. But has Apple done enough to make this a consumer friendly device? And I think they have. A few tricks up the sleeve. They have the whole integration with the ecosystem. They have, integration with various immersive movie players.'cause now they're gonna take full advantage of this 3D movie capability. I do think one of the best use cases of this device is movies, watching movies and maybe on a plane.'cause it's like completely immersive. You get 180 degree. Field of movie experience, you could kind of move your head around a little bit and it'll move the image to give you that immersive that you're kind of there. They can also do the full 3D movies if they want to, but, the use case for that is difficult. Think Apple will restrict the technology to highlight their use cases, even if it's not the pure technical use of the technology.

PJ: 14:04

So what's fascinating about that use case you just mentioned is that you don't need mixed reality to do the 3D movie case. Like you could have a VR headset to do that just as

Rob: 14:18

Yeah, absolutely. Like it's, it's a VR mixed reality device. It's the mixed reality just comes from projecting the real world into the VR space. Uh, that is mixed reality and rendering on top of it. If you want to, the movie use case could be done by Oculus and all that, but did Facebook and Oculus prior to Facebook, didn't have the connections that Apple have and didn't have the ecosystem to integrate into. So I'm gonna be very interested to see what the ecosystem is of. How does this thing integrate. Do any of these make a compelling case for a must have application? The killer app as we call it? I don't think it does. I don't think this device would be a hit. I didn't work on it'cause it was gonna be a hit. I worked on it'cause it was cool technology and I still don't think AR is ready for the consumer space, no matter how you spin it. I do think they'll have better apps than everyone else has had. It'll be a step in the right direction, but it's not gonna be a. Like, whoa, look at this. It's, it's, Apple's gonna do it all better than everybody else. The betterness will come literally from the ecosystem and the integration with that ecosystem. Not because they did anything groundbreaking, different than anyone else has already tried.

PJ: 15:30

I sort of stick on one particular point,'cause we talked a lot earlier about the different focal planes as being, you know, one difference between what VR was doing is just splatting it onto a screen versus what we discussed with Magic Leap of attempting to have sort of multiple layers of lenses that describe effectively multiple focal lengths. The Vision Pro does not solve. It's still one focal plane that we're talking about that you're splatting the image onto.

Rob: 16:00

Yeah, it's very difficult to solve the cure. Focal plane problem in a VR or XR space.'cause ultimately there is a screen in front of your face and you can do lens in, in front of that screen to project it at a different depth. The problem is, is your eyes are still focused on a plane in front of you. You take the real world that has. All the variable focus and everything you sp splitt it onto two camera images, which makes it flat. Then you put those flat images on, two flat monitors and you visualize it as a, a fully 3D scene and your eyes know that's not true.'cause they're focused on a flat plate. And if you're moving your eyes around the scene, your eyes are not changing focus based on distance as they would in the real world. They're focused purely on. The screen that's in front of you, even if the distance information that's coming from that image is changing and it's like looking at the real world through a dirty window, your eyes might focus on the glass and the real world's just back there. But in that case, it'll all be blurry. It's'cause your eyes focus of, uh, close up and that will unfocus the distance. These are weirdly, infinitely focused at a, a very near distance, and it, it can cause eye strain and again, it gives me a headache pretty damn quickly.

PJ: 17:16

You mentioned it earlier in terms of the resolution required, but I do think it's worthwhile to talk about some of the numbers. You talked about it as like a multiple, but in terms of like what would be required to actually, uh, have what you call the ultimate screen resolution given. You know the limitations we have. I would love for you to kind of like reveal exactly what you calculated out, how many pixels you'd need to actually make it look, you know, real world I.

Rob: 17:47

So pixels become a weird number at some, uh, in this space because it depends on the field of view. If you have a 1920 by 10 80, 10 80 p resolution screen, and you have, let's say 180 degrees field. View effectively, like that's only about 10 or 11 pixels per degree of field of view, just there's nowhere near enough. They always say, and I think this number is also completely wrong, you need, effectively 300 pixels per inch of one foot, and that is. Not even close. Ideally, you need probably about a pixel per arc minute is what they say. That isn't really correct, but even that gives you a horizontal resolution of about 10,000 pixels. So 4K is realistically nowhere near. The resolution we need. Now, of course, you can reduce the field of view. If you make it 90 degrees, that could half the size of the, of the screen, but now you're limited to this 90 degree field of view and not the usual peripheral 180 degree that we have in natural vision. Technically more than 180, but 180 would be a, a great step in the right direction. But yeah, for 180 degree field of view, like a full hemisphere in front of you. You're looking at over 10,000 pixels in each resolution, which would be a hundred million pixels in the entire display. So 4K, eight K, yeah, whatever. Doesn't matter. We are nowhere near. So the screen door effect will exist in some capacity at some distance with some content. It has to. There's no way around it.

PJ: 19:35

So for the foreseeable future, AR will remain relatively near field. And you know what? And that might be within, I don't know, just a few meters really. Right.

Rob: 19:47

I mean within a room, I think is the vision Pros use case, and I think it'll be near field. Like you say, if you do a virtual avatar for chat, it'll be someone standing at the same desk as sit at the same desk or standing next to you or something along those lines. It's gonna be very close, but the question remains, what if you take this thing outside? People will, I guarantee you, I guarantee you someone will driving it. They shouldn't, but I guarantee they will. And it's all the killer use cases for AR outdoors. I want a navigation app that will draw on the road and be like, turn here and literally draw in the turn lane where I need to go. And I literally follow the yellow brick road. Um, that needs to work with occlusion. It needs to know about moving object, high speed, moving objects. Um, you've got all the weather to deal with and. All the different outdoor things that we encompass, which we don't see indoors. There aren't many killer AR use cases for indoors other than what hopefully Apple will tackle as media playback. But I think you have a lot to say on this is movies are supposed to be social things. I don't want to sit watch a movie with a headset on my head with my girlfriend next to me with a headset on her head. Watching potentially something different. I have no idea she's even watching the same thing. And it's the same problems 3D TV's had, but much worse. Like 3D TVs had their, their thing in the two thousands where your friends would come over, you'd all wear a pair of glasses, and you'd watch these awful movers in 3D. And if you didn't have a pair of glasses, the movie wasn't even watchable. It wasn't bearable to look at the screen. So it made it very antisocial for people who weren't. This is a step in the wrong direction because now not only is it anti-social, you've got your head covered and everybody needs one of these things. Again, you might or might not be in the same environment doing the same thing. It just seems like a, a dystopian use of technology. I get the use case of you're on an airplane and I can just watch a movie fully immersed in it and great. That's one very specific use case for a$3,500 device. I don't see me choosing it over, laying on the couch, watching the TV with the speakers blaring and the fireplace on, and my dog and all the things that I want near me when I watch a movie. And we've talked all through this podcast. We've only been talking about graphics. We haven't even touched on audio. And audio is a huge part of the immersive experience, and I believe the Vision Pro is gonna ask you to wear EarPods it. It has speakers, but. They can't be very good'cause they're not in your ears. It's a headset, so it has kind of bone UCT speakers. You can hear, you can definitely figure out what's going on, but truly immersive audio is gonna require more than what it has. It's gonna require a set of AirPods or AirPods Pro or something on your head. I doubt you can do the pros. Maybe you can. It's gonna be a lot of straps and a lot of things on your head. Uh, but, uh, the maxis, sorry, not the pros. AirPod Pro, what you're gonna have to wear to get decent audio, and even that's not great audio. I'd much rather have my HiFi, big speakers play an audio in the room that everyone experiences. I still don't think, no matter how good you make the audio, you, unless you have really high quality headphones, then. You're gonna get the experience of a, a true Hi-Fi system, and headphones are going down that path of being antisocial again.

PJ: 23:21

This dovetails really nicely into kind of like the, your question that you've kept bringing up, which is like, where is the killer app and really where is the killer app for the technology where it's at today? Because as you point out. So much of the killer apps would be, far field, uh, ar outside, you know, navigation and if we're containing it. And a lot of the, the demo is really like, focus on this if we're containing this into a space that's primarily entertainment. I started doing a lot of thinking around the history of media, or the history of entertainment really. Where you can go back as far as, even a prema, technological mass media days of, you know, your, your mass media may have been plays or newspapers where you had effectively this communals, you know, whether you're all in a, a theater together or you're watching something and there's this kind of unifying effect and then. When you have technology, so you have radio at that point in time where it's not even confined to a given locale because you can have stations carrying it across a nation. So now you have, again, a communal experience that's being experienced even with folks that are not right next to you. Then you have television. You have movies, you have these, Places where you're gathering together, either outside your home or inside your home to experience some media together. And I think we've, I mean, what's really fascinating when I think about what's happened with social networks, because I think that probably had aspects of mass communication that were similar, but over time they have become very hyper-personalized. So we're, we're, we've been down this road. Where it's like no one's seeing the same thing. We're all getting, you know, different entertainment, different news, different personalized feeds, and you could look at like the emergence of what were happening with, what's happening with the Vision Pro, where it's a device that is really isolating to a given person. And the same could be said of, of any other VR XR headset. Because it really is just like you alone experiencing that unless you decide, Hey, I wanna buy, you know, a$3,500 piece of equipment for everyone in my family, which is a pretty cost prohibitive thing. I feel like this kind of isolation, it's gonna be an amazing tech toy for a lot of folks, I think. And maybe there's a professional class that is gonna get to use this and it'd be. Really useful in terms of their work. Maybe they connected to a keyboard or something like that. And then you have, you know, giant sized monitors. I don't see the intersection right now with it being a mask market consumer device, and I, I see that as being a big barrier to its consumer success.

Rob: 26:39

I completely agree. I think for$3,500, it's out of the price point that pretty much anybody's going to be. Well, well, let's just see what it's like. If it's 500 bucks, a thousand dollars. I think Oculus is price points. It is in the realm where tech enthusiast be like, yeah, I'll drop$500 on that just to see and if I never use it, whatever. But, uh,$3,500, it's out of most people's. play money fund to uh, to just go and see. I mean, maybe the Apple store will have a bunch of them. You can go try'em on there. Again, without that killer app, I don't need an A 3D avatar to chat. I can just chat on a keyboard. But the integration's nice. It's nice to see it happening. It's step in the right direction, but it isn't, I think, a step that I'm very interested.

PJ: 27:25

Do you have any sense for whether this is, I mean, uh, so again, I have a family of six. Uh, if I wanted to buy a Vision Pro for everybody, that would be$20,000. If I wanted to buy an Oculus today, I think it's 3000. So for less than One Vision Pro, I could equip everybody with a Meta Quest three.

Rob: 27:48

Or you could just have a TV on the wall and you all watch TV together

PJ: 27:51

or I have a TV that probably costs less than two grand. do you have any thoughts, like, is this a loss leader for Apple? Like this is a, you know, high-end concept car, like Mercedes-Benz might put out with the idea that this technology will get then. Filter down to lower end units. That makes it more mass market. It. It's not necessarily Apple's strategy, but I, I do wanna try and explore whether there's something we're missing here.

Rob: 28:18

So when I was at Apple, I asked some hard questions and never got any suitable answer, and some of them relate to this. The first obvious question, which many people have asked is, if this is a headset device with a puck that's in your pocket, even if it's wired, why is that puck not my phone? Why is that? Puck another piece of device that has effectively a cell type cell phone type processor and cell phone type architecture running an OSS that's derived from the same OSS as the cell phone. Why isn't it that? And the answer is the 14 camera streams of like there is custom silicon in there. So is there a low end version that's just a display that's coming out that drives your phone, that has less cameras and things like that? Then, I don't know, it wasn't talked about. The other thing that, is difficult in xr, in fact all of them, vr, a, ar, and, uh, XR is input, my head's covered. I can't type, so what sort of input device do I use? Is it accurate enough to see my fingers so I can type on a virtual keyboard? Things like that. I think you avoid those use cases by going down the media path. And is there a lower end one? I really don't know. It's called the Pro Vision Pro. There could be a vision. Looks like there's a MacBook and a MacBook Pro. Uh, I, I don't know what their future plans are. It was, this product was still in development when I left. So I don't really know. And if I did say if I did know I said it, apple will probably take this podcast down before we could even get it up.

PJ: 29:55

Yeah, yeah, yeah, yeah. I'm not, uh, no, no need to like, break any NDAs or anything like that. This is, this is speculation. Part of the question comes from I remember listening to John Carmack chatting with Lex Friedman and discussing his time at Oculus. And there was also news articles that came out. So this is not secret that there was conflicts between Carmack and Zuckerberg in terms of the particular realization of VR with CarMax saying we should actually have much lower cost units. So what's fascinating to me is that, compared to a Vision Pro, you know the Medi Quest three at$600,$500. It looks vastly cheaper, but it's fascinating to me that Carmack actually wanted to have it even cheaper than that to have a mass market device.

Rob: 30:51

I agree with'em

PJ: 30:52

yeah, I I I, think this is an interesting, like, bullet point though.

Rob: 30:56

I think if you pick the technology very carefully, good screens, low latency processing. Fed may be from a PC instead of from. Another computer that's less hardware to buy already have a pc, then you can make very high quality, fairly low cost VR devices. With modern technology, it's when you start putting in custom processors and standalone processors, the cost inevitably goes up. You've got the hardware, you've got the development costs. You have everything in there that you're feed in, audio, you're feeding 14 cameras custom, five nanometer silicon, blah, blah, blah. The cost has to be high. How much of it is actually necessary for the killer app? The killer app could be done without most of that. And obviously the, the low cost VR device that CarMax talking about doesn't have xr, it has no cameras. Uh, so is the killer app just vr? Do we need the cameras? Again, I don't know the answer to any of these questions. I'm more of a technologist on the AR side, VR side, more than a use case person, and. Made plenty of tech that's useless, but really cool tech. And this unfortunately might fit in the same category in a much bigger scale. Going back to some things you talked about earlier, the social experience for, ar vr, like you talked about, I think Sony did a good job here with PlayStation VR because they're device as in HDMI out so you can feed the TV with what you see in it. I think it feeds the left eye to the TV screen. So all the people in the room could help. They can be like, turn left, do this, blah, blah, blah. Makes it a more social experience. Only one of you is getting the full experience, but it's a step in the right direction. It opens up the living room to more people being involved in the experience. So you could switch the headset around and all play and all switch roles. I think that's a very good thing. Uh, I think vr, uh, carmack and. Oculus do some of this too, because the PC monitor can have a copy of what's on the

PJ: 32:57

Yeah, you actually can kick it out. You can cast it to a Chrome cast

Rob: 33:01

Yeah. So you could do all these things. I don't know if the Vision Pro can do that. I assume it can. I assume it to be able to take the same thing, take one eye and then airplay a two, another Apple device. So it's more social. Hopefully. Yeah. Hopefully that's the case. It opens up the same way that the PlayStation originally did with its HDMI connection. So again, I don't know if that's really supported or will be supported or down the long road of things that, uh, they intend to to implement.

PJ: 33:36

It is interesting that the media examples that I recall seeing from the keynote really were focused around. Playing movies, playing media, a little bit of desktop stuff and the, or the tabletop stuff, I should say. The tabletop ar. One of the areas that I felt was underrepresented was games. And I feel like it is, slanting more onto the VR side than the AR side. It is one of those like natural applications that one could imagine where. Maybe your phone becomes your controller and you are, you know, you're seeing everything through that headset. There's a part of me

Rob: 34:14

There's lots of use cases and I'm, I've obviously, AR Kit will be fully supported in its new form with the, uh, inside view of AR for the, uh, StereoVision headsets. Point of view. So developers will be able to make games and will be able to make their own content. But I think they took away some of that from the keynote because of the scene understanding problem. These problems are not solved. They're not solved by anybody anywhere, although you can technically solve some of them altogether. Not a solved problem. So I think you'll see AR content creeping in. It'll be in advertising. Apple do their little AR displays for the keynotes and things like that. Obviously that will all be there and, but it's all tech demo. It's like, yeah, it's like once you've seen it, you've seen it like, okay, don't need to see that again. I do think in the future, like SS AI will play a much bigger role in AR because it has to, and I think games, games, are hard because of the whole. Mobility thing of like if you're dancing around with a headset on your head, then you've got the whole wall of a fall over problem and the motion problem if, I mean, it is a mixed related device, so effectively you could walk around your house and solve puzzles and things like that. Do you want to,

PJ: 35:34

Yeah. Is that actually

Rob: 35:35

There will be use cases for this and people will experiment with it. I assume new use cases and things that we thought would not be fun end up being fun and vice versa. So I think developers will run away and do cool stuff. I hope Apple give them the ability to do so. Apple's problem with developers, and this may be why there's no games for the Apple ecosystem as a whole, at least AAA games, is Apple always assume they are the smartest person in the room. They give you SDKs. Do exactly what they thought you want to do. They don't give you tiny building blocks where you go, go and break this. Go and see what the limits are. It's all very carefully constructed kits where if they haven't thought of it, you can't do it. And that's one of Apple's biggest things. And they are absolutely not the smartest people in the, and some of the big game developers are like, I don't wanna play with that. I also think this came out in the news this last week, is they don't really have the media system pinned down either, because there won't be a Netflix app, um, Hulu app and

PJ: 36:36

A Spotify app either.

Rob: 36:38

or Spotify app. Well, Spotify is kind of that back on the audio pathway. I don't think Vision Pro has great audio in its natural form. You have to go buy all the stuff on top of$3,500. It's gonna be a big problem. And I don't think it comes with a pair of AirPods. So you are gonna buy more stuff if you want the full experience. And for Spotify it's purely audio. I, uh, could see why they wouldn't wanna do it. And 3D audio is just as old as 3D graphics.

PJ: 37:05

This is an article actually that just popped up. So Disney, believe it or not, is working on the VR treadmill that supports virtual users. From what little I've been able to tell about it, uh. I think it's almost like it looks to me like beads on the ground that you're like walking over. So that'll be really interesting to see if that ever makes it to a home environment.

Rob: 37:31

See in an urban environment, you could do that. You could have like, oh, I could do a Pacman game, and I'll just, I'll Pacman, I'll just run around and pick the dots up or something. Now take that outdoors, make that into, I'm now skiing down a hill and there are rings that I can ski through and it's gonna teach me to do various tricks and turns. It's all went good until you ski off the edge of the run and go off a cliff and die'cause the AR was not quite correct and those things happened because of the depth problem. Uh, I'm a skydiver. It would be great to have like information in my headset so I could see what's going on and I could do it with a head up display. It'd be great to point, like, okay, the drop zones behind you type thing. It takes the fun out of it. But there's many use cases or the quasi indoor outdoor ones I want to watch a video of fixing a car. I have some vintage cars. I like working on them. I want a video right now. I watch a YouTube video. I go and do it. I watch a YouTube video. I go and do it. How about I can watch a YouTube video while I'm doing it and it marks up various parts on the engine, like replace this part, we replace that part. And again, scene understanding, am I even looking at the car when I'm looking at this video? It's like, should it even play the markups if it's not sure I'm looking at the correct thing. So I think there is new technology to be invented and I think getting these headsets out enables that to be invented. It could be interesting to see if that could be standardized into things that multiple things can play. But as we know, apple doesn't play nice with others. So even if they make technology like that, it would only be on their platform.

PJ: 39:10

One of the fascinating things I've found about looking at Magic Leap, looking at HoloLens, looking at Apple at this point in time is. They very much seem like very cool pieces of tech that are in search of a problem to solve. And you know, where we've been able to say, oh, okay, this is like even through my phone, like tabletop ar. It's like, okay, cool. But I'm not using that on a daily basis and it's not something that's like particularly compelling for me on a daily basis. So I, I do think that there is like this rich garden that has been created of vision technologies, of graphics technologies, of pipelines, of tracking. I think it's awesome. I just don't quite see how it, it intersects with my daily life yet though. And that's been I think one of the big, tricky problems with all of this.

Rob: 40:03

I've said that from day one. Exactly that. Like I said, I'm a technologist. I work on this stuff'cause it's fun to work on. There are some cool problems to solve. The use case is not my problem and I still haven't seen a compelling one. Even knowing how the technology works, it could hint, like, push it this direction'cause. Fits the technology. It's still an awful use case, and it's, it is what it is. It it, it's coming out and we'll see what happens.

PJ: 40:30

Well, not to put too much of a pun on it, but I think it's a thing we'll be tracking over time. To see how well this thing actually like evolves as the products get out there. And who knows, maybe by end of year there's something magical and amazing and it's like, oh yeah, everyone needs a, a Vision pro or an XR headset and we'll see what happens.

Rob: 40:52

Yep, I agree. And before we close, I just want to go back and address some numbers. I just did some back of the envelope. Calculations while we were talking and I was off by an order of magnitude on the required display resolution.

PJ: 41:05

Okay, so we said 10,000 pixels, which we're still far from right now. What is the actual number you think, Rob?

Rob: 41:13

there's not, again, it depends on field of view, but if you

PJ: 41:17

180 180 Field of View.

Rob: 41:19

if you want a 180 field of view and you don't want to have any of the, higher order perception artifacts that. That obviously occur when you have lower resolution screens, like motion artifacts and artifacts over time and such like that you potentially need maybe a hundred thousand by a hundred thousand pixels. So I was literally off by an order of magnitude

PJ: 41:47

each direction,

Rob: 41:49

in each direction, in each eye too. So, uh, I don't think I'm gonna be seeing a. A hundred by a hundred K display in my lifetime

PJ: 42:00

uh, especially at the size it would need to be at, in order to be on one of those headsets so there's some technical challenges still to come. And hopefully we can derive some utility in the meantime.

Rob: 42:11

yep. And one more thing also I wanna point out is you mentioned the tabletop AR from your phone. That is mixed reality. That is not ar. That is true mixed reality. It is no different on your phone than it is in your headset, except you are doing it with different tracking. You're doing it stereo and things like that. It's the same problem, the same things, and I think you'll get the same contrived use cases when you try to do it in a headset.

PJ: 42:36

Yeah. Great, great point. You know, you mentioned earlier, why not just use the phone? And I'm curious, like, do you think there's actually an opportunity for, let's say the iPhone 15 and above is now, you know, high speed USBC, which I think supports 10 gigabits per second,

Rob: 42:57

Not even close to enough.

PJ: 42:59

not, not a close to enough. Not for VR

Rob: 43:02

14. There are use cases for it. Yes. I mean, going back to CarMax point, you could make. With the available technology, you could use your phone as a base station and have a certainly VR experience.

PJ: 43:15

Yes. Yeah.

Rob: 43:16

You could potentially do an AR experience with lower resolution cameras and fewer of them. Again, what is the use case? Do we need what it has? I think carmack hit the nail squarely on the head where a, an apple headset that just fed from your phone. Would sell a lot more than a$3,500 headset that has no use case. Right now,

PJ: 43:39

Well, what I'm thinking very specifically about is the media use case where if we're really talking about, I wanna watch a movie on a plane. I don't need any cameras for that. Fundamentally, I don't need cameras looking at my eyes. I don't need cameras. Looking at the outside world. I just need displays, and if I could just pop that under my phone and then watch whatever I wanted, like, okay. That covers like the media use case that everyone's talking about right

Rob: 44:07

That is Carmack's argument right there. That is it. You can do the Killing app that we have right now for a lot less and a lot easier than what we're currently doing it, dad, we can do it for it. Significant cost reduction and get that same movie experience. All we need for that movie experience is the display and potentially the head tracking. If there's, uh, any 3D component to it where you can kind of move your head around or interact with the scene. All that's cheap and easily doable and would be perfectly doable from an iPhone. Drive such a display. Why they don't do that? I don't know. I don't think all the extra hardware adds much to the use case because that is the app that is the only use case that I can see the Vision Pro having. And we can do it much, much cheaper and I think others will. It Won't have the Apple ecosystem to back it. And that may be ultimately the killer feature is the Apple ecosystem and not any of these little Tech demo apps.

PJ: 45:08

I think this is gonna be the big question, is the ecosystem enough to basically overpower all of these obstacles that we've talked about for ar, and it's gonna be an interesting test case, frankly.

Rob: 45:20

Are you gonna buy one?

PJ: 45:21

I am not as of yet gonna buy one. Uh, we will see, uh, I will admit I do wanna play around in this space. So I did go the cheaper route and I just got a Quest three on order.

Rob: 45:33

Nice. And I dunno, either, I don't if we're gonna buy one, I don't really fancy drop$3,500 on one of these things and hopefully maybe they give one. I don't know. We'll, I'll, I'll prod around, see if I can

PJ: 45:46

See if we can get one. That'd be, I think it'd be fun to do like some sort of bake off between the two making it like the ecosystem be the centerpiece. Like, is, is this enough? Like, I think that's a really valid thing. Well, I have a feeling we're gonna be returning to the subject. At least once or twice more this year as we see the results of these devices. Are they catching on? Are they not?

Rob: 46:11

Yeah, I can't wait to see what people say. It's gonna be real fun to see how it pans out and see how pan my technology gets.

PJ: 46:18

At the end of this, I think there's a two B continued dot, dot dot.

Rob: 46:23

I think so.