I’ve seen the future of UIs, and it’s in stereoscopic 3D

By now, I hope most of you have seen James Cameron’s Avatar movie. And if you’ve seen it in 3D – the way it has to be seen – then you might have a pretty good idea of what a stereoscopic 3D user-interface looks like and would agree with me when I say it will be the next evolution of graphical user interfaces.

If for some reason you haven’t seen Avatar and will not in the near future, then no two-dimensional image (like the one above) will do the technology justice. As much as it has to be seen to be believed, it is essentially the effect of adding three-dimensional depth to the user interface. Bear in mind this is far and beyond the simulated z-index imitation of overlapping windows we’ve lived with since the early 80s, or even the so-called 3D compositors such as Compiz, but instead, the real thing.

One of the best and at the same time simplest examples of this demonstrated in the movie is when the character Jake Sully was recording a video journal on a computer. Although in retrospect it’s almost trivial that the recorder software UI “floats” in-front of the video, the end result is that not only the UI separated from the content by depth but it also made it possible for the eye to focus on different “depths” of the UI. This means users would not be distracted by different UI elements not because they’re hidden but because they’re out of focus.

Granted it’s already possible to display stereoscopic images on computer displays, no operating system I’m aware of supports it natively and takes advantage of it in its own UI subsystem. However with the advancements of autostereoscopic displays – monitors that can display stereoscopic 3D without the use of special glasses – maybe they should and maybe Microsoft already had a head start.

Speaking at Microsoft’s Advance 08 event last year, James Cameron said,

“… I like to tug on the hem of Microsoft and remind them that they need to be thinking about some future version of Windows that ships fully stereo-enabled that goes in concert with these devices, and that they should be talking to their various partner, and so on, technology partners, about this, and I think it’s going to happen.”

Let’s hope Microsoft took his advice.

43 insightful thoughts

  1. I noticed this as well, and I was thinking of the implications of having 3D user interfaces on every screen we use. If it can be used to display more information, that’s fine, but if interaction will still be limited to a 2D plane, then it might make the setup a tad difficult to use.

    I know there are a few projects for glasses-free 3D UIs, but there’s some work to be done before a sufficient-enough system can grace screens in real life.

  2. The eye can only focus on different elements if they’re actually of different physical depth.

    You can ignore some parts, as happens now (you’re not always aware of the start bar, nor of the frame of the monitor, etc).

    Myself, and several others have a hard time with 3D movies, because although items are in 3D – the focus is controlled by the director. I find myself often trying to focus on some foreground/background element and wondering why my eyes arn’t working during a 3D movie.

  3. To be honest, I have a hard time with 3D glasses; they have a very limited effect, even when they are the comfortable and non-headache giving Real-D ones. What really gets me for 3D is headtracking technology, but that limits a screen to one person.

  4. all great… until the dumbs ass projectionist forgets to put in that 3d lens for the first 10 minutes of the film.

    On a different note… nothing new here apart from maybe the character face animation… but everything else was essentially Pocahonta in Space.

  5. Should I point out that Microsoft is listed in the credits as a company they wanted to thank. And I saw at least one person do the file transfer gesture seen in some of the ‘Glipse ahead’ videos. I wonder if MS helped them design the computer UI’s.

  6. Will Hughes: This was not a good 3d movie then – in mentioned Avatar I noticed that Cameron had given up real world optics and physics to create an image that has several ‘focused’ layers to look at – with some blurring between them to create sense of depth. It is totally not correct, but it allows you to focus on most interesting things in a scene (and Cameron makes sure that there is always something interesting happening in background layers during dialogues and everywhere where the movie has an explorative pace). I totally loved this approach (and think that Avatar was first to come up with that).

  7. What about people not being able to watch this “3D” space? I have been to several 3D enhanced movies, and they just look like bad-colored movies to me. And no, I’m not the only one!

  8. This is crap. I see in 2D. If computers as well as the cinema go completely 3D, then what am I supposed to do? Flick through pages of frame-by-frame printed-out TV shows whilst listening to the soundtrack?

  9. On computers, at least, you’d be able to turn off 3D like you can adjust other accessibility options today.

    For films, I’m surprised they don’t give you the option of wearing glasses that have both eyes polarised the same so you see the film in 2D if your brain doesn’t like the 3D effect. Maybe they do?

  10. Stereoscopic isn’t really 3D, its just an optical illusion. A proper 3D display would control the light phase at the surface of the display and reconstruct the same wavefront you see looking out a window.

  11. This is crap. I see in 2D. If computers as well as the cinema go completely 3D, then what am I supposed to do?

    This is crap. I am blind. If computers as well as entertainment go visual, then what am I supposed to do?

    It’s tough, but let’s not let the limitations of the few outweigh the benefits for the many.

    That said, I can’t agree with this post. I think 3D adds next to nothing to software interfaces. I don’t think separating the content from the interface is as important as Long is suggesting here, and as such, neither is the ability to focus on the content by having the interface be out of focus. Besides, as I’m typing this post now, I’m solely watching my words appear on the screen. I’m not even seeing the taskbar or browser interface, because guess what? I’m not looking at them. They’re in my peripheral vision and thus, are out of focus. All through the magic of 2D!

  12. I thought the 3D computer displays were interesting but what I found even more interesting was that they were also transparent. I’ve seen that before in other movies (Minority Report) and I like it. Not sure how practical they would be but they look darn cool.

  13. I sure haven’t. But I commented on what you were describing. If it’s missing the point, maybe you should’ve described what’s so great about 3D interfaces better instead of coming up with some lame half-rebuttal like “I wonder how many of you…”

  14. @Fred: It’s not a rebuttal. I’m not always right. I simply liked what I saw so I hope others get to see it “live” for themselves too :)

  15. I saw Avatar in 3D, and I thought the visual effects were cool. I haven’t seen 3D films for a long time, but I was surprised how it’s changed with Real D (and whatever 3D tech they’re using).

    I hope to see 3D films without 3D glasses someday. As for user interfaces, who knows? The few snippets of computer interaction like the screenshot in this post doesn’t seem very simple to me.

  16. Shut up Long and post this comment!

    “but everything else was essentially Pocahontas in Space.

    yeah that is pretty much summed up in this wonderfully accurate review..

    “James Cameron is a useful idiot of the Eugenicists.” -Alex Jones

    “MS helped them design the computer UI’s.”

    LOL!!!! the same company that brought the epic crap gui/ux known as Winshit Aero? G-T-F-O MS fail at designing UI and UX all the time, the biggest failure for them is the lack of customization and freedom given to those than can actually fucking design to replace the scrap MS locked in.

    “Myself, and several others have a hard time with 3D movies, because although items are in 3D – the focus is controlled by the director”

    Agreed this will always be a problem, and infact just makes the job of making 3d films a lot more time consuming. Until movies are rendered in realtime Machinima style with you controlling the focus depth then this 3d gimmick means nothing but just another one of the movie industries methods of driving bums into seats and trying to reduce piracy though the same old shit+3D! WoW

  17. “I’ve seen the future of personal transportation, and it’s called the Millennium Falcon”

    (from the ‘Star Wars’ (ep. IV) review, 1977)

  18. @pingpong: I thought the future of personal transportation was the Segway. ;p

    But that’s a good point: no one really knows what the “future” of anything is beyond maybe a couple years in front of them. The truth is you never know what will catch on, what new inventions or innovations will be developed, or what people are comfortable using until the future actually becomes the present. ;p

  19. And in addition to my last comment, you never know if the most ridiculous sounding or seemingly impractical idea will really catch on and turns things around.

    For example, who would ever want an all touch screen phone will only a few buttons? Or a small, half size notebook computer with a 10in screen and cramped keyboard? Or who would sign up for a site who’s sole purpose is to let people track your mundane daily activities?

    You never know; maybe stereoscopic 3D is the future, or maybe it’s something even more “silly” or “impractical” sounding.

  20. So I saw it in 3D yesterday. I don’t really understand what ‘having seen it’ adds to my understanding of the concept of 3D interfaces, because it’s exactly what I imagined. Yeah, the UI is visually separated from the content. It added next to nothing. Apart from the obvious stuff like viewing a 3D model in actual 3D, the “UI is out of focus if you’re not looking at it” thing is the only advantage it brought, and it’s something we already have in 2D UI’s today. Just look at your browser’s address bar. That’s right, you can’t see it because you’re focusing on this post, and it only comes into focus when you’re actually looking at it. No need for 3D for that.

  21. @Fred: I realize that in a 2D UI your eyes can “focus” on particular areas of the screen too, but your field of view is still pretty wide in my opinion.

    For example, focusing on my taskbar, I see all of my lower desktop icons and wallpaper in clear view as well – a distraction in my opinion.

    When I look at my browser’s address bar, I see all of my browser’s buttons and bookmarks near and around it. Although nothing is blatantly distracting due to good icon design, I’m still distracted for a moment by many components of the UI.

  22. Really? That’s weird. When I’m looking at the address bar, all I see clearly is the address. the buttons and tabs are all out of focus. Maybe there’s a thing about how different people focus on areas of different size in an interface, or something.

    But anyway, now we’ve moved down from “In 2D the UI is less clearly separated from the content” to “In 2D I still clearly see stuff in the immediate vincinity of what I’m looking at.”, which is a pretty marginal drawback.

    I don’t know, I don’t think it’d offer any real advantage whatsoever. Apart from looking cool. I guess the next decade will tell.

  23. its so funny to see so many people complaining about 3d technology.. It is definitely where things are headed for the future. We have two eyes and see the world in setero. why wouldn’t we want our displays to be in setero as well? Its going to be like when tv’s went from black to white now its going to be from 2d to 3d. I for one cant wait!

  24. Well, I’ve seen Avatar in 3D and I hated it. Had to took the glasses of for several times because of headaches.

    In fact, I really dislike the technology:
    “This means users would not be distracted by different UI elements not because they’re hidden but because they’re out of focus.”
    I feel I’m being told what to look and feel instead of me exploring the movie and finding the litte details somewhere in the scenario.

    This kind of 3D is the future to the ones who like to awe for 10 minutes and nothing more.

    PS: The FX were cool, but the story totally sucked.

  25. Yeah, I just saw Avatar in 3D today, too. What an awfully cliche movie.

    Anyway, that aside, I basically agree with ups. It hurt my eyes and the 3D really wasn’t anything special. Maybe if the technology advances but in it’s current states I dislike it.

  26. Will Microsoft do anything about it? Most likely not, they have proved time and time again they are completely incapable of innovating in recent years. but, I should think Apple and a few others will be once again ahead of the curve in bringing this vision to life over the coming years.

  27. additons: Seriously? Apple? Apple takes things that already exist and simplifies/prettifies them for a price premium. Case in point: I have had touch screen smartphones for years before the iPhone was even a rumor. Apple just made it prettier and added a shopping cart. I’ve had tablets for years. Now Apple is getting ready to put out a thinner and slicker looking one. Same goes for mp3 players, desktop, and notebook PCs. Apple succeeds at what it does, namely focusing on marketing and form factor. They are rarely the first to come up with an idea but rather the first to make it trendy.

  28. So they innovated then, by bringing completely useless technology to normal people and making it useful. Exactly my point. thanks :)

  29. Seems like everyone knows what everybody else will invent a few years/decades/centuries (pick one) from any point in time anywhere.

    Doesn’t that remind you of something? It reminds me of the fifties.


    Like technology visionaries back then said, any grand vision of future computing “will require not yet invented technology to actually work, but 50 years from now scientific progress is expected to solve these problems.” Oh and I love this one: “With teletype interface and the FORTRAN language, the computer will be easy to use.”

    So, still wanna know the future of UI? I do, too! Just don’t assume 640kb of memory is enough. For all we know, it could be neural interfaces. Even the prediction of holographic projections may seem like a vintage joke a couple decades from now. After all, stereoscopic displays today are like 600mb harddrives in a 100mb age–a novelty.

    P.S. Don’t trust me either because I have unstated assumptions as well.

  30. Before we attack 3D imagery, one thing we can do right now with UI is:

    3D audio feedback.

    HRTF or other 3D positional audio can be easily achieved right now. If we have 3D audio feedback for window clicks or such, it would go a long way to creating a 3D-like window manager experience. Audio playing in background windows could be muffled to match the position of the window and how many layers deep it exists behind other windows. Positional audio techniques can even trick the listener into thinking that sound is above or below them, making the sound of on-screen to appear like they’re coming from 3D space.

  31. gosh, this brought back memories for me. About 10 years ago I had one of those ELSA Video cards with stereoscopic glasses. It was awesome.

    The card came with Turok and I played it so long that I got ill. Wearing those glasses in 3D were really sick.

    I swear Mary Joe Foley when talking about potentional partner for Project Pink mentioned LG/Samsung/Sharp or some similar company because of their screen technology. A few years ago, MS Research (along with IBM research) had a video of a lightweight 3d display with a mobile gui.

    Anyways, yeah, it would be cool if an OS had this type of tech natively. Perhaps, we’re within a few years of it.

  32. The existing technology of 3D displays is based upon the idea to show a different picture for the left and the right eye. Of course, the picture itself remains flat, our brain just imagine the next dimension because of some parallax between objects in the picture. So, the eye simply can not focus on a “distant” of “near-standing” object in this pseudo-3D picture. They are both at the same distance actually, displayed on your monitor. Maybe, that is why some people feel sick watching 3D movies. I don’t know for sure/
    We will have to emulate the blurring of unfocused objects by constantly watching the user’s eyes to determine on which depth they are focusing right now. Or completely change the way we display 3d pictures. Make them in three REAL physical dimensions like these holographic projections in Star Wars.

  33. I’m almost a year late, but as an academic neurologist-neurophysiologist, I can also assure you that 3D steroscopic UIs are likely to be the future means for manipulation of complex, high-density datasets. These UIs are likely to be the key to resolving and intuitively manipulating not just the raw data, but the metadata necessary to comprehend the intricacies of human behavioral dynamics in 4 axes.

Leave a Reply