It turns out 2019 is getting closer every day. At the moment, Microsoft’s chief research and strategy officer Craig Mundie is doing the rounds at a number of prestigious colleges in the States showing off Microsoft’s vision for technology to solve the world’s biggest problems. Of course, one must use the latest in natural user interfaces for this task.
A feature of this year’s tour appears to be a next-generation computer – one that docks and undocks from a transparent glass display and allows for not only pen and voice input as you’d come to expect from natural user interfaces, but also incorporates touchless gestures and eye-tracking to interact with the information at hand.
Personally I’ve never seen eye-tracking used as an input before, but after seeing this demo, it makes so much sense to skim vasts amounts of information with your eye.
I’ll let these two videos do the rest of the talking.
No doubt with any fancy prototype it’s usually very difficult to distinguish just how much of the demo is real and how much of it is simulated – either by timers or remote-control, but knowing Microsoft Research and what they’re capable of, I’m willing to bet it’s all real.
On a related note, it appears now Microsoft’s new vision is a glass-display on every desk. Time to get in the window cleaning business perhaps?
33 Comments
@ThomMck
That shot in the second video, where the image comes up on to he glass, is amazing.
Slightly similar concept to 10gui.com to prevent arm-ache when using a touchscreen monitor.
I wonder if Windex shares have gone up?
AlSki
Hehe.
Does being a Linux installer, make you a “Windows” cleaner?
Jamie Thomson
It looks fantastic but is anyone else a little bit fed up with seeing “concepts” from Mundie’s team(s) rather than actual products? The only one I can really think of to come through in the last few years was Surface.
Another recently was Yusuf Mehdi’s demo earlier this year: http://www.ditii.com/2009/09/28/microsoft-next-gen-office-wall-prototy pe-demo/ . This stuff really doesn’t feel any closer to coming to fruition y’know!
-Jamie
NCTech
Jamie, you have to understand that much of the work that Mundie’s researchers come up with end up embedded in other products rather than as standalone products. His teams came up with a lot of the features in Aero and Glass in Win7. Project Natal, the Wii-like (but better) interface for Xbox (and more). came out of MS Research.
Roundtable, the 360-degree video microphone for conference call, which MS sold off to another company, was another MS Research development.
John Sedlak
That second video looks pretty fake to me.
But a cool concept none-the-less.
rob
Regardless of whether or not this is fake, I’m SUPER pleased that this issue is being raised and being worked on. The entire idea of controlling a computer with a mouse, while nice because it is accurate, it is also at times very slow. Imagine being a 3D artist, and being able to manipulate your 3D object with gestures as if it were sitting there right in front of you. Or imagine organizing your files/folders in your computer the same way you would if you had an actual filing cabinet in front of you. It only makes sense to move in this direction, and I’m so glad this is finally being worked on. I have envisioned this (I keep thinking of Minority Report) for a long time and I think it is entirely feasible and should be done.
8675309
i remember ms had a concept demo for a double sided mutitouch handheld pc/pda looked cool!
Ooh
@John: It isn’t fake. Look closely and you’ll see a projector behind the glass (in the table). Looking to the scene from the top, in the front is the projector, then the thin glass, then his keyboard and finally him. You can also see the projection being worse at the top end of the glass screen while it is much better at the bottom end.
Someone
I may be wrong, but I am not convinced that either of these demos will prove to be useful.
For the first, suppose I focus on a document, and the computer automatically zooms in on it. Now, if I move my eye to the top left of the magnified window, how is the computer going to figure out whether I want to take a closer look at the zoomed-in-upon document or whether I want to look at a different document? The demo makes it look like the system will zoom in on another document as soon as one’s eye leaves the area where the unzoomed document was. If so, it will be as good as impossible to see more from the zoomed-in document than from the stamp-sized picture.
For the second, using outstretched arms for anything but occasional UI would tire really, really soon. I would rather have, for example, some gadget in a pocket that allowed me to do this 3D pan/zoom/etc stuff.
In my opinion, these make for nice demos, but the things they show would be lousy products.
ToddF
I agree with “Someone” – great eye candy, but completely out of touch with solving real usability problems. How does making you wave your arms around to do something you can currently do by rolling a scroll wheel on a mouse qualify as an improvement? Good for museums and stuff like that, crappy for 99% of computer/human interaction models.
Robert Chappell
I was especially interested to see that Microsoft is looking at eye tracking as a user interface. I have an eye tracker mounted on my monitor and I use it every day. It replaces the mouse for most pointing actions. It is also fast and intuitive, but a little less accurate than the hand mouse.
Full disclosure: I am president of EyeTech Digital Systems , the manufacturer.
Cliff
@ToddF: It does look a lot like eye candy now, but the important thing is that people are thinking about alternate ways to present and interact with data. Even if Microsoft doesn’t get it right, someone else will see this and think “Hey, I know how to make this work.”
sometwo
Eye tracking? Surely something is projecting light into our eyes to reflect it. Are our eyes safe from radiation?
hologram
Someone will get it right, like Apple.
Joey1058
The good thing about “eye candy” is that it gives you something to wait for. The bad thing about “eye candy”, is that it makes you wait forever. I have no doubt whatsoever that this is cutting edge technology. But getting it out of the labs and to the general public is not only up to the business developing it, but also a matter of political decision: “What can we release that will make our competition look bad?”
Last year they announced Project Natal, but I have yet to see it anywhere close to implementation for the public. Before that, it was the Surface. There are some out in selected markets. I haven’t seen one anywhere except online. As long as the bean counters call the shots, this stuff is still cutting edge SF.
Sorry, Microsoft, you’ve let me down on Project Natal. How am I supposed to keep the faith in pie in the sky handwavium? When I can try it myself, I’ll be impressed.
Peter van Dam
good video, but whats the point of projecting things on such glass? I mean, u see it in this video, u see it happening in tons of movies and tv shows, but I can’t believe it’s helping me to work better with the work. I mean, I’m almost sure I see the content visible on that glass better when I can’t look trough the glass. Why do I need to learn to make those hand gestures, and hold it up that high when pressing a small button in 0,008 second on my keyboard does the same. Sure it’s usefull when ur not able to have a keyboard and don’t want to touch your device, but other then that I fear they are working alot on just eye candy stuff that does’t make any sense for business users.
Other then that, how are those maps really usefull to me. Whats the point of showing me 1000’s of boxes? Have u ever googled/binged over 1000’s pages? Give me 10 to 50 very good results and I’ll be soo much happier.
In my opinion i’m a bit dissapointed with both videos. They aren’t looking like anything usefull to me, aren’t build on real scenarios, and is just made for eye-candy purposes rather then improving the stuff we work with today.
NCTech
Joey1058 — When MS announced Natal it was demoing it and releasing it to developers. At the time MS said it would be spring/summer 2010 before anything hit the consumer market.
Alex
Second video shows how you might save on gym by buying Microsoft software.
RsrMusic
Can’t wait till they hack this..
Just kidding.. like the concept!
reader
Computer! Tea, Earl Gray, hot.
tom
would’nt it be like very anoying???
working on your computer and seeing your background.
dj_cityboy
although i believe this to be real as well, it is highly unlikely that we will ever see it, not to mention i believe this dont would work in your typical office space, given that most follow the formfactor of cubicles and as such this type of technology would be left for executives and “higher-ups (management)” i am pleased that MS is willing to push the boundaries of the modern PC, but i have seen to many demos from the MS research group, that have never come to fruitiion, mind you, this leaves me “warm and fuzzy inside” but is highly doubtful that it will ever come, Phodeo, MediaBrowser, TimeQuilt and Phototriage were almost all based on the same idea, grouping your media together by way of metadata, this demo goes a bit further than that with the fancy display and such.
i also wonder if this is “real”, it very well could be, but at the same time MS have shown lots of fancy slideshow/macromedia demos in the past showing off technology they werent capable of or of things that never came to be.
as tom mentioned above, i would find that being able to look through my display would be very distracting and would blow “privacy” out the window, not to mention if this is what the pc experience would look like in 2019, showing it now is a bit premature, dont ya think? lol
peas
cityboy
Joey1058
NCTech – I did forget about the announced release date. I stand corrected.
Wellescent Health Blog
It is interesting to see more of these types of interfaces designed to work around people instead of having people work around the limitations of the machine. Where these interfaces could truly make a difference is in the case of those who have mobility issues. An interface with more advanced eye tracking capabilities would certainly be of significant benefit to them.
Nnamdi Iregbulem
Personally I think this kind of technology is amazingly cool and futuristic.
I guess the question then becomes how useful will it actually be. Having cool looking stuff is nice but if the end user experience isn’t improved then there isn’t much point to it.
I hope Microsoft is thinking about this as they design these next-gen computers.
Jonathan Dickinson
While very cool this is a contrived example. The big problem with NUIs is making them generic (think mouse,window,button,radio button). A NUI generally has to be written for every possible use-case and it’s far more complicated than slapping a few buttons onto a window. If they can crack that barrier they might actually have something *really* special.
dps
Eye-tracking is a new thing? BOUAAAHAHHAAAAA
http://en.wikipedia.org/wiki/Canon_EOS#Eye-controlled_focusing
Tore
This isn’t that far off.
Metadata sorting is quite feasible – the question is how good is the metadata and how much data are you looking at. An existing application that uses a similar approach is WinDirStat (http://windirstat.info/).
The glass surface is required in order to be able to read the gestures. I suspect it uses the MS Surface technology that alternately projects dual images (to surface and through surface to optional dynamic secondary surface) and senses through the surface. It just takes the glass off of the table.
Gesture interfaces are happening – the iPhone uses it extensively. The demo is just taking the multi-touch concept (built in to Windows 7 AFAIK, and available as add-on to Vista(?)) to the through-the-glass sensors. The hardware cost needs to come down before it gets wide distribution.
I enjoyed playing with the MS Surface last year, and it is really great – at least for some things. For typing-intesive tasks, maybe less so. But then a mouse is more of a drag than pointing. Once you try it, you won’t quickly forget. Very cool indeed.
Lim
Very cool, especially the flicking of the wind turbine onto the glass screen. However I still have my doubts on speech vs clicking. We think faster than we speak, so the mouse is still more productive in my opinion because you can work faster with it, clicking vs talking.
Ted
If I have to have a big piece of glass in front of me, between myself and a client, why not just project the display onto a wall? It would display much bigger and could still have all the functionality. If your going to create something get rid of the glass and create a 3D hologram that I can work with? Then I can spin the object around and zoom in or out on it. You could display the hologram in the middle of a conference table and it could be the center of focus as your display should be. You could still control it from your computer where you are giving your sales pitch, whatever it may be.
Tore
It is the direct manipulation of the displayed image that makes it so powerful. It is one of those things you just have to experience.
Today’s technology requires the glass (it’s not just plain glass) and uses a rear projection and through the glass sensing. Holograms are still a ways off – at least for mainstream application.
Stephen
*sigh* There goes Microsoft copying Apple AGAIN! /sarcasm
lol.
Antony
How interesting, we will all be working in star trek like offices lol.
A x
Leave a Reply
Microsoft Research demo prototype glass screen PC with gesture and eye-tracking control - SlashGear
[...] [via istartedsomething] [...]
Microsoft Research demo prototype glass screen PC with gesture and eye-tracking control | Phone-Review.org
[...] on Microsoft’s College Tour virtual press room, or at istartedsomething linked below. [via [...]
Microsoft messes with ‘natural’ user interface, featuring eye tracking, voice control and wild body gestures | Netbook Gizmo
[...] Read | Permalink | Email this | Comments [...]
Microsoft Research zeigt transparenten Monitor | Spreeblick
[...] Zheng zeigt auf seinem Blog I started something (das gerade schwer zu erreichen ist) ziemlich beeindruckende Videos der Microsoft College Tour 09, [...]
Cool UI – Multi-Touch without Touch « Kindle Review – Kindle 2 Review, Books
[...] the Microsoft College Tour 2009 which redefines multi-touch (jump to 00:40 for the magic, courtesy istartedsomething) - [...]
links for 2009-11-06 | AndySowards.com :: Professional Web Design, Development, Programming Freelancer, Hacks, Downloads, Math and being a Web 2.0 Hipster?
[...] Microsoft College Tour 09: mindblowing natural user interface concept demos from Microsoft Research … This is pretty freaking awesome – Look to the future! (tags: innovation video computers future interaction now tech) [...]
inovus.mx » Blog Archive » Microsoft trabaja en un Display de vidrio transparente
[...] hay unos videos que muestran el prototipo en [...]
It’s a see-through future according to Microsoft
[...] Thanks to Long Zheng over at his istartedsomething.com blog he has some pretty interesting videos of speeches given recently by Craig Mundie at various big name colleges around the United States. [...]
RSS For Gadgets » Microsoft messes with ‘natural’ user interface, featuring eye tracking, voice control and wild body gestures
[...] Read | Permalink | Email this | Comments Rate this topic: (No Ratings Yet) Popularity: 0 You can follow any responses to this entry through the RSS 2.0 feed. You can skip to the end and leave a response. Pinging is currently not allowed. [...]
Video: Microsoft Demonstrates Next-Gen Interface, with Motion Sensing and Eye Tracking : MIDEASTA
[...] [iStartedSomething via Gizmodo] [...]
מיקרוסופט מציגה את ממשק העתיד | Newsgeek
[...] מציג קייג מונדי, סמנכ"ל מחקר ואסטרטגיה במיקרוסופט, ממשק המבוסס על זיהוי תנועה ומסך שקוף בדיוק כמו שראינו בסרט דו"ח [...]
Alex (alexdeibel) 's status on Saturday, 07-Nov-09 15:11:18 UTC - Identi.ca
[...] Microsoft #win http://www.istartedsomething.com/20091106/microsoft-college-tour-09/ [...]
Windu.be Blog » Microsoft toont computer met ‘oogbesturing’
[...] research Craig Mundie van Microsoft toonde hoe de computer bestuurd kon worden met onder meer input via een pen of met de vingers, maar ook [...]
Windu.be Blog » I’ll let these two videos do the rest of the talking. No doubt with any fancy prototype it’s usually very difficult to distinguish just how much of the demo is real
[...] Microsoft College Tour 09: mindblowing natural user interface concept demos from Microsoft Research… November 6th, 2009 AEST by Long Zheng [...]
Microsoft demonstra tela transparente que funciona com rastreamento de visão e gestos sem toques | Notícias de Tecnologia
[...] controlar o computador apenas olhando para ele após balançar minhas mãos. [iStartedSomething via Slashgear] Categoria: [...]
Microsoft demos new Natural User Interfaces
[...] Istartedsomething.com have come across two videos showing Craig Mundie demoing Microsoft’s idea of what the user interface of the future will look like. [...]
Microsoft demos new Natural User Interfaces | Phone-Review.org
[...] Istartedsomething.com have come across two videos showing Craig Mundie demoing Microsoft’s idea of what the user interface of the future will look like. [...]
Microsoft demos new Natural User Interfaces | Phone-Review.org
[...] post to watch Flash video] Istartedsomething.com have come across two videos showing Craig Mundie demoing Microsoft’s idea of what the user [...]
Microsoft Research demo prototype glass screen PC with gesture and eye-tracking control | TMS Tech Blog
[...] [via istartedsomething] [...]
Microsoft Concept Interface Sees You | GottaBeMobile.com
[...] istartedsomething has video of a natural user interface concept being demonstrated around U.S. colleges by Microsoft chief research and strategy officer Craig Mundie. Includes a brief demo of predictive handwriting recognition, much like what we have in Windows 7, but also some stuff that is beyond cutting edge. A feature of this year’s tour appears to be a next-generation computer – one that docks and undocks from a transparent glass display and allows for not only pen and voice input as you’d come to expect from natural user interfaces, but also incorporates touchless gestures and eye-tracking to interact with the information at hand. [...]
Digital Shiv » Microsoft is developing the next generation of User Interfaces: eye-tracking, motion-tracking & voice recognition
[...] istartedsomething has posted a couple videos that may be of interest to those who follow the latest developments in User Interfaces: “Microsoft’s chief research and strategy officer Craig Mundie is doing the rounds at a number of prestigious colleges in the States showing off Microsoft’s vision for technology to solve the world’s biggest problems. Of course, one must use the latest in natural user interfaces for this task. [...]
Microsoft Research demo prototype glass screen PC with gesture and eye-tracking control | ToGadget , The Guide to Gadgets
[...] [via istartedsomething] [...]
Microsoft Research Demos Touchless, Transparent Glass Display | Gizmodo Australia
[...] has videos showing the prototype in action, and it looks like it’s got quite some potential. I [...]
links for 2009-11-07 | AndySowards.com :: Professional Web Design, Development, Programming Freelancer, Hacks, Downloads, Math and being a Web 2.0 Hipster?
[...] Microsoft College Tour 09: mindblowing natural user interface concept demos from Microsoft Research … This is pretty freaking awesome – Look to the future! (tags: innovation video computers future interaction now tech) [...]
OhMyGeek! – Sin palabras, Microsoft presenta interface de gesturar y seguimiento de ojos
[...] vía: istartedsomething.com | [...]
Happy Interface Day! « BrentonKlik
[...] Natural User Interface Microsoft Research Project (via istartedsomething) No Comments | Posted in General | [...]
DEVELOPMENT - Microsoft Year One - Random Things To Blog
[...] this year http www ditii com 2009 09 28 microsoft next gen office wall prototy pe demo …source: http://www.istartedsomething.com/20091106/microsoft-college-tour-09/Re lated Bookmarks :Smart Phone Win 6 1 GPS JAVA CP0290 US 168 00 www sivewww siverbox com Smart [...]
Microsoft présente un écran transparent aux propriétés étonnantes | Et-demain.com
[...] : iStartedSomething Tweetez-le !Partagez-le sur FacebookPartagez-le sur FriendFeedPartagez-le sur del.icio.usAjoutez-le [...]
Microsoft toont computer met ‘oogbesturing’ | OfficeFlows
[...] research Craig Mundie van Microsoft toonde hoe de computer bestuurd kon worden met onder meer input via een pen of met de vingers, maar ook [...]
Lo schermo che risponde ai comandi vocali e al movimento degli occhi | Notizie
[...] già dei video in rete, e sembrerebbe essere un’invenzione dal buon potenziale. Pensate a quando potremo comandare [...]
Microsoft bastelt an neuen Eingabetechnologien
[...] Hauptverantwortlicher für Entwicklung, Forschung und Strategie bei Microsoft hat auf seiner College Tour in den USA neue Technologien zur Steuerung von Computern vorgestellt. Diese sind wirklich [...]
gadget.logg | Microsoft mostra il computer trasparente
[...] [Via: i started something] [...]
Microsoft Minority Report | NoNiente.com
[...] intende agire, è alla base dei sogni per le interfacce del futuro. Interfacce su cui Microsoft ha sguinzagliato i propri laboratori di ricerca, con Craig Mundie (Chief Research and Strategy Officer) pronto ora a [...]
Weenks = Weekly Links [inks of the week] | More Than Technical
[...] http://www.istartedsomething.com/20091106/microsoft-college-tour-09/ Microsoft taking natural UI forward. [...]
LINKS XXXV - endkampf
[...] Microsoft Natural User Interface [...]