Multitouch desktop keyboard

For 30 years the interface for a computer has been tied closely to a typewriter. The keyboard allows for quick text input in a simple and predictable way. The problem is that the keys on the keyboard do not always map well to the application that you are using. For example when using a music application the keyboard interface is limited to keyboard shortcuts for mau commands. When using a visual program like PhotoShop the keys allow you to change tools in a very non-visual way.

Recently the optimus keyboard has suggested that the keys don’t have to be static, they can each be a small screen that provides visual feedback to the current application or function. This approach is nice in that it stays with the traditional keyboard but it very expensive and is ultimatly still limiting from an interface design perspective.

The iPhone from Apple has also receently been showing off some keyboard ability particularly in the multi-touch space. Multi-touch is the ability to independantly detect multiple fingers on the screen at the same time. Despite the seemingly ‘newness’ of the technique the principals of multi-touch have been around for over 20 years.

A concept that we’ve been exploring at Raizlabs is combing aspects of the adaptive keyboard and multi-touch technology. The result is a keyless keyboard:

Multi-touch keyboard concept

The traditional keys are replaced with a multi-touch screen technology. This allows you to type in a very traditional way for basic input but allows the interface to be customized or adapt to specific applications such as music:

Multi-touch music concept

Individual applications can create application specific keyboard layouts or they can use the traditional keyboard. The multi-touch aspects also allow you to integrate the mouse into the exact same touch screen interface. Here’s a closer look:

Keyboard layout

Underneath each hand is a visual nub. Shown here as a blue dot. As you’re typing you can grab either blue dot and use it like a mouse. You can drag it around the screen. As you move it left and right mouse button appear under your fingers and the keyboard fades out. So the mouse is litteraly at your fingertips, your hands never leave the keys. Let go of the mouse and the keyboard comes back.

Now let’s talk about multi-touch. If you grab both dots and drag both down this is the same as scrolling.  Grab both dots and push them together and this simulates the keyboard shortcut for Copying text. Push both dots out and this simulates the Paste command. Additional two and three finger gestures could also be created. I’m only giving a couple quick examples.

Potential problems?  Tactile feedback is the largest potential problem. Since you don’t feel the key go down the interface will need to compensate with subtle audio and visual cues to show you when a key has been pressed. There’s a lot of research that shows that this type of feedback works and can overcome some of these drawbacks. It may also be possible to have the keyboard ‘push-back’ offering some haptic feedback as well.

We’re excited by the concept and have started contacting display manufactureres to guage the feasability of the display technology.
If we built this as a stand alone keyboard and it cost under $500 would you buy it?

20 thoughts on “Multitouch desktop keyboard”

  1. Can you post (or email me) some cites to the research on effectively using acoustic and visual feedback for non-mechanical controls? I’ve tried some implementations and I was not impressed, but that was a number of years ago, and I’d like to get up to speed on this.

  2. I can’t tell if I would buy it, without trying it. Definitely, it is a bright concept and I am curious how it will be received by the market.

  3. Great concept, a device which propose the most appropriate HCI system according to task… great principe, only one problem : you need an interaction feedback. when finger hits a key, it needs a response. Would be curious to see how they solve this issue.

    Thanks for the post

  4. Peace people

    We love you

  5. We’re talking about a totaly generic, tactile feedback keyboard?

    Hell yes.
    It’d be pricy, but I would.

  6. I would definitely buy it! As long as it was compatible with the likes of Logic, Fruity Loops etc and you could display track volume & pan sliders etc.

    I started thinking about the idea since the iPhone announcement in Jan, & even though they aren’t out here yet so haven’t been able to experience what it’s like, i’m really keen on the concept.
    I have been drawing up some ideas of what each applications controls would look like and have now moved onto doing a mock up (only using VBA – not interested in making it ‘control’ anything yet, more a visual design eg sliders move etc.)

    Only problem is that I have no idea how to get someone to make it! (Come on Apple – your the best people for this (& can you give me a job!!))
    Sure, you’re gonna get a few ‘nay-sayers’ at first, but it’s the same with any new concept – people will warm to it and it’ll become the norm (like computers are now).

  7. I already have: http://www.fingerworks.com/ST_product.html (sans display of course). Took some getting used to, but gestural input is the wave of the future. I got mine retail before they got bought out by Apple. You can find them on ebay now for around a thousand US dollars.

  8. emm.. nice post dude..

  9. if it had interchangeable interfaces for different applications, or even a “designable GUI” for the user. I def would. Using it in applications for audio production would be very helpfull.

  10. This is the future. No doubt about it.

  11. I had this same idea and was looking around when I found your blog. I’ve though some improvements over the basic idea and would like to tell you. I don’t have the technology to development them but, if you can, I’m sure you’ll like what I have to tell. Please email me.

    Best regards.

  12. wow wow wow…..
    ive been talking about this idea for a very very long time.. and since im just a poor guy hanging out i never DID anything about it. so today i just did some random searches and found this!!
    my idea was from the movie “Minority Report”
    and the way tom cruise’s character interacted with the screen was great. then apple did their touch stuff. and now microsoft has a giant table top sized one.(ah and dont forget the PDAs)
    anyway my idea (in my mind) was basically a
    two screen laptop sized …thing.
    the top screen is the basic display as most laptops.
    but instead of a keyboard and mouse place..
    we have a touch screen. (like a giant PDA or ipod) so since its actually a display we could have it be a mirror of the top display
    or we could let it switch to a very long single display.
    i noticed people (who dont know better) love to try to touch the screen on my laptop.
    so this would let them just touch the bottom screen.
    also i use A LOT of internet but not much
    typing (until today). so opening and closing
    web pages… or surfing with a finger
    (really point and click) would be great.
    plus there are already touch keyboard software
    all over the place. so when we want to type
    we just turn on the program, and type away.
    and yes of course there should be a
    desktop version. (giant 17″ PDA) connected
    by whatever signal you want. sell that for
    500$ and yup…. ill buy it.
    put it in a laptop… i think it would be all over.
    call it a “touchboard” and have apple and the PCs install it like Intel.
    just my thoughts.

  13. ….after a bit more searching…
    i found this :

    http://www.artlebedev.com/everything/optimus-tactus/

    and this (which is exactly what i meant) :

    http://www.xconomy.com/boston/2008/05/20/negroponte-unveils-2nd-generation-olpc-laptop-its-an-e-book/

    well…
    i guess there really are…
    “no more ideas”.

    good luck though!!

  14. I’m not 100% sure this would work properly, any touch screen device would do what you wan’t it to do every time. Human fingers are clumsy and Haptic feedback isn’t good enough yet. I should know I have a touch screen phone and the so called “Haptic feedback” just feels like the phone is vibrating, it doesn’t feel like a button in any way shape or form. All it does if go “yes you pressed a key and I registered it” but not “You pressed the right key”.

  15. There was a company which made the exact thing you describe – FingerWorks. They sold a combination keyboard-mouse which recognized quite a few gestures (double-right-click, control-shift-uparrow, etc., etc.). They disappeared and their patents were bought up. I suspect this is the enabling technology for all of the multitouch objects being made now. I -really- wish they were still in business. I loved the no-pressure-required typing. It was MUCH easier on my fingers than the mandated 50gram pressure keyboards available now.

  16. Good, interesting article, but where took information?

  17. I am including the video of the one handed SoftFrog on tablets and iPhone. These are both in Alpha stage, yet are working very well. Because the FrogPad IP is multi touch, we are using both multi touch and single touch technologies on our SoftFrog. FrogPad is now a proven User Interface.

    This is the alternative/additional user interface for all Multi Touch applications.

    http://www.youtube.com/watch?v=te9O8HavQpQ

  18. YES!!!! I would run and buy it ASAP and so would millions of others with carpal tunnel wrist injuries from computers!

    • actually you are right, the hardware FrogPad does just that.

  19. If you could make it for $100 then I would definately buy it. I love the idea of building in the mouse like that. I’d like to see a few physical “dots” on the keyboard like are on the f and j keys already.

    Better yet, create an interface for an existing tablet to do just this so that people can use it now.

Leave a Comment