1 Mobile Computing CSE 40814/60814 Spring 2017

2 The Past – The Present

3 Steve Mann 1970s, pre-laptop, early computer era.Building computers he could wear. Inventor of wearable computing.

4 Steve Mann 1991: Started the ”Wearable Computing Project” at MIT.1995: World’s first covert wearable computer – camera and display concealed in ordinary eyeglasses. 1997: PhD from MIT in the field he himself had invented. Today: Works at University of Toronto.

5 Steve Mann

6 What is Wearable Computing?Mann constant and always ready, unrestrictive, not monopolizing of user attention, observable and controllable by the user, attentive to the environment, useful as a communication tool, and personal. “A wearable computer is a computer that is subsumed into the personal space of the user, controlled by the user, and has both operational and interactional constancy, i.e. is always on and always accessible. Most notably, it is a device that is always with the user, and into which the user can always enter commands and execute a set of such entered commands, and in which the user can do so while walking around or doing other activities”

7 What is Wearable Computing?Seven attributes of wearable computing [Steve Mann, 1998]: Unmonopolizing of the user’s attention. User can attend to other events. Unrestrictive to the user. Allows interaction while user carries out normal functions. Observable by the user. As the system is being worn, there is no reason why the wearer cannot be aware of it continuously… but this contrasts with 1! Better phrasing: User can identify computational and non-computational components of their clothing. Controllable by the user. User can take control at any time. Attentive to the environment. Can enhance the user’s environment and situational awareness. Communicative to others. Can be used as a communications medium. Shares the same physical and situational context as the user.

8 Wearable Computing Intelligent agent (user) (Non- wearable) computerOutput Output Intelligent agent (user) (Non- wearable) computer Intelligent agent (user) Wearable computer Input Input Input Input Output Output (eg. Camera records optic flow) Sensation Actuation (eg. walking) Environment

9 Purpose: Mediate InteractionsWearable computers will help provide a consistent interface to computationally augmented objects in the physical world. Example: Gesture Pendant One gesture could provide an intuitive command for many devices

10 Purpose: Aid CommunicationThe wearable can also assist in human-to-human communication. Wearable computers can also help manage interruption in the user’s daily life.

11 Purpose: Provide Context-Sensitive RemindersInstead of simply acting as a virtual secretary, the wearable could be proactive and intimate, listening to the wearer’s conversations and providing reminders as appropriate

12 Purpose: Augmented RealityAugmented reality overlays information-rich virtual realities onto the physical world. In a sense, augmented reality is a combination of the application domains described previously. https://www.youtube.com/watch?v=vDNzTasuYEw

13 Components of a Wearable Device

14 Head-Mounted Display (HMD)Small screen, typically covering one of your eyes. Works like an ordinary monitor, providing an image floating in the air in front of you. Transparent vs opaque.

15 Camera (& Sensors) Suitable placement Head, follows user’s gaze.Shoulder, more stable.

16 Input Device Keyboard Mouse BrainGate Canesta’s IR keyboard.Arm-strapped keyboard. FrogPad Twiddler chording keyboard Mouse Twiddler, again. BrainGate

17 Input Device Gestures Voice recognition Multi-modal interfacesGesture Pendant (controlling smart homes) Voice recognition Siri Multi-modal interfaces Something new?

18 Output device Sight – Visual output Hearing – Audio/sound/speech/musicHMD, wristwatch... Hearing – Audio/sound/speech/music Speakers, earplug/headset... Touch – Tactile feedback Taste and smell

19 The Computer Itself Anything small, but powerful enough SmartphonesSmartwatches Embedded computers

20 Network Connection Benefits of having a networkAccess to the Internet Communication Localization Wireless network connection WLAN 3G, 4G, LTE Bluetooth, ZigBee InfraRed

21 Other Challenges Connecting all pieces Power supply Heat dissipationWires (embedded into clothing?), wireless (security?), body as conduit Power supply Batteries (rechargeable; solar power) Human powered devices Body heat, 0.6 – 4.8W (wetsuit clothes) Breath, 0.4 – 2.5W (pressure mask) Blood pressure, 0.2W Limb motion, 0.3 – 1.5W Finger motion, 0.019W (keyboard typing) Walking, 5 – 8W (shoe generator) Heat dissipation

22 Examples Technicians Field workers Military personnel ResearchersBlueprints, etc. Field workers Access to information given by remote experts. Military personnel Soldiers, monitoring health, equipment, etc. Maps and terrain. Infrastructure (sewers, roads) in urban areas. Researchers

23 Examples Wearables for sports trainingKarate trainees are instrumented with acceleration sensors. Sensor data is translated directly into sound output. Trainees can now ‘hear’, as well as see instructor’s movements. Trainees can also hear themselves: attempt to match own sound to sound of instructor. Martial arts training is about reproducing patterns over time, not just matching static poses; therefore, sound is a useful sensory stimuli to introduce to training. Result: Trainees with system tended to learn faster than trainees without system.

24 Examples Wearables for the military: Future Force Warrior (FFW)Onboard physiological/medical sensor suite to accelerate casualty care Netted communications to maximize robustness and integration of small teams Embedded training (similar to martial arts example?) Enhanced situational awareness (heads-up display?) Synchronized firing of weapons from team. Bone conduction technology: “talking and speaking without sound or hearing”

25 Examples: Smartphone Extensions & Fitness Trackers

26 Examples: Fitness & Health

27 Examples: Fitness & HealthHexoskin Clothing

28 Applications Mediated Reality Augmented RealityExperiencing the world through the computer Allows computer to process the sensory cues before reaching the user E.g. block commercial billboards Augmented Reality Overlaying virtual information on the real world E.g. allow architects to build virtual houses E.g. the AR Quake or AR Pacman game Both realities can enhance your senses

29 Applications – Augmented MemoryTrivial example, finding your way ”Where did I park my car?” Camera on your body records the way Replay helps you find your way back Only key events need to be recorded Example: Intersections at a car park

30 Applications – Augmented MemoryElderly or people with poor memory Remember name and face of people Image processing can recognize a face and map it to the person’s name and affiliation How should it be presented?

31 Applications – Annotated Reality

32 Applications - Advertising

33 Applications - Entertainment

34 Applications - EntertainmentPrioVR gaming

35 Applications – Aiding the Visually DisabledSome forms of low vision can not be alleviated by use of ordinary glasses User wears non-transparent glasses with integrated displays, experiences the world through a camera Computer processed video stream Enhance contrast Adjust colors Night vision Enlarged view

36 Applications – Aiding the Visually DisabledFisheye lense for reading text. Remapping around blind spots.

37 Applications – Additional Vision Tricks”Edgertonian” eyes Freeze-frame effect, fast shutter Reading text on a tire of a speeding car Clearly seeing the rotor blades of a helicopter Counting the number of bolts holding an airplane rotor together in mid-air Plus lots of other interesting effects https://www.youtube.com/watch?v=yr3ngmRuGUc

38 Applications – Additional Vision TricksGiant’s eyes Enhances depth perception of distant objects

39 Applications – First Response

40 Applications – Social softwareUsually designed for urban settings. Interface to groups or individuals. Safety net Heart rate, perspiration, breath rate Alert friends in case of abnormal values Friend finder

41 Google’s Vision for GlassCreate a closer relationship with technology. Move away from technology that competes with real life Ex: Take pictures and record videos as you experience them. Allows you to capture the experience and remain in the moment. There when you want it, gone when you don’t. Ex: Glass is not in field of vision. Remain connected to what you doing. By bringing technology closer, we can get it more out of the way.

42 Development Vid Link August 2011, a Glass prototypeProduct concept of the Google [X] Moonshot Factory Google Glass Google co-founder Sergey Brin, first Googler spotted wearing Glass San Francisco Charity Event February 20, 2013 application for Explorer Program Events in LA, SF, NY Google puts together video Vid Link August 2011, a Glass prototype Weighed 8 LBS https://www.youtube.com/watch?v=eo29M8Yk3Qc

43 Glass Wi-Fi/ Bluetooth How it worksLocation: Sits above the line of sight Control: Look up to activate/ Look down to activate sleep mode Glass Location: Inside the right arm of Glass Control: Links to Wi-Fi or devices with Bluetooth. Wi-Fi/ Bluetooth

44 How it works Touchpad Speaker Location: Right arm of GlassControl: By swiping and tapping touchpad Speaker Location: Right arm of Glass just behind the ear Control: Automatic voice commands

45 Voice Commands Glass Feature Voice Command Explanation Take Picture“Ok Glass” take a picture Hands free and quick Take Videos “Ok Glass” take a video In the moment recording Video Chat “Ok Glass” video chat [Person] Lets others see what your seeing Send Texts “Ok Glass” send a text [Person] Hand free texting like Siri Translate “Ok Glass” translates [Word/Phrase] Translates and recites back with proper pronunciation Directions “Ok Glass” give direction to [Place] Turn by turn directions Weather “Ok Glass” what is the weather Temperature, Chance of Rain and if it is currently sunny or cloudy Google Hangout “Ok Glass” hangout [Group Name] Group interactions with selected friends

46 Medical Wearables Wearable Belts Sensor Insoles Smart ClothingHeadsets Patches Respiratory Biofeedback and Body Sensor Networks Sensors in Sheets

47 Smart Watches Sony SmartWatch, Apple iWatch, Galaxy Gear … Pros & ConsUses NFC, Bluetooth, and RF technology for communication Pros & Cons Pro: More integration, gateway to Internet of Things (IoT) Con: Distraction on the road, battery life, smaller screens

48 Challenges Privacy Price Health risks Accidents

49

50 Wearable Computing & the MarketGrowth in wearable Computing 35% growth by 2019 Smartwatch will be leading product Will rise by 41% Apple Watch market share Account for 40% market share in 2015 Fitness bands, smart eyewear, etc. 36% percent market share this year