1 NUI e Biometrics in Windows 10
2 Andrea Sassetti Software Engineer, Siemens andreasassetti.wordpres.com @andreasassetti
3 Marco Dal Pino SW Eng - Consultant [email protected]mobileprog.com (ENG) mobileprog.net (ITA) Microsoft MVP Intel Black Belt Software Developer Intel Realsense Innovator Realsense & IoT @MarcoDalPino about.me/marcodalpino
4 User interaction evolutionCLI GUI NUI
5 Natural User InterfaceUse movements / gestures inherent in human nature or in the actions of every day Rapid functionality learning Can help people with disabilities to use the software
6 Microsoft Kinect v2 Kinect for Windows gives computers eyes, ears. With Kinect for Windows, businesses and developers are creating applications that allow their customers to interact naturally with computers by simply gesturing and speaking.
7 Kinect v2 - Sensor Components
8 Kinect v2 - Sensor ComponentsDepth Sensor Depth Sensor Power Light IR Emitters IR Emitters RGB Camera Microphone Array
9 Kinect v2 - Hardware requirementsPhysical dual-core 3.1 GHz (2 logical cores per physical) or faster USB 3.0 port dedicated to the Kinect for Windows v2 sensor 2 GB of RAM Graphics card that supports DirectX 11 Windows or Windows Embedded 8 or 8.1, 10 Kinect Configuration Verifier tool
10 Physical Kinect SensorKinect v2 - High level architecture Kinect Drivers Kinect Runtime .NET API Apps Native API Native Apps WinRT API Windows Store Apps C#, VB, JS, C++/CX Physical Kinect Sensor
11 Kinect v2 - High level architectureService Source Reader Application Multiple Kinect-enabled applications can run simultaneously
12 Kinect v2 - High level architecturePC One connection Multiple connection PC HUB SERVER
13 Kinect v2 – Frame source Color Depth Infrared Body index Body Audio
14 Kinect v2 – Color frame source1920 x 1080 array of pixel Different sensor than depth 30 fps Will maintain brightness and quality by dropping to 15 fps in low-light
15 Kinect v2 – Depth frame source512 x 424 array of pixel 30 fps Distance range 0.5 – 4.5 meters
16 Kinect v2 – Infrared frame source512 x 424 array of pixel 30 fps Ambient light removed From the same physical sensor as depth
17 Kinect v2 – Body index frame source512 x 424 array of pixel Up to 6 simultaneous bodies 30 fps Same resolution as depth
18 Kinect v2 – Body frame sourceFrame data is a collection of Body object each with 25 joints (each joint has position in 3D space and orientation) Up to 6 simultaneous bodies 30 fps Hand state (Open, Closed, Lasso) Lean
19 Kinect v2 – Body frame source
20 Kinect v2 – Audio source Data is audio samples captured over a specific interval of time Audio is associated with an “audio beam”
21 Kinect v2 – Multi frame sourceAllows to get a matched set of frames form multiple source on a single time. Delivers frames at the lowest fps of the selected sources.
22 Kinect v2 – Coordinate MapperIt provides conversion between each systems. It is possible perform single or multiple conversion. Name Applies to Dimension Units Range Origin ColorSpacepoint Color 2 pixel 1920x1080 Top left DepthSpacePoint Depth, Infrared, Body index 512x424 CameraSpacePoint Body 3 meter NA Infrared / Depth camera
23 Kinect v2 – Application LifecycleAdd reference to Microsoft.Kinect assembly First step: KinectSensor kinectSensor = KinectSensor.GetDefault(); if (!kinectSensor.IsAvailable) return; if(!kinectSensor.IsOpen) kinectSensor.Open(); // CODE HERE kinectSensor.Close(); Microsoft Kinect NuGet Packages available
24 Kinect v2 – Basic flow of programmingFrame Source Sensor Reader Data KinectSensor.GetDefault().DepthFrameSource.OpenReader().AcquireLatestFrame().CopyFrameDataToArray(frameData);
25 Kinect v2 – Basic flow of programmingReaders allow to access to the Kinect frame. It is possible to read the frame via pooling: var colorFrameReader = kinectSensor.ColorFrameSource.OpenReader(); using (var colorFrame = colorFrameReader.AcquireLatestFrame()) { if (colorFrame == null) return; // CODE HERE } var colorFrameReader = kinectSensor.ColorFrameSource.OpenReader(); colorFrameReader.FrameArrived += OnColorFrameReader; --- private void OnColorFrameReader(object sender, ColorFrameArrivedEventArgs e) { using (var colorFrame = e.FrameReference.AcquireFrame()) if (colorFrame == null) return; // CODE HERE } or by event:
26 Kinect v2 – Face Detection and AlinementThrough Kinect Face it is possibile detects body faces point and expression. Face points: The user's left eye point The user's right eye point The user's nose point The user's left corner of the mouth point The user's right corner of the mouth point Each point can be visualized in color or Infrared space.
27 Kinect v2 – Face ExpressionFace expressions: The user's happy facial expression The user's right eye is closed The user's left eye is closed The user's mouth is open The user's mouth has moved since the previous frame The user is looking at the sensor The user is wearing glasses
28 Kinect v2 – Face Detection and AlinementAdd reference to Microsoft.Kinect.Face assembly Use FaceFrameSource and FaceFrameReader classes Use FaceFrame class in order to get face information Microsoft Kinect NuGet Packages available
29 Kinect v2 – High Definition FaceThrough Kinect High Definition Face it is possibile detects over 1000 facial points in the 3D space. 1347 facial points 2340 triangles Hair color Body color Face 3D Model Not sufficiently documented Many API are available only in C++
30 Kinect v2 – High Definition FaceAdd reference to Microsoft.Kinect.Face assembly Use HighDefinitionFaceFrameSource and HighDefinitionFaceFrameReader classes Use HighDefinitionFaceFrame class in order to get high definition face information Microsoft Kinect NuGet Packages available
31 Kinect v2 – Hand Pointer GesturesEngagement Targeting Press Panning / Zoom
32 Kinect v2 – Hand Pointer Gestures - PressRecommended Minimum Size: 208 x 208 (in 1080p resolution) Press attraction towards center Larger buttons will just attract away from the edge Adapting to smaller visual sizes: Make large-size hit testable Set KinectRegion.KinectPressInset (Thickness) to non-visible part of button Attraction region is smaller
33 Kinect v2 – Hand Pointer Gestures - WPFAdd Reference to Microsoft.Kinect.Wpf.Controls Add KinectRegion as container for your Windows Control Run it!
34 Kinect v2 – Hand Pointer Gestures – Store AppAdd Reference to Microsoft.Kinect.Xaml.Controls Enable Microphone + Camera capabilities in App Manifest Add KinectRegion as container for rootFrame in App.xaml.cs Run it!
35 Hand Pointer Gestures and Frame sourceDemo Hand Pointer Gestures and Frame source
36 Kinect v2 – Recording and PlaybackKinect Studio Record sample clip of data from the Kinect v2 device. Playback a recorded sample clip of data Play data from a live stream directly from a connected Kinect v2 Device
37 Kinect v2 – Visual Studio VisualizerVisualizers are components of the Visual Studio debugger user interface. A visualizer creates a dialog box or another interface to display a variable or object in a manner that is appropriate to its data type. It is possible to create custom visualizer just implementing some .NET interfaces.
38 Kinect v2 – Fusion Microsoft KinectFusion provides 3D object scanning and model creation using a Kinect for Windows sensor. The user can paint a scene with the Kinect camera and simultaneously see, and interact with, a detailed 3D model of the scene.
39 Kinect v2 – Custom gesture / poseInside Kinect v2 SDK it is available a tool that allow you to create custom gesture. Visual Gesture Builder It is possible define discrete or continuous gesture using Machine Learning technology Adaptive Boosting (AdaBoost) Trigger: Determine if player is performing a gesture Random Forest Regression (RFR) Progress: Determine the progress of the gesture performed by the player
40 Kinect v2 – Custom gesture / poseVisual Gesture Builder Organize data using Project/Solution Give meaning to data by tagging gesture Build gesture using Machine Learning technology Analyze & Test the result of the gesture detection Live preview of result
41 Kinect v2 – Visual Gesture BuilderBasic steps for using Visual Gesture Builder: Create a solution. Create one or more projects. Add a set of clip files to each project. Tag the frames in the clip files associated with each gesture. Build a training gesture database. Use the built gesture database into your application.
42 Kinect v2 – Custom gesture/poseHeuristic ML with Visual Gesture Builder Gesture is a coding problem Quick to do simple gestures/poses (hand over head) ML can also be useful to find good signals for Heuristic approach Gesture is a data problem Signals which may not be easily human understandable (progress in a baseball swing) Large investment for production Danger of over-fitting, causes you to be too specific – eliminating recognition of generic cases
43 Demo Custom Gestures
44 Kinect v2 – Other Framework
45 Kinect v2 – Other FrameworkFace APIs Computer Vision APIs
46 Kinect v2 – Official ResourcesGeneral Info & Blog Developer Forums Download Kinect v2 SDK Twitter @KinectWindows
47 Intel Realsense Camera & SDK
48 Explorar o público (colocar essas perguntas na pesquisa final)Quem conhece RealSense? Quem desenvolve utilizando o RealSense SDK versão 2014? Quem desenvolve outros SDKs que trabalham com este tipo de tecnologia? (Leap Motion, Kinect, etc)
49 Intel® RealSense™ 3D Camera F200A principal câmera que temos no momento é a F200 que vocês podem ver nessa foto.
50 F200 Camera Specs Spec Color Depth (IR) Resolution 1080p VGAAspect Ratio 16:9 4:3 Frame Rate 30/60/120 FPS FOV (D X V X H) 77º x 43º x 70º 90º x 59º x 73º Other Effective Range 0.2 – 1.2 m Environment Indoor/Outdoor Color Camera Resolution: (FHD) Active Pixels: 1920x1080 (2M) Aspect Ratio: 16:9 Frame Rate: 30/60/120 FPS Field of View: 77º x 43º x 70º (Cone) (D x V x H) Color Formats: YUV4:2:2 (Skype/Lync Modes**) Depth (IR) Camera Resolution: (VGA), (IR) Active Pixels: 640x480(VGA) Aspect Ratio: 4:3 Frame Rate: 30/60/120 FPS (Depth) 120FPS (IR) Field of View: 90º x 59º x 73º (Cone) IR Projector FOV: N/A x 56º x 72º (Pyramid) Color Formats: N/A
51 Where to find the Intel® RealSense™ technology?Onde está o RealSense? Até o ano passado estivemos trabalhando com o ecosistema para distibuir kits de desenvolvimento e garantir que desenvolvedores como vocês começassem a criar usos bacanas para essa tecnologia. E agora estamos trabalhando para espalhar o RealSense por vários devices junto com vários parceiros do ecosistema para que os primeiros devices cheguem ao mercado entre 2014 e Inclusive alguns já foram anunciados na CES: mostram alguns deles.
52 Intel® RealSense™ 3D CamerasIntel® RealSense™ Snapshot Intel® RealSense™ 3D Camera (Rear R200) Intel® RealSense™ 3D Camera (Front F200) Basicamente trabalhamos com duas partes: A Intel RealSense 3D Camera Intel RealSense SDK Mas além disso estamos trabalhando em duas câmeras adicionais para ampliar os modelos de uso e possibilidade de integração, e teremos: R200 Snapshot: camera mais simples com foco em telefones e tablets e uso mais casual. Trabalha com pós processamento das imagens depois da captura. Sim é a camera que esta no tablet da Dell. R200: a F200 é uma camera com objetivo de capturar interações com o usuário (Front or User Facing). Já a R200 é conhecida como World Facing camera e tem a proposta de capturar informações sobre o ambiente onde o device esta inserido. F200: Interações naturais, imersivas, colaboração, jogos, aprendizagem e scaneamento 3D Snapshot: Medições de distância, foco e cor, filtros de movimento R200: Realidade Aumentada, criação de conteúdo, scaneamento de objetos
53 Explorar o público (colocar essas perguntas na pesquisa final)Quem conhece RealSense? Quem desenvolve utilizando o RealSense SDK versão 2014? Quem desenvolve outros SDKs que trabalham com este tipo de tecnologia? (Leap Motion, Kinect, etc)
54 Free tools and APIs to develop apps with NUI in a simple wayFocus where is important: the content Acessible for beginners and extensible for experts APIs e ferramentas para implementação de NUI Acessível para iniciantes e extensível para experts Foco do app aonde mais importa: conteúdo
55 Hands - Tracking and JointsDetects Body Side Tracks X, Y and Z positions from detected hands Arrumar fundo azul da imagem ** Reference Links: https://software.intel.com/sites/landingpage/realsense/camera-sdk/2014gold/documentation/html/manuals_programming_guide_gesture.html https://software.intel.com/sites/landingpage/realsense/camera-sdk/2014gold/documentation/html/manuals_general_procedure_2.html https://software.intel.com/sites/landingpage/realsense/camera-sdk/2014gold/documentation/html/manuals_hand_calibration_data.html https://software.intel.com/sites/landingpage/realsense/camera-sdk/2014gold/documentation/html/manuals_alternative_hand_tracking_solu.html Em Rastreamento e Pontos, podemos colher os dados de cada um dos 22 pontos, sabendo também qual o lado do corpo (esquerda ou direira) pertencem.
56 Hands – Gestures and PosesReference Links: https://software.intel.com/sites/landingpage/realsense/camera-sdk/2014gold/documentation/html/manuals_pose_and_gesture_recognition.html https://software.intel.com/sites/landingpage/realsense/camera-sdk/2014gold/documentation/html/manuals_gesture_interaction_guide.html ** ERRO NAS IMAGENS ** TAP GESTURE Aqui observamos os gestos que compõem a RealSense SDK: SpreadFingers ou Big5, gesto estático, o mais simples identificando uma mão aberta; V-Sign, ou mais conhecido V de vitória. Também estático; Tap, representa um gesto dinâmico de pressionar; Wave, gesto dinâmico que representa para nós um “tchau”;
57 Face – Tracking Tracking ( 4 faces) Landmarks Tracking (1 Face)X, Y, Width and Height Average Depth TENTAR ARRUMAR FUNDO AZUL* https://software.intel.com/sites/landingpage/realsense/camera-sdk/2014gold/documentation/html/manuals_general_procedure_face.html https://software.intel.com/sites/landingpage/realsense/camera-sdk/2014gold/documentation/html/manuals_face_location_data.html Ilustração da detecção de rostos. No máximo 4 rostos podem ser detectados ao mesmo tempo. Apenas 1 desses 4 podem ter todos os pontos do rosto rastreados (landmarks). Mostrar Implementação
58 Face – Landmarks 78 landmark points: Nose Mouth Jaw Eyes EyebrowsTENTAR ARRUMAR FUNDO AZUL** https://software.intel.com/sites/landingpage/realsense/camera-sdk/2014gold/documentation/html/manuals_face_landmark_data.html Ao todo podem ser detectados 78 pontos no rosto, dividindo-se nos seguintes grupos: Nariz; Boca; Mandíbula; Olho (esquerdo e direito); Sombrancelha (esquerda e direita). Mostrar Implementação
59 Face – Pose Tentar arrumar fundo azul ****https://software.intel.com/sites/landingpage/realsense/camera-sdk/2014gold/documentation/html/manuals_face_pose_data.html Ilustração dos ângulos X,Y,Z da cabeça: Pitch (y), para cima e para baixo; Yaw (x), olhar para esquerda ou direita; Roll (z), movimentar cabeça para esquerda ou para direita, mantendo X e Y. ** melhorar descrição... Mostrar Implementação
60 Face – Expressions https://software.intel.com/sites/landingpage/realsense/camera-sdk/2014gold/documentation/html/manuals_facial_expression_data.html Expressões Faciais; Em uma forma geral conseguimos captar, através do módulo Face, 8 expressões faciais (com a intensidade aplicada): Sorriso; Boca aberta; Beijo (?), Duck face :^ Olho fechado (esquerdo e direito); Movimento das pupilas (esquerda,direita, cima, baixo) de cada olho; Sombrancelha levantada (esquerda e direita); Sombrancelha abaixada (?) (esquerda e direita); Mostrar implementação...
61 Face – Emotions (Experimental)https://software.intel.com/sites/landingpage/realsense/camera-sdk/2014gold/documentation/html/manuals_emotion_detection_via_senseman.html Apesar de ainda experimental, implementamos um sample utilizando o módulo de emoções. OBS: ele fica separado do módulo Face. Baseando nas expressões detectadas, o algoritmo tenta prever as emoçõs do usuário, apresentando a intensidade de cada uma. 6 emoções podem ser detectadas: Irritado (ou Raiva); Descontente; Com medo; Alegre; Triste; Surpreso. Mostrar implementação...
62 Speech – Recognition Switch Weapon! Reference links:https://software.intel.com/sites/landingpage/realsense/camera-sdk/2014gold/documentation/html/manuals_speech_recognition_procedure.html https://software.intel.com/sites/landingpage/realsense/camera-sdk/2014gold/documentation/html/manuals_command_control_and_dictation.html https://software.intel.com/sites/landingpage/realsense/camera-sdk/2014gold/documentation/html/manuals_handle_recognition_events.html https://software.intel.com/sites/landingpage/realsense/camera-sdk/2014gold/documentation/html/manuals_about_audio_recording_level.html https://software.intel.com/sites/landingpage/realsense/camera-sdk/2014gold/documentation/html/manuals_about_confidence_level.html Neste modo, dizemos quais os comandos podem ser pronunciados, fazendo então um filtro para o algoritmo de detecção. Um exemplo de uso seria o comando “play” para tocar uma música na sua aplicação.
63 Your Score is: Five Hundred and Sixty Five!Speech – Synthesis Your Score is: Five Hundred and Sixty Five! https://software.intel.com/sites/landingpage/realsense/camera-sdk/2014gold/documentation/html/manuals_speech_synthesis.html Basicamente um text-to-speech. Definimos o texto a ser falado e o módulo encarrega-se de reproduzir através do AudioManager instanciado.
64 Segmentation https://software.intel.com/sites/landingpage/realsense/camera-sdk/2014gold/documentation/html/manuals_user_segmentation.html O módulo de segmentação encarrega-se de gerar as imagens removendo boa parte do background. Feita a remoção, podemos adicionar o fundo que queremos via código.
65 Object Tracking 2D 3D Reference Links:https://software.intel.com/sites/landingpage/realsense/camera-sdk/2014gold/documentation/html/manuals_object_tracking.html https://software.intel.com/sites/landingpage/realsense/camera-sdk/2014gold/documentation/html/manuals_object_tracking_via_sense_manager.html https://software.intel.com/sites/landingpage/realsense/camera-sdk/2014gold/documentation/html/manuals_configuration_and_tracking_dat.html https://software.intel.com/sites/landingpage/realsense/camera-sdk/2014gold/documentation/html/manuals_the_metaio_toolbox.html Temos dois modos de rastreamento de objetos: 2d e 3d Iremos mostrar uma implementação do Object Tracking 2d. Passamos uma imagem/foto do objeto (no caso uma tag), carregamos nas configurações do módulo e o mesmo se encarrega de rastrear. Devemos lembrar que, para o reconhecimento de objetos (3d ou 2d), devemos antes de tudo calibrar a câmera (podemos mostrar como fazer isso, mas depende do tempo que tivermos);
66 3D Scanning Reference Links:https://software.intel.com/sites/landingpage/realsense/camera-sdk/2014gold/documentation/html/manuals_object_tracking.html https://software.intel.com/sites/landingpage/realsense/camera-sdk/2014gold/documentation/html/manuals_object_tracking_via_sense_manager.html https://software.intel.com/sites/landingpage/realsense/camera-sdk/2014gold/documentation/html/manuals_configuration_and_tracking_dat.html https://software.intel.com/sites/landingpage/realsense/camera-sdk/2014gold/documentation/html/manuals_the_metaio_toolbox.html Temos dois modos de rastreamento de objetos: 2d e 3d Iremos mostrar uma implementação do Object Tracking 2d. Passamos uma imagem/foto do objeto (no caso uma tag), carregamos nas configurações do módulo e o mesmo se encarrega de rastrear. Devemos lembrar que, para o reconhecimento de objetos (3d ou 2d), devemos antes de tudo calibrar a câmera (podemos mostrar como fazer isso, mas depende do tempo que tivermos);
67 Microsoft Passport e Windows Hello
68 Going beyond passwordsProblems: Passwords are hard to remember Passwords are re-used -> server breach attacks Microsoft passport solution: User has to remember only one PIN or can use Windows Hello No secret is stored on servers -> Two factor authentication with asymmetric keys
69 Windows 10 and Microsoft Passport Simple for DeveloperReduces the cost associated with password compromise and reset Native API support for strong authentication via Universal Windows Platform JavaScript API support for browsers coming later this year.
70 Windows Hello: Sensor SupportFingerprint All current fingerprint capable devices are supported Face All current (f200) and future Intel® RealSense™ are supported for Windows Hello face authentication All devices that include an IR sensor that meets Microsoft sensor spec Iris A selection of devices will be arriving to market within the next 12 months More details on sensor requirements and support coming soon
71 Windows Hello Windows 10 support three type of biometric Auth:Fingerprint Face Iris All of three provide gesture and action recommended on the hardware supported Integrated with the Windows Biometric Framework use the same scenary and experience Share the same language for use and definition
72 Microsoft Passport: ProvisioningBuild 2015 12/6/ :06 PM Microsoft Passport: Provisioning Two Step Verification Create Microsoft Passport Register Public key with Server KeyCredentialRetrievalResult kcResult = await KeyCredentialManager.RequestCreateAsync(accountID, KeyCredentialCreationOption.FailIfExists); IBuffer userPublicKey = kcResult.Credential.RetrivePublicKey(); © 2015 Microsoft Corporation. All rights reserved. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION.
73 Microsoft Passport: UsageBuild 2015 12/6/ :06 PM Microsoft Passport: Usage Open Microsoft Passport Sign challenge from Server with Microsoft Passport Send digital signature to server KeyCredentialRetrievalResult kcResult = await KeyCredentialManager.OpenAsync(accountID); KeyCredentialOperationResult kcOpResult = await kcResult.Credential.RequestSignAsync(serverChallenge); © 2015 Microsoft Corporation. All rights reserved. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION.
74 Microsoft Passport: DeletionBuild 2015 12/6/ :06 PM Microsoft Passport: Deletion Open Microsoft Passport Delete Passport Delete Public Key on Server KeyCredentialRetrievalResult kcResult = await KeyCredentialManager.OpenAsync(accountID); KeyCredentialManager.DeleteAsync(accountID); © 2015 Microsoft Corporation. All rights reserved. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION.
75 DEMO SharpSenses
76 Reference Build SessionMicrosoft Passport and Windows Hello: Moving Beyond Passwords and Credential Theft https://channel9.msdn.com/events/Build/2015/2-639 API Windows.Security.Credentials namespace Https://msdn.microsoft.com/en-us/library/windows/apps/windows.security.credentials.aspx
77 Q&A
78 Thank you! Colocar contatos: Carrara, Felipe e João Pedro