In mid-April, Meta's Mark Zuckerberg announced newAI-based Virtual Assistant Features.One month later, Meta was granted a patent focused on adding a digital assistant to future Quest Headsets and smartglasses via a hybrid architecture built upon both client-side processes and server-side processes that in-part uses AI. More specifically, the patent covers a method that includes presenting a suggestion to a user of a head-mounted device by the head-mounted device via an assistant xbot during a dialog session between the user and the assistant xbot, wherein the suggestion is associated with a plurality of actions to be performed by an assistant system associated with the assistant xbot, accessing signals from inertial measurement unit (IMU) sensors of the head-mounted device by the head-mounted device during the dialog session, determining a head gesture performed by the user during the dialog session by an on-device head-gesture detection model and based only on the signals from the IMU sensors, and executing a first action from multiple actions by the assistant system executing on the head-mounted device, wherein the first action is selected based on the determined head gesture during the dialog session.
Detecting Head Gestures Using Inertial Measurement Unit Signals
A future Meta digital assistant system may assist the user via a hybrid architecture built upon both client-side processes and server-side processes.
In particular embodiments, the assistant system may enable users wearing head-mounted devices such as Quest VR Headset of smartglasses to provide a response using head gestures. The assistant system may understand such response by detecting the head gestures using a client-side head-gesture detection model and correlating the detected head gestures with their predefined indications. The head-gesture detection model may detect head gestures using only input from an inertial measurement unit (IMU) sensor of the head-mounted device. The client-side head-gesture detection model may be capable of inferring user head gestures (e.g., nodding, shaking side-to-side, etc.) with low latency and low power consumption.
In particular embodiments, head gesture detection may be performed by other systems rather than the assistant system. As an example and not by way of limitation, a client system (e.g., a head-mounted device) alone may detect head gestures and provide the detected head gestures to the assistant system or another application executing on the client system.
As another example and not by way of limitation, a virtual-reality (VR) system may detect head gestures when a user is wearing a VR headset and provide the detected head gestures to the assistant system or another application executing on the VR headset. As yet another example and not by way of limitation, an augmented-reality (AR) system may detect head gestures when a user is wearing an AR device and provide the detected head gestures to the assistant system or another application executing on the AR device.
In particular embodiments, the assistant system may present, by the head-mounted device via an assistant xbot, a suggestion to a user of the head-mounted device during a dialog session between the user and the assistant xbot. The suggestion may be associated with a plurality of actions to be performed by the assistant system associated with the assistant xbot
Meta's patentFIG. 12below illustrates an example artificial neural network (“ANN”) #1200; Meta's patentFIG. 2illustrates an example architecture of the assistant system. In particular embodiments, the assistant system may assist a user to obtain information or services. The assistant system 140 may enable the user to interact with the assistant system via user inputs of various modalities (e.g., audio, voice, text, vision, image, video, gesture, motion, activity, location, orientation) in stateful and multi-turn conversations to receive assistance from the assistant system. As an example and not by way of limitation, a user input may comprise an audio input based on the user's voice (e.g., a verbal command), which may be processed by a system audio API (application programming interface) on client system. The system audio API may perform techniques including echo cancellation, noise removal, beam forming, self-user voice activation, speaker identification, voice activity detection (VAD), and/or any other suitable acoustic technique in order to generate audio data that is readily processable by the assistant system. In particular embodiments, the assistant system 140 may support mono-modal inputs (e.g., only voice inputs), multi-modal inputs (e.g., voice inputs and text inputs), hybrid/multi-modal inputs, or any combination thereof.
Meta's patent FIG. 5 below illustrates an example person head with sagittal, frontal, and transverse planes; FIG. 6 illustrates example movements by a person head.
Meta's patent FIG. 7 below illustrates an example sequence diagram for user assistance based on head-gesture detection; FIGS. 8C-8E illustrates an example scenario for taking actions responsive to an up-down sagittal head nod.
Meta's patent FIGS. 10A-10D below illustrates another example scenario for taking actions responsive to a left-right transverse head shake. FIG. 10A illustrates that a user #1010 wearing a VR headset #130 a a is playing a VR game. The user's phone #130b may be on the table. b The user's phone may be a companion device for the VR headset. FIG. 10B illustrates that the assistant system #140 speaks to the user, via the VR headset, that “your brother is calling you. Would you like to answer it? #1020” FIG. 10C illustrates that the user shakes his head. FIG. 10D illustrates that the assistant system detected the shaking head gesture. The assistant system may then determine the user doesn't want to answer the phone while playing the game. As a result, the assistant system may execute an action of declining the phone call on the phone. The assistant system may further speak to the user via the VR headset stating “okay, will hang up the call. #1030”
Translated from: patentlyapple
English 
Chinese