CN116088642A - Control system and control method for wearable device and wearable device - Google Patents

Control system and control method for wearable device and wearable device Download PDF

Info

Publication number
CN116088642A
CN116088642A CN202310104728.8A CN202310104728A CN116088642A CN 116088642 A CN116088642 A CN 116088642A CN 202310104728 A CN202310104728 A CN 202310104728A CN 116088642 A CN116088642 A CN 116088642A
Authority
CN
China
Prior art keywords
user
wearable device
image
video
intelligent
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Pending
Application number
CN202310104728.8A
Other languages
Chinese (zh)
Inventor
童伟峰
张亮
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Heng Xuan Technology Beijing Co ltd
Original Assignee
Heng Xuan Technology Beijing Co ltd
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Heng Xuan Technology Beijing Co ltd filed Critical Heng Xuan Technology Beijing Co ltd
Priority to CN202310104728.8A priority Critical patent/CN116088642A/en
Publication of CN116088642A publication Critical patent/CN116088642A/en
Pending legal-status Critical Current

Links

Images

Classifications

    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F1/00—Details not covered by groups G06F3/00 - G06F13/00 and G06F21/00
    • G06F1/16—Constructional details or arrangements
    • G06F1/1613—Constructional details or arrangements for portable computers
    • G06F1/163—Wearable computers, e.g. on a belt
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F1/00—Details not covered by groups G06F3/00 - G06F13/00 and G06F21/00
    • G06F1/26—Power supply means, e.g. regulation thereof
    • G06F1/32—Means for saving power
    • G06F1/3203—Power management, i.e. event-based initiation of a power-saving mode
    • G06F1/3206—Monitoring of events, devices or parameters that trigger a change in power modality
    • G06F1/3231—Monitoring the presence, absence or movement of users
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
    • G06F3/01—Input arrangements or combined input and output arrangements for interaction between user and computer
    • G06F3/011—Arrangements for interaction with the human body, e.g. for user immersion in virtual reality
    • G06F3/013—Eye tracking input arrangements
    • G—PHYSICS
    • G10—MUSICAL INSTRUMENTS; ACOUSTICS
    • G10L—SPEECH ANALYSIS TECHNIQUES OR SPEECH SYNTHESIS; SPEECH RECOGNITION; SPEECH OR VOICE PROCESSING TECHNIQUES; SPEECH OR AUDIO CODING OR DECODING
    • G10L15/00—Speech recognition
    • G10L15/22—Procedures used during a speech recognition process, e.g. man-machine dialogue

Landscapes

  • Engineering & Computer Science (AREA)
  • Theoretical Computer Science (AREA)
  • Physics & Mathematics (AREA)
  • General Engineering & Computer Science (AREA)
  • Human Computer Interaction (AREA)
  • General Physics & Mathematics (AREA)
  • Computer Hardware Design (AREA)
  • Computational Linguistics (AREA)
  • Health & Medical Sciences (AREA)
  • Audiology, Speech & Language Pathology (AREA)
  • Acoustics & Sound (AREA)
  • Multimedia (AREA)
  • User Interface Of Digital Computer (AREA)

Abstract

The present application relates to a control system, a control method and a wearable device for a wearable device, the control system comprising an eye image sensor, a processor, an external image sensor and a wireless module, the eye image sensor being configured to acquire a first image and/or a first video of a user's eye in real time; the external image sensor is configured to acquire a second image and/or a second video of the wearable device and the surrounding environment; the processor is configured to: obtaining a gaze point and/or gaze location of a user and a pose of a wearable device; a first smart device associated with a gaze point and/or gaze location of a user is determined based on a position or location of at least one smart device, a pose of a wearable apparatus, and the gaze point and/or gaze location of the user, and a first notification instruction is sent to the first smart device for response by the first smart device to speech information uttered by the user. The control system can solve the problem of determining which smart device is to execute the user's voice instructions.

Description

Control system and control method for wearable device and wearable device
Technical Field
The present disclosure relates to the field of wearable devices, and more particularly, to a control system, a control method, and a wearable device for a wearable device.
Background
With the improvement of social progress and the improvement of living standard of people, wearable devices such as intelligent glasses, bluetooth glasses, AR glasses and the like gradually enter the life of people, and become a common object for people. The intelligent glasses and the like can be used for completing functions of shooting photos, videos, video calls, namely map navigation and the like, and can also be used for interacting with intelligent equipment. After receiving the voice command sent by the user using the intelligent glasses, the intelligent device can execute the voice command to complete the appointed action. But in the case of multiple smart devices in the environment of the user, it is not easy to determine which smart device is to respond to the user's voice instructions.
Disclosure of Invention
The present application is provided to address the above-mentioned deficiencies in the prior art. There is a need for a control system, a control method and a wearable device for a user to select a uniquely determined first smart device by determining a gaze point and/or a gaze direction of a user's gaze so that a first voice command issued by the user can be executed by the first smart device, thereby solving the problem that it is not easy to determine which smart device is used to execute the voice command of the user in case of a plurality of smart devices around the user, and improving the accuracy of the smart glasses to control the smart devices.
According to a first aspect of the present application, there is provided a control system for a wearable device, the control system comprising an eye image sensor, a processor, an external image sensor, and a wireless module, the eye image sensor configured to: and acquiring a first image and/or a first video of the eyes of the user in real time, and sending the first image and/or the first video to the processor. The external image sensor is configured to: a second image and/or a second video of the wearable device and surrounding environment is acquired and sent to the processor. The processor is configured to: performing eyeball detection based on the first image and/or the first video to obtain a gaze point and/or a gaze direction of a user; obtaining a pose of the wearable device based on the second image and/or the second video; a first smart device associated with the gaze point and/or gaze location of the user is determined based on the position or orientation of at least one smart device, the pose of the wearable apparatus, and the gaze point and/or gaze location of the user, and a first notification instruction is sent to the smart device by the wireless module so that the first smart device responds to voice information sent by the user.
According to a second aspect of the present application, there is provided a control method for a wearable device, the wearable device including a control system including an eye image sensor, a processor, an external image sensor, and a wireless module, the control method comprising: acquiring a first image and/or a first video of the eyes of the user in real time by the eye image sensor and sending the first image and/or the first video to the processor; acquiring a second image and/or a second video of the wearable device and the surrounding environment by the external image sensor and sending to the processor; by the processor: performing eyeball detection based on the first image and/or the first video of the eyes of the user to obtain a gaze point and/or a gaze direction of the user; obtaining a pose of the wearable device based on a second image and/or a second video of the wearable device and surrounding environment; a first smart device associated with the gaze point and/or gaze location of the user is determined based on the position or orientation of at least one smart device, the pose of the wearable apparatus, and the gaze point and/or gaze location of the user, and a first notification instruction is sent to the smart device by the wireless module so that the first smart device responds to voice information sent by the user.
According to a third aspect of the present application, there is provided a wearable device, characterized in that the wearable device comprises the control system for a wearable device according to the first aspect.
According to the control system, the control method and the wearable device for the wearable device, provided by the embodiments of the invention, the change of the eyeballs of the eyes of the user is obtained through the eye image sensor, the wearable device and the surrounding environment are obtained through the external image sensor, the fixation point and/or fixation position of the user and the position and the gesture of the wearable device can be obtained through the identification of the processor, and the first intelligent device corresponding to the fixation point and/or fixation position of the user can be determined by combining the position or the position of the intelligent device, so that the intelligent device which the user wants to specify can be correctly selected, the unique intelligent device can execute the user instruction, the problem that the voice instruction of the user is not easy to be executed by the intelligent device is solved under the condition that a plurality of intelligent devices are arranged around the user, and the control accuracy of the intelligent glasses to the intelligent device is improved.
Drawings
Fig. 1 shows a schematic structural diagram of a control system for a wearable device according to an embodiment of the present application;
FIG. 2 shows a schematic diagram of a user wearing a wearable device interacting with a smart device according to an embodiment of the present application;
FIG. 3 shows a flowchart of a control method for a wearable device according to an embodiment of the present application; and
fig. 4 shows a schematic structural diagram of a wearable device according to an embodiment of the present application.
Detailed Description
In order to better understand the technical solutions of the present application, the following detailed description of the present application is provided with reference to the accompanying drawings and the specific embodiments. Embodiments of the present application will now be described in further detail with reference to the accompanying drawings and specific examples, but are not intended to be limiting of the present application.
The terms "first," "second," and the like, as used herein, do not denote any order, quantity, or importance, but rather are used to distinguish one element from another. The word "comprising" or "comprises" and the like means that elements preceding the word encompass the elements recited after the word, and not exclude the possibility of also encompassing other elements.
Fig. 1 shows a schematic structural diagram of a control system for a wearable device according to an embodiment of the present application. The control system 100 includes an eye image sensor 101, a processor 103, an external image sensor 102, and a wireless module 104. The wearable device may be glasses or other types of head-mounted wearable devices, which may include smart glasses, bluetooth glasses, AR glasses, and the like. The eye image sensor 101 is configured to: a first image and/or a first video of the user's eyes is acquired in real time and sent to the processor 103. The eye image sensor 101 may be a camera, a depth sensor, an infrared sensor, or the like. Taking the camera and the glasses as examples, the camera may be disposed inside the glasses and facing the eye position, whereby the first image and/or the first video of the user's eye may be obtained in real time.
The processor may include hardware circuitry (such as hardware ISP (Image Signal Processing) modules, other hardware image, video processing circuitry), various RISC (reduced instruction set computers), DSP (Digital Signal Processing), CPU, GPU, etc. as processing units to perform corresponding functions. The processor may include only one processing unit or may include a plurality of processing units. In particular, the processing unit has thereon a number of modules implemented thereon, such as, but not limited to, an image recognition module, a positioning module, a memory and a buffer, etc. In some embodiments, custom versions of these modules may also be developed autonomously on off-the-shelf processing units. The user may implement an image recognition module, a location module, a memory, a buffer, etc. by purchasing based processing units or self-developed processing units in order to reduce power consumption and cost. For example, a user may also implement various image recognition modules, positioning modules, memories, buffers, etc. using an FPGA (field programmable gate array), and may be used to verify the stability of the hardware design.
The external image sensor 102 is configured to acquire a second image and/or a second video of the wearable device and the surrounding environment and send to the processor 103. The external image sensor 102 may be a camera, a depth sensor, an infrared sensor, and the like. Taking glasses as an example, the external image sensor 102 may be disposed at a position of a body or the like of a side portion of the glasses so that a second image and/or a second video of the glasses and the surrounding environment may be photographed. In some cases, the external image sensor 102 may capture at least a second image and/or a second video of all objects including the smart device within a visual range in front of the user's eyes in order to locate the respective smart device.
The processor 103 is configured to perform eye detection based on the first image and/or the first video to obtain a gaze point and/or a gaze location of the user. The processor 103 may extract image information from the first image and/or the first video to obtain a change in the eyeball position, and find a corresponding smart device according to the changed position, so as to obtain the smart device gazed by the user. In some embodiments, image activity detection may also be performed on the first image, and only when the first image changes, eye detection of the user is performed again, so as to obtain the gaze point and/or gaze direction of the user. Thus, the occupation of the processor by continuous eyeball detection can be avoided, and the power consumption of the processor 103 can be greatly reduced. In addition, various actions of eyes of the user, such as blinking and eyeball rotation, etc., can be detected through detection of the first image and/or the first video. Based on the various eye movements detected, the interaction of the glasses with the user may be accomplished, e.g., various instructions may be sent using a blink movement confirmation. In addition, actions such as blinking may be used as confirmation of a menu or a dialog box, or by virtually displaying the menu and/or the dialog box buttons by using smart glasses somewhere in space, the wearer may look at the menu or the dialog box buttons for more than a certain time to act as a key-press action of the buttons.
The processor 103 may be further configured to obtain a pose of the wearable device based on the second image and/or the second video, wherein the pose refers to a position or orientation of the wearable device and a pose of the wearable device. When the wearable device worn by the user is in a different pose and/or has a different relative positional relationship with the subject, the second image and/or second video captured thereof is also different, and thus in some embodiments, the processor 103 may determine the pose and position or orientation of the wearable device based on the second image and/or second video. In some embodiments, SLAM (Simultaneous Localization and Mapping, synchronous positioning and mapping) techniques may be utilized to determine the pose of the wearable device. For example, the wearer may be instructed to make a preset posture in advance and take a reference image of the external environment, and a series of actual images of the external environment taken later are registered with the reference image to calculate the spatial transformation of the former with respect to the latter, so as to solve the posture of the wearable device. For another example, a fixed marker may be set in the external environment and its reference orientation determined in advance when the wearer makes a preset posture, at least part of the marker is recognized from a series of actual images of the external environment taken later, and a transition between the recognized actual orientation of the marker and the reference orientation is determined, thereby resolving the posture and the position or orientation of the wearable device.
In other embodiments, the wearable device may also be fitted with an Inertial Measurement Unit (IMU), such as a gyroscope or accelerometer, for example, where the wearable device is glasses, may be fitted at the temple, frame, etc. of a single or left and right pair of glasses. However, the IMU is characterized in that the change of the posture of the wearable device can be accurately obtained in a short time, and the measurement error is easily generated due to the influence of zero offset for a long time, so that the posture of the wearable device can be determined by using the IMU, and the zero offset of the IMU is calibrated by using the second image and/or the second video, so that the posture, the position or the azimuth of the wearable device with excellent real-time performance can be obtained with significantly higher accuracy.
The processor 103 is further configured to determine whether the user's gaze point and/or gaze direction has a smart device based on the second image and/or second video for image recognition. If no smart device is detected at the gaze point or gaze location of the user based on the second image and/or the second video, then no first smart device is present, such as the location of a wall, sofa, window, etc. in the gaze chamber; if there is a smart device, indicating that there is at least one smart device in the environment, then a determination process of the first smart device is performed based on the location or orientation of the at least one smart device, such as finding a television or a sound box at the gaze point and/or gaze orientation of the user.
The processor 103 is further configured to determine a first smart device associated with the gaze point and/or gaze location of the user based on the location or position of at least one smart device, the pose of the wearable apparatus, and the gaze point and/or gaze location of the user, send a first notification instruction to the smart device via the wireless module 104 so that the first smart device responds to the voice information uttered by the user. The position or the azimuth of the intelligent device can be the position of the intelligent device relative to the ground, the position relative to the indoor ground or the position or the azimuth relative to the wearable device, and different references can be selected to represent the position or the azimuth of the intelligent device according to the requirements. The gaze point and/or the gaze location of the user can be accurately determined by combining the gaze point and/or the gaze location of the user with the pose of the wearable device, and the intelligent device currently gazed at by the user can be accurately determined based on the position or the location of the at least one intelligent device. The first notification instruction may be notification information sent to the gazed smart device alone, and the first smart device receives the first notification instruction. For example, through the pre-stored WIFI communication address or bluetooth communication address corresponding to each intelligent device, a first notification instruction is sent to the corresponding intelligent device, and the first intelligent device that receives the first notification instruction can then respond to the voice information sent by the user.
In some embodiments, a first notification instruction containing information about the first smart device is sent to the smart device via the wireless module 104 so that the first smart device responds to the voice information sent by the user. The first notification instruction may be transmitted by broadcasting the transmission information. The first intelligent device related information included in the first notification instruction may be a name of the first intelligent device, a MAC address of the first intelligent device, a bluetooth address of the first intelligent device, a product shape feature, and other information that may identify the first intelligent device. After receiving the first notification instruction, each intelligent device can analyze the first notification instruction, if the analyzed related information of the first intelligent device is the same as the related information of the intelligent device, the instruction is the first intelligent device, the intelligent device responds to the voice information of the voice information sent by the user, and other intelligent devices do not perform the following operation. Thus, the intelligent device at which the user looks can be accurately positioned.
By way of example only, where there is only one smart device in the environment and the location of the smart device is known, it may be determined that the only smart device is the first smart device when it is determined that there is a smart device at the gaze point and/or gaze location of the user. In the case where there are two smart devices in the environment, and only the position or orientation of one of the smart devices is known, when it can be determined that it is looking at a certain smart device by the gaze point and/or gaze orientation of the user, it can be further determined whether the gazed at is a smart device of a known position, and if not, it is necessarily the other smart device.
In other embodiments, where there are N (N is greater than or equal to 2) smart devices in the environment, a first smart device associated with the gaze point and/or gaze location of the user may be determined based on a position or location of at least one of the smart devices, a pose of the wearable apparatus, and the gaze point and/or gaze location of the user. For example, there are 5 intelligent devices in a room, firstly, the gaze point and/or gaze location of the user are determined, and then, a certain intelligent device corresponding to the gaze point and/or gaze location of the user is determined by combining the positions or locations of the N intelligent devices. The positions or orientations of the N intelligent devices can be determined based on the position or orientation of at least one intelligent device and the result of detecting the second image and/or the second video, and the positions of other devices can be obtained through the position of at least one intelligent device when the second image and/or the second video is subjected to image recognition. The positions or orientations of the N smart devices may also be determined based on the position or orientation of at least one smart device in combination with the wireless connection of the N-1 smart devices to the wearable apparatus, respectively, and other smart devices may report the positions of the smart devices relative to the known positions or orientations to the wearable apparatus via the wireless connection, so that the positions of the N smart devices may be obtained.
In other embodiments, where there are N (N is greater than or equal to 2) smart devices in the environment, a first smart device associated with the gaze point and/or gaze location of the user may be determined based on the locations or orientations of at least N-1 smart devices therein, the pose of the wearable apparatus, and the gaze point and/or gaze location of the user. For example, in an indoor environment, 4 intelligent devices such as a sound box, an alarm clock, a refrigerator, a kitten's fairy and the like may be arranged near the user, the position relationship between the intelligent devices around the user is obtained through the second image and/or the second video, and the first intelligent device associated with the gaze point and/or the gaze direction of the user can be determined by combining the positions or the orientations of at least 3 intelligent devices. After determining the first smart device, a first notification instruction may be sent to the first smart device by the wireless module 104 automatically or in the participation of the user, and the voice information sent by the user may be executed by the first smart device in a subsequent process. Therefore, when it is not determined that the user is looking at a certain smart device, the wearable device does not issue the first notification instruction, and in this case, each smart device does not execute any voice instruction, so that misoperation of the smart device due to recognition of the user voice in the case that the user is not intended to operate the smart device is avoided.
According to the control system 100 for the wearable device, the intelligent device for executing the voice information sent by the user is determined through the fixation of the user and the first notification instruction is sent to the intelligent device, so that on one hand, the operation flow of the intelligent device for selecting and executing the voice instruction is greatly simplified, misoperation of the intelligent device without executing the instruction is avoided, and therefore power consumption of each intelligent device can be greatly reduced.
In some embodiments, when the wearable device has the first microphone, the wearable device recognizes the voice information to obtain a first voice command, and adds the first voice command to the first notification command to send to the intelligent device or sends the first voice command and the first notification command to the intelligent device at the same time. Therefore, the microphone of the intelligent device is not required to collect audio signals, and the microphone of the intelligent device is not required to be normally open, so that the power consumption of the intelligent device can be reduced, and the situation that other voices are arranged around a user or the user continuously sends out a plurality of voice commands is avoided, and wrong voice commands are executed. In other embodiments, under the condition that the intelligent device is provided with the second microphone, each intelligent device respectively collects and identifies the voice information to obtain a first voice instruction before receiving the first notification instruction, and the first voice instruction is executed by the first intelligent device which receives the first notification instruction; or the first intelligent device after receiving the first notification instruction collects and identifies the voice information to obtain a first voice instruction, and executes the first voice instruction. The power consumption of the wearable device can be reduced. In addition, the wearable device and the intelligent device can also send the voice information to the cloud server for processing of voice recognition. In the case where both the wearable device and the smart device have microphones, it is possible to set in advance which of the smart device and the wearable device is to perform the collection of the voice information and the recognition process.
In some embodiments, in the case of authentication performed by the wearable device, the first notification instruction further comprises authentication pass information generated in the case of a first user identity and a second user identity matching, wherein the first user identity is obtained based on eye recognition of the first image and/or the first video, and the second user identity is obtained based on recognition of voice information acquired by a first microphone of the wearable device, the processor is further configured to: and converting the voice information acquired by the first microphone into a first voice instruction, and sending the first voice instruction to the intelligent equipment through the wireless module, so that the first intelligent equipment executes the first voice instruction under the condition that the user authentication passes. Taking glasses as an example, since there may be other people or devices around the user making sounds, in order to avoid interference of other people or devices than the wearer of the glasses, it is necessary to perform identity matching on the voice information collected by the microphone to verify whether it is the voice of the wearer of the glasses. Under the condition that the glasses execute the identity verification process, a verification result is sent to the first intelligent device, and the first intelligent device is prevented from responding to the voice of the non-glasses wearing user. In addition, the glasses can also convert the voice information of the user into a first voice instruction containing instruction information sent by the user and forward the first voice instruction to the intelligent equipment at the same time, so that the first intelligent equipment in the plurality of intelligent equipment executes the voice information corresponding to the verification identity, further avoid executing wrong voice information and improve the reliability of implementing control on the intelligent equipment through the wearable device.
In some embodiments, the first notification instruction further includes a first user identity, wherein the first user identity is derived based on eye recognition of the first image and/or first video; the intelligent device is provided with a second microphone so as to recognize the voice information acquired by the second microphone to obtain a second user identity under the condition that the intelligent device is a first intelligent device, and respond to the voice information sent by the user under the condition that the first user identity and the second user identity are judged to be matched. The intelligent device with the second microphone can also collect voice information sent by the user, and under the condition that the intelligent device is provided with the processing unit and can complete user identity verification, the intelligent device can complete the identity verification process. Although the first notification instruction can be received by each intelligent device, only the first intelligent device executes the authentication process after each intelligent device analyzes the first notification instruction, so that the power consumption of other intelligent devices can be reduced. In the case where both the wearable device and the smart device have microphones, the authentication process by which of the smart device and the wearable device is performed may be preconfigured before the wearable device is used. For example, the wearable device may be preconfigured according to a time period of the world time, or may be preconfigured according to a time when the user uses the wearable device. Under the condition that the intelligent device completes identity verification, the wearable device sends the obtained first user identity to the intelligent device, so that the intelligent device completes identity verification. Under the condition of identity matching, the voice information sent by the user is responded, wrong voice information (such as the words of other people beside) is prevented from being executed, only the voice sent by the wearer of the wearable device is executed, and the reliability of controlling the intelligent equipment through the wearable device is improved.
In some embodiments, the first voice command is obtained by identifying voice information sent by a user by one of the wearable device, the first intelligent device and the cloud end. Under the condition that the identity verification is executed by the wearable equipment, a first microphone is arranged on the wearable device, for example, the first intelligent equipment is an alarm clock, the wearable device can utilize voice information of a user to carry out the identity verification, after the identity verification is passed, the voice information is recognized as a first voice instruction, the first voice instruction is sent to the first intelligent equipment, so that the first intelligent equipment executes the first voice instruction, the intelligent equipment without the microphone such as the alarm clock can still execute the voice instruction of the user, and the use convenience of the wearable device is improved. The first voice command may be sent to the intelligent device in the first notification command or sent to the intelligent device separately at the same time, so that the condition that the intelligent device executes the voice command is confused, for example, the condition of two voice commands sent by a user continuously is avoided, the two commands may be different, each command is sent to the intelligent device together with the first notification command, and the first intelligent device can execute the first voice command first and then execute the second voice command.
The intelligent device is provided with the second microphone, and under the condition that the first intelligent device can perform identity verification, for example, the first intelligent device is a sound box or a cat fairy, after the first intelligent device receives voice input of a user, the first intelligent device can perform identity verification by utilizing voice information of the user, and after the first intelligent device passes the verification, the voice information is recognized as a first voice instruction and executed, so that the power consumption of the wearable device can be reduced.
In addition, the first intelligent device can also forward the corresponding voice information to the cloud for voice recognition to obtain a first voice command after the voice information of the user passes the identity verification, and then execute the first voice command; the wearable device can also transmit the corresponding voice information to the cloud for voice recognition after the voice information of the user passes through identity verification, so as to obtain a first voice instruction, and then the first voice instruction is included in the first notification instruction and sent to the intelligent device, or the first voice instruction and the first notification instruction are separately and simultaneously sent to the intelligent device. Therefore, on one hand, the intelligent equipment without voice recognition capability can successfully execute the first voice instruction, and on the other hand, the processing load of voice recognition is dispersed to each intelligent equipment as much as possible, so that the processing load and the power consumption of the wearable device end are avoided.
In some embodiments, the processor 103 may extract information such as eye white, iris, pupil, etc. from the image information via the first image and/or the first video, and compare the extracted information with pre-stored biometric information associated with the eye of the user, and obtain and record the first user identity of the wearer of the wearable device via biometric identification.
By way of example only, a user using a wearable device may pre-save the eyeball characteristics and the voice characteristics corresponding to the identity ID in order to facilitate comparison verification between the first user identity and the second user identity when needed, where the user using the same wearable device may be multiple.
In some embodiments, the second user identity is obtained by voiceprint recognition of voice information collected by the first microphone or the second microphone, and/or the second user identity is obtained by voice feature detection of voice information collected by the first microphone or the second microphone. The unique second user identity can be determined by identifying the voice input of the user in terms of sound wave spectrum or identifying prosodic features and/or spectral features in the voice, and the like, so that wrong voice instructions are avoided being executed.
In some embodiments, the second user identity is obtained by identifying voice information sent by the user by one of the wearable device, the first smart device and the cloud. The wearable device may set a microphone to collect voice input of the user, and the processor 103 of the wearable device may perform a voice recognition process to obtain voice characteristics of the user, and obtain the second user identity through matching, so that power consumption of the first smart device may be reduced. The first intelligent device can also set a microphone to collect voice input of a user, for example, the first intelligent device is a sound box or a cat fairy, and the matching process of the first user identity and the second user identity can be performed according to voice information, so that the power consumption of the wearable device can be reduced. In addition, the wearable device or the first intelligent device can also forward the voice input of the user to the cloud for voice information recognition, so that the voice characteristics of the user are obtained. The wearable device or the first intelligent device can respectively input voice characteristics of a user using the wearable device in advance, and when the wearable device is used, the result after voice input recognition is matched to obtain the identity of the user so as to reduce the power consumption of the wearable device or the first intelligent device.
In some embodiments, performing eyeball detection based on the first image and/or the first video to obtain a gaze point and/or a gaze location of the user specifically includes: identifying a first image or a first video of the eyes of the user and calculating the motion vector of the eyeball so as to obtain the motion coordinate of the eyeball; and obtaining the gazing point and/or gazing position of the user based on the motion coordinates of the eyeballs. The first image or the first video is used for extracting image information, so that the position change of the center position of the eyeball on the image can be obtained, and the movement direction and the moved position of the center of the eyeball can be reflected in the coordinate system through the established coordinate system, so that the fixation point and/or the fixation azimuth of the eyeball can be positioned. The gaze point and/or gaze direction of the eyeballs of the user are tracked and positioned, so that the interaction between the wearable device and the user can be conveniently and accurately completed in real time, and the convenience of using the wearable device is improved.
In some embodiments, the processor 103 may be further configured to: based on the second image and/or second video, and the pose of the wearable device, a location or orientation of the at least one smart device is determined using SLAM technology. The position information of the wearable device and the surrounding environment can be extracted through the second image and/or the second video, and the position or the azimuth of at least one of intelligent positioning sound boxes, intelligent refrigerators, alarm clocks and other intelligent equipment near the wearable device can be positioned relative to the wearable device in the map by utilizing SLAM to build the map in combination with the position information of the surrounding environment and the position and the gesture of the wearable device. The position and posture of the wearable device are different, and the position or orientation of the smart device with respect to the wearable device is different, because after the position and posture of the wearable device are changed, the angle, etc. of the smart device with respect to the wearable device may be changed. So through the position appearance of wearable device, utilize SLAM to confirm the position or the position of at least one smart machine, be convenient for follow-up confirm the position or the position of first smart machine to the smart machine that accurate location user gazes responds the voice command of user.
In some embodiments, where the wearable device further has an inertial measurement unit IMU, the processor 103 is further configured to: based on the second image and/or the second video, and the pose of the wearable device, determining the position or orientation of the at least one smart device with the aid of an IMU using SLAM. The IMU may be a sensor element such as a gyroscope and accelerometer that may better assist the SLAM in determining the location or orientation of at least one smart device.
In some embodiments, the location or position of the at least one smart device is determined by wireless positioning between the wearable apparatus and the at least one smart device, and/or the location or position of the at least one smart device is determined by wireless positioning between the respective smart devices. The wearable device and at least one smart device or each smart device may be positioned wirelessly by means of radio frequency identification and sensors to determine each other's position. When the positions of the intelligent devices are determined through wireless positioning, the wearable device can send the wireless positioning positions corresponding to other intelligent devices to the wearable device based on the position or the azimuth of at least one intelligent device, so that the relative positions of the other intelligent devices can be obtained, and the first intelligent device of the gaze point and/or the gaze azimuth is judged by combining the positions and the postures of the wearable device, so that the intelligent device gazed by the user can be accurately positioned to respond to the voice instruction of the user.
In some embodiments, based on the second image and/or the second video, and the pose of the wearable device, the position or orientation of the at least one smart device may be determined by using a combination of SLAM and wireless positioning. The accuracy of positioning can be further improved.
In some embodiments, the control system 100 is further configured to: the indication information of the first intelligent device is received through the wireless module 104, or the indication information of the first intelligent device is received and played through the wireless module 104. After the first intelligent device executes the first voice command sent by the user, feedback needs to be sent to the user, such as the first voice command of inquiring time sent by the user, the first intelligent device needs to send indication information containing time to a wearable device worn by the user so that the user knows the indication information, and privacy is good in this way. The wearable device may also play the indication information, such as broadcasting the current time. The comfort and convenience of use of the wearable device can be further enhanced through such interaction. In addition, in other embodiments, the first intelligent device may also directly play the indication information by using the playing device, so that the user knows the indication information.
In some embodiments, the wireless module 104 is implemented by at least one of WIFI communication, classical bluetooth communication, BLE communication, LE audio, ANT communication, RF4CE communication, zigbee communication, NFC communication, UWB communication module. Bluetooth module or WIFI module etc. can be configured in the wearable device to realize different communication modes with the intelligent device.
Fig. 2 shows a schematic diagram of a user wearing a wearable apparatus interacting with a smart device according to an embodiment of the application. The user 201 wears the wearable device 202, the sound box 203, the refrigerator 204, the washing machine 205 and the alarm clock 206 are arranged around the user 201, when the user looks at the sound box 203, the wearable device 202 sends a first notification instruction 207 to the sound box 203 after judging that the sound box 203 is a first intelligent device, and the first notification instruction can comprise information related to the first intelligent device, an authentication passing result and a first voice instruction. In addition, the sound box can also utilize the second microphone to collect voice information, directly recognize the voice information sent by the user as a first voice command, and make feedback according to the first voice command, and at the moment, the first notification command can comprise information related to the first intelligent equipment and authentication passing information.
In other embodiments, the wearable device 202 may also collect the voice information sent by the user by using the first microphone, and directly forward the voice information sent by the user to the speaker, where the speaker performs the recognition process of the voice information.
After the speaker 203 executes the first voice instruction, the indication information 208 is sent to the wearable device 202.
The embodiment of the application also provides a control method for the wearable device, and the control method for the wearable device according to the embodiment of the application is specifically described below.
Fig. 3 shows a flowchart of a control method for a wearable device according to an embodiment of the present application. The wearable device includes a control system including an eye image sensor, a processor, an external image sensor, and a wireless module, the control method comprising: step 301, acquiring a first image and/or a first video of a user's eyes in real time by the eye image sensor, and sending the first image and/or the first video to the processor. Step 302, acquiring a second image and/or a second video of the wearable device and the surrounding environment by the external image sensor, and sending the second image and/or the second video to the processor. And 303, performing eyeball detection by the processor based on the first image and/or the first video of the eyes of the user to obtain the gazing point and/or gazing position of the user. Step 304, obtaining a pose of the wearable device based on the wearable device and a second image and/or a second video of a surrounding environment. Step 305 of determining a first smart device associated with a gaze point and/or gaze location of the user based on a position or orientation of at least one smart device, a pose of the wearable apparatus and the gaze point and/or gaze location of the user, sending a first notification instruction to the smart device via the wireless module so that the first smart device responds to voice information uttered by the user. Wherein the order between steps 301 and 302 and between steps 303 and 304 may be reversed. Therefore, when it is not determined that the user is looking at a certain smart device, the wearable device does not issue the first notification instruction, and in this case, each smart device does not execute any voice instruction, so that misoperation of the smart device due to recognition of the user voice in the case that the user is not intended to operate the smart device is avoided.
In some embodiments, in the case of authentication performed by the wearable device, the first notification instruction further includes authentication pass information generated in the case of a first user identity and a second user identity matching, wherein the first user identity is obtained based on eye recognition of the first image and/or the first video, and the second user identity is obtained based on recognition of voice information acquired by a first microphone of the wearable device, by the processor: and converting the voice information acquired by the first microphone into a first voice instruction, and sending the first voice instruction to the intelligent equipment through the wireless module, so that the first intelligent equipment executes the first voice instruction under the condition that the user authentication passes. And wrong voice information is avoided being executed, and the reliability of controlling the intelligent equipment through the wearable device is improved.
In some embodiments, the first notification instruction further includes a first user identity, wherein the first user identity is derived based on eye recognition of the first image and/or first video; the intelligent device is provided with a second microphone so as to recognize the voice information acquired by the second microphone to obtain a second user identity under the condition that the intelligent device is a first intelligent device, and respond to the voice information sent by the user under the condition that the first user identity and the second user identity are judged to be matched. Avoiding execution of erroneous speech information (such as speech by other people nearby), executing only speech uttered by the wearer of the wearable device improves the reliability of the control of the smart device by the wearable device.
In some embodiments, the second user identity is obtained by voiceprint recognition of voice information collected by the first microphone or the second microphone, and/or the second user identity is obtained by voice feature detection of voice information collected by the first microphone or the second microphone. A unique second user identity may be determined avoiding execution of erroneous speech instructions.
In some embodiments, the second user identity is obtained by identifying voice information by one of the wearable device, the first smart device, and the cloud. To reduce power consumption of the wearable device or the first smart device.
In some embodiments, performing eyeball detection based on the first image and/or the first video to obtain a gaze point and/or a gaze location of the user specifically includes: identifying an image or video of the eyes of the user and calculating the motion vector of the eyeballs so as to obtain the motion coordinates of the eyeballs; and obtaining the gazing point and/or gazing position of the user based on the motion coordinates of the eyeballs. The gaze point and/or gaze direction of the eyeballs of the user are tracked and positioned, so that the interaction between the wearable device and the user can be conveniently and accurately completed in real time, and the convenience of using the wearable device is improved.
In some embodiments, the control method further comprises, by the processor: based on the second image and/or second video, and the pose of the wearable device, a location or orientation of the at least one smart device is determined using SLAM. So through the position appearance of wearable device, utilize SLAM to confirm the position or the position of at least one smart machine, be convenient for follow-up confirm the position or the position of first smart machine to the smart machine that accurate location user gazes responds the voice command of user.
In some embodiments, the wearable device further has an inertial measurement unit IMU, the control method further comprising, by the processor: based on the second image and/or the second video, and the pose of the wearable device, determining the positions or orientations of the at least two intelligent devices with the aid of an IMU using SLAM. The IMU may be a sensor element such as a gyroscope and an accelerometer, and may better assist SLAM to obtain the pose of the wearable device and the position or orientation of the at least one smart device.
In some embodiments, the location or position of the at least one smart device is determined by wireless positioning between the wearable apparatus and the at least two smart devices, and/or the location or position of the at least one smart device is determined by wireless positioning between the respective smart devices. When the positions of the intelligent devices are determined through wireless positioning, the wearable device can send the wireless positioning positions corresponding to other intelligent devices to the wearable device based on the position or the azimuth of at least one intelligent device, so that the relative positions of the other intelligent devices can be obtained, and the first intelligent device of the gaze point and/or the gaze azimuth is judged by combining the positions and the postures of the wearable device, so that the intelligent device gazed by the user can be accurately positioned to respond to the voice instruction of the user.
In some embodiments, the control method further includes receiving, by the wireless module, the indication information of the first smart device, or receiving, by the wireless module, and playing the indication information of the first smart device. The comfort and convenience of use of the wearable device can be further enhanced through such interaction.
Fig. 4 shows a schematic structural diagram of a wearable device according to an embodiment of the present application. The wearable device 400 comprises a control system 401.
Furthermore, although exemplary embodiments have been described herein, the scope thereof includes any and all embodiments having equivalent elements, modifications, omissions, combinations (e.g., of the various embodiments across), adaptations or alterations as pertains to the present application. Elements in the claims are to be construed broadly based on the language employed in the claims and are not limited to examples described in the present specification or during the practice of the present application, which examples are to be construed as non-exclusive. It is intended, therefore, that the specification and examples be considered as exemplary only, with a true scope and spirit being indicated by the following claims and their full scope of equivalents.
The above description is intended to be illustrative and not restrictive. For example, the above-described examples (or one or more aspects thereof) may be used in combination with each other. For example, other embodiments may be used by those of ordinary skill in the art upon reading the above description. In addition, in the above detailed description, various features may be grouped together to streamline the application. This is not to be interpreted as an intention that the features of the non-claimed application are essential to any claim. Rather, the subject matter of the present application is capable of less than all features of an embodiment of a particular application. Thus, the following claims are hereby incorporated into the detailed description as examples or embodiments, with each claim standing on its own as a separate embodiment, and it is contemplated that these embodiments may be combined with one another in various combinations or permutations. The scope of the invention should be determined with reference to the appended claims, along with the full scope of equivalents to which such claims are entitled.
The above embodiments are only exemplary embodiments of the present application and are not intended to limit the present invention, the scope of which is defined by the claims. Various modifications and equivalent arrangements may be made to the present invention by those skilled in the art, which modifications and equivalents are also considered to be within the scope of the present invention.

Claims (20)

1. A control system for a wearable device, characterized in that the control system comprises an eye image sensor, a processor, an external image sensor and a wireless module,
the eye image sensor is configured to: collecting a first image and/or a first video of eyes of a user in real time, and sending the first image and/or the first video to the processor;
the external image sensor is configured to: acquiring a second image and/or a second video of the wearable device and the surrounding environment, and sending the second image and/or the second video to the processor;
the processor is configured to:
performing eyeball detection based on the first image and/or the first video to obtain a gaze point and/or a gaze direction of a user; obtaining a pose of the wearable device based on the second image and/or the second video;
a first smart device associated with the gaze point and/or gaze location of the user is determined based on the position or orientation of at least one smart device, the pose of the wearable apparatus, and the gaze point and/or gaze location of the user, and a first notification instruction is sent to the smart device by the wireless module so that the first smart device responds to voice information sent by the user.
2. The control system of claim 1, wherein the first notification instructions further comprise authentication pass information in the case of authentication performed by the wearable device, the authentication pass information generated in the case of a matching of a first user identity based on eye recognition of the first image and/or first video and a second user identity based on recognition of voice information acquired by a first microphone of the wearable device, the processor further configured to:
and converting the voice information acquired by the first microphone into a first voice instruction, and sending the first voice instruction to the intelligent equipment through the wireless module, so that the first intelligent equipment executes the first voice instruction under the condition that the user authentication passes.
3. The control system of claim 1, wherein the first notification instruction further comprises a first user identity, wherein the first user identity is based on eye recognition of the first image and/or first video;
the intelligent device is provided with a second microphone so as to recognize the voice information acquired by the second microphone to obtain a second user identity under the condition that the intelligent device is a first intelligent device, and respond to the voice information sent by the user under the condition that the first user identity and the second user identity are judged to be matched.
4. A control system according to claim 2 or 3, wherein the second user identity is obtained by voiceprint recognition of the voice information collected by the first or second microphone and/or the second user identity is obtained by voice feature detection of the voice information collected by the first or second microphone.
5. The control system of claim 2 or 3, wherein the second user identity is obtained by identifying voice information sent by a user by one of the wearable device, the first smart device, and a cloud.
6. A control system according to any of claims 1-3, characterized in that performing eye detection based on the first image and/or first video to obtain a gaze point and/or gaze location of a user comprises in particular:
identifying a first image or a first video of the eyes of the user and calculating the motion vector of the eyeball so as to obtain the motion coordinate of the eyeball;
and obtaining the gazing point and/or gazing position of the user based on the motion coordinates of the eyeballs.
7. The control system of any of claims 1-3, wherein the processor is further configured to:
Based on the second image and/or second video, and the pose of the wearable device, a location or orientation of the at least one smart device is determined using SLAM.
8. A control system according to any of claims 1-3, wherein the wearable device further has an inertial measurement unit IMU, the processor being further configured to:
based on the second image and/or the second video, and the pose of the wearable device, determining the position or orientation of the at least one smart device with the aid of an IMU using SLAM.
9. A control system according to any of claims 1-3, characterized in that the position or orientation of the at least one smart device is determined by wireless positioning between the wearable apparatus and the at least two smart devices, and/or the position or orientation of the at least one smart device is determined by wireless positioning between the respective smart devices.
10. A control system according to any one of claims 1-3, wherein the control system is further configured to: and receiving the indication information of the first intelligent device through the wireless module, or receiving and playing the indication information of the first intelligent device through the wireless module.
11. A control method for a wearable device, the wearable device comprising a control system including an eye image sensor, a processor, an external image sensor, and a wireless module, the control method comprising:
acquiring a first image and/or a first video of the eyes of the user in real time by the eye image sensor and sending the first image and/or the first video to the processor;
acquiring a second image and/or a second video of the wearable device and the surrounding environment by the external image sensor and sending to the processor;
by the processor:
performing eyeball detection based on the first image and/or the first video of the eyes of the user to obtain a gaze point and/or a gaze direction of the user;
obtaining a pose of the wearable device based on a second image and/or a second video of the wearable device and surrounding environment;
a first smart device associated with the gaze point and/or gaze location of the user is determined based on the position or orientation of at least one smart device, the pose of the wearable apparatus, and the gaze point and/or gaze location of the user, and a first notification instruction is sent to the smart device by the wireless module so that the first smart device responds to voice information sent by the user.
12. The control method according to claim 11, wherein in case of authentication performed by the wearable device, the first notification instruction further contains authentication passing information generated in case of matching of a first user identity obtained based on eyeball recognition of the first image and/or first video and a second user identity obtained based on recognition of voice information acquired by a first microphone of the wearable device, by the processor:
and converting the voice information acquired by the first microphone into a first voice instruction, and sending the first voice instruction to the intelligent equipment through the wireless module, so that the first intelligent equipment executes the first voice instruction under the condition that the user authentication passes.
13. The control method according to claim 11, wherein the first notification instruction further includes a first user identity, wherein the first user identity is obtained based on eyeball recognition of the first image and/or first video;
the intelligent device is provided with a second microphone so as to recognize the voice information acquired by the second microphone to obtain a second user identity under the condition that the intelligent device is a first intelligent device, and respond to the voice information sent by the user under the condition that the first user identity and the second user identity are judged to be matched.
14. Control method according to claim 12 or 13, characterized in that the second user identity is obtained by voiceprint recognition of the voice information collected by the first or second microphone and/or the second user identity is obtained by voice feature detection of the voice information collected by the first or second microphone.
15. The control method according to claim 12 or 13, wherein the second user identity is obtained by identifying voice information by one of the wearable device, the first smart device, and a cloud.
16. The control method according to any one of claims 11-13, wherein performing eye detection based on the first image and/or the first video to obtain a gaze point and/or a gaze location of the user specifically comprises:
identifying an image or video of the eyes of the user and calculating the motion vector of the eyeballs so as to obtain the motion coordinates of the eyeballs;
and obtaining the gazing point and/or gazing position of the user based on the motion coordinates of the eyeballs.
17. The control method according to any one of claims 11 to 13, characterized in that the control method further comprises, by the processor: based on the second image and/or second video, and the pose of the wearable device, a location or orientation of the at least one smart device is determined using SLAM.
18. The control method according to any one of claims 11-13, wherein the wearable device further has an inertial measurement unit, IMU, the control method further comprising, by the processor:
based on the second image and/or the second video, and the pose of the wearable device, determining the positions or orientations of the at least two intelligent devices with the aid of an IMU using SLAM.
19. Control method according to any of claims 11-13, characterized in that the position or orientation of the at least one smart device is determined by wireless positioning between the wearable apparatus and the at least two smart devices and/or the position or orientation of the at least one smart device is determined by wireless positioning between the respective smart devices.
20. A wearable device, characterized in that the wearable device comprises the control system for a wearable device of claim 10.
CN202310104728.8A 2023-01-30 2023-01-30 Control system and control method for wearable device and wearable device Pending CN116088642A (en)

Priority Applications (1)

Application Number Priority Date Filing Date Title
CN202310104728.8A CN116088642A (en) 2023-01-30 2023-01-30 Control system and control method for wearable device and wearable device

Applications Claiming Priority (1)

Application Number Priority Date Filing Date Title
CN202310104728.8A CN116088642A (en) 2023-01-30 2023-01-30 Control system and control method for wearable device and wearable device

Publications (1)

Publication Number Publication Date
CN116088642A true CN116088642A (en) 2023-05-09

Family

ID=86206205

Family Applications (1)

Application Number Title Priority Date Filing Date
CN202310104728.8A Pending CN116088642A (en) 2023-01-30 2023-01-30 Control system and control method for wearable device and wearable device

Country Status (1)

Country Link
CN (1) CN116088642A (en)

Similar Documents

Publication Publication Date Title
US10628714B2 (en) Entity-tracking computing system
EP2509070B1 (en) Apparatus and method for determining relevance of input speech
US11407106B2 (en) Electronic device capable of moving and operating method thereof
CN104820488B (en) User's directional type personal information assistant
US20210196174A1 (en) Apparatus, method and program for determining a cognitive state of a user of a mobile device
KR102481486B1 (en) Method and apparatus for providing audio
CN102932212A (en) Intelligent household control system based on multichannel interaction manner
JP7351383B2 (en) Information processing device, information processing method, and program
CN109145847B (en) Identification method and device, wearable device and storage medium
US20180129303A1 (en) Information processing apparatus, information processing method, and program
CN112106004A (en) Information processing apparatus, information processing method, and program
KR101728707B1 (en) Method and program for controlling electronic device by wearable glass device
WO2023081605A1 (en) Context-aided identification
US11996093B2 (en) Information processing apparatus and information processing method
JP2020077271A (en) Display unit, learning device, and method for controlling display unit
US10627898B2 (en) Control method, control device, system and motor vehicle comprising such a control device
CN116088642A (en) Control system and control method for wearable device and wearable device
US20230120092A1 (en) Information processing device and information processing method
Bellotto A multimodal smartphone interface for active perception by visually impaired
CN110600024A (en) Operation terminal, voice input method, and computer-readable recording medium
KR102308970B1 (en) System and method for inputting touch signal by wearable glass device
Limbu et al. Experiences with a barista robot, FusionBot
KR20240031830A (en) DISPLAYING METHOD AND APPARATUS WHILE IN WEARING HMD(head mounted display) device
CN118116065A (en) Action recognition method, communication system and electronic equipment

Legal Events

Date Code Title Description
PB01 Publication
PB01 Publication
SE01 Entry into force of request for substantive examination
SE01 Entry into force of request for substantive examination