WO2022001791A1

WO2022001791A1 - Intelligent device interaction method based on ppg information

Info

Publication number: WO2022001791A1
Application number: PCT/CN2021/101863
Authority: WO
Inventors: 邹永攀; 朱培钊
Original assignee: 深圳大学
Priority date: 2020-07-01
Filing date: 2021-06-23
Publication date: 2022-01-06
Also published as: CN111831116A

Abstract

Disclosed is an intelligent device interaction method based on PPG information. The method comprises: collecting a PPG signal from an ear of a target user; processing the collected PPG signal to extract a waveform feature value of the PPG signal that includes a head action; and inputting the extracted waveform feature value of the PPG signal into a deep learning model to identify the type of the current head action of the target user, wherein the deep learning model is obtained by means of training with the waveform feature value of the PPG signal of the target user as an input, and the corresponding type of the head action as an output. By means of the present invention, a head action of a user can be accurately identified, and the type of the action is then combined with a function of an intelligent device to realize a new-type interaction mode, and improve the human-machine interaction experience of the user.

Description

A kind of intelligent device interaction method based on PPG information

technical field

The present invention relates to the technical field of smart device interaction, and more particularly, to a smart device interaction method based on PPG information.

Background technique

With the wider application of wearable devices, different human-computer interaction methods have become a focus of attention in recent years. Due to the limitation of the size of the wearable device and the contact surface, the existing interaction methods such as keyboard input, touch screen, etc. cannot well meet the needs of the user to interact with the device. At the same time, these interaction methods usually require the user's full attention, and require the use of hands and eyes, which greatly occupy the user's attention and cannot smoothly complete the interaction. Therefore, it is urgent to study new interaction methods.

At present, there are some interactive methods, such as voice control, facial expression control, tongue control, etc. However, voice control requires the device to have a microphone, which is inconvenient to use in a quiet environment, and is easily affected by noise in a noisy environment and cannot be accurately recognized. Facial expression-controlled interactive systems usually require cameras or sensors attached to the user's face, the former infringing on the user's privacy and inability to work effectively in dark conditions, and the latter causing inconvenience to the user. Tongue-based interaction requires special sensors attached to the tongue, which is not suitable for daily use. Each of these systems has some problems and cannot adapt to the usage scenarios of daily life.

In addition, there are currently technical solutions that focus on the use of the ear. As an important human receptor, the ear plays a very important role in the interaction with smart devices. Taking earphones as an example, as people wear earphones for longer and longer, the ear has become a very useful part for collecting information. At the same time, compared to sight, hearing takes up much less attention, making the interaction process more fluid. For example, the touch of the mobile phone and the ear is used as an interactive gesture, and the camera of the mobile phone is used to detect the user's intention through image recognition. For another example, attaching a camera and an optical sensor to the earphone conveys different intentions through optical signals changed by different pulling actions on the ear. Another approach creates an ear-hung touchpad that detects where a finger touches the ear to generate position-based control signals. It can be seen that although these methods do not need to use vision, they still need to do some specific gestures to complete the interaction, which cannot meet some scenarios where the hands cannot be used. In addition to this, the extra equipment can be a burden on the ears, causing discomfort to the user.

To sum up, adding a barometer and making an ear-mounted touchpad requires modifying the existing smart device system and adding additional sensors. For some wearable heart rate devices (such as headphones) that already carry an IMU (inertial measurement unit) or control keys, there is no substitute for a PPG (photoplethysmography) sensor for acquiring heart rate data.

SUMMARY OF THE INVENTION

The purpose of the present invention is to overcome the above-mentioned defects of the prior art, and to provide an intelligent device interaction method based on PPG information, which utilizes the PPG signal sensor to effectively detect the head movement by measuring the PPG signal in the user's ear, and provides feedback to the user. The relevant equipment feeds back corresponding control signals to realize the control of the equipment by the head.

The present invention provides an intelligent device interaction method based on PPG information. The method includes the following steps:

Collect the PPG signal of the target user's ear;

Process the collected PPG signal, and extract the waveform characteristic value of the PPG signal including the head movement;

Input the extracted PPG signal waveform characteristic value into the deep learning model to identify the current head movement type of the target user, wherein the deep learning model takes the waveform characteristic value of the target user's PPG signal as input, and uses the corresponding head Action type is output, obtained through training.

In one embodiment, processing the collected PPG signal, and extracting the waveform characteristic value of the PPG signal including the head movement of the target user includes:

For the collected PPG signal, use a band-pass filter to remove noise and extract the envelope of the signal;

For the extracted envelope, the envelope segment containing the head motion is identified and cut out.

In one embodiment, the R-wave peak detection algorithm is used to extract the envelope of the PPG signal, and the endpoint detection method is used to cut out the envelope segment including the head movement.

In one embodiment, the deep learning model includes linear classifiers, support vector machines, naive Bayes, K-nearest neighbors, decision trees, ensemble models, recurrent neural networks, convolutional neural networks.

In one embodiment, the smart device is a smart earphone, and the PPG signal in the ear of the target user is collected by using a PPG sensor built in the smart earphone.

In one embodiment, the head motion types include nodding, raising head, swinging left, swinging right, turning left, and turning right.

In one embodiment, the method of the present invention further comprises: associating the identified head motion type with the function of the smart device, so as to control the smart device to work through different head motion types.

Compared with the prior art, the present invention has the advantage that the PPG sensor is used to replace the function of the IMU or the control key to realize head motion recognition, and the motion recognition is combined with the function of the smart device, thereby realizing interaction with the smart device. As a new type of earphone interaction mode, the present invention can reduce the cost of the smart device and improve the user's human-computer interaction experience.

Other features and advantages of the present invention will become apparent from the following detailed description of exemplary embodiments of the present invention with reference to the accompanying drawings.

Description of drawings

The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments of the invention and together with the description serve to explain the principles of the invention.

1 is a flowchart of a method for interacting with smart devices based on PPG information according to an embodiment of the present invention;

FIG. 2 is a structural diagram of an in-ear smart earphone according to an embodiment of the present invention.

detailed description

Various exemplary embodiments of the present invention will now be described in detail with reference to the accompanying drawings. It should be noted that the relative arrangement of components and steps, the numerical expressions and numerical values set forth in these embodiments do not limit the scope of the invention unless specifically stated otherwise.

The following description of at least one exemplary embodiment is merely illustrative in nature and is in no way intended to limit the invention, its application, or uses.

Techniques, methods, and apparatus known to those of ordinary skill in the relevant art may not be discussed in detail, but where appropriate, such techniques, methods, and apparatus should be considered part of the specification.

In all examples shown and discussed herein, any specific values should be construed as illustrative only and not limiting. Accordingly, other instances of the exemplary embodiment may have different values.

It should be noted that like numerals and letters refer to like items in the following figures, so once an item is defined in one figure, it does not require further discussion in subsequent figures.

The present invention provides an intelligent device interaction method based on PPG information, which can use the measured PPG signal in the user's ear to effectively detect head movements, and feed back corresponding control signals to the device (or smart device), Realize the control of the device by the head. The device of the present invention includes a wearable smart device, and for the sake of clarity, in some cases, an earphone will be used as an example for description.

Specifically, as shown in FIG. 1 , the PPG information-based smart device interaction method provided by the embodiment of the present invention includes the following steps:

In step S110, the PPG signal of the ear is collected.

For example, a photoplethysmography (PPG) detection device is used to collect the PPG signal of the subject's ear, and the PPG detection device can be any earphone or other device that can collect the ear PPG signal.

Referring to Figure 2, taking an in-ear headphone as an example, it includes a headphone circuit (or a headphone chip), a headphone shell, a PPG sensor, a headphone cap, and a headphone cable, wherein the PPG sensor is a built-in type and is used to detect PPG signal.

In step S120, the collected PPG signal is processed to extract the feature value of the envelope segment including the head motion.

Specifically, processing the collected PPG signal includes:

Step S121, use a band-pass filter to remove noise in the PPG signal.

By setting an appropriate band-pass filter, high-frequency components such as power supply and low-frequency components such as capillary density changes and temperature changes can be eliminated.

Step S122, extract the envelope from the denoised PPG signal.

For example, the envelope of the PPG signal is extracted using an R-wave peak detection algorithm. For example, a pattern recognition algorithm is used to identify envelope parts.

Step S123, according to the head movement feature, cut out the envelope segment including the head movement from the envelope.

For example, according to the characteristics of the PPG signal (such as peaks, troughs, AC components, etc.) when the head is moving, the endpoint detection method (such as the CFAR algorithm) is used to cut out envelope segments that may be head movements. A feature that characterizes a single action.

Step S130, input the extracted feature value of the envelope segment into the trained deep learning model, and identify the corresponding head motion type.

Specifically, the extracted envelope segment feature values are put into a machine learning or deep learning model to identify the subject's ongoing head movements. The deep learning model is obtained through training using a training sample set. The training sample set includes multiple samples, and each sample represents the correspondence between the subject's head movement type and the feature value of the envelope segment.

Deep learning models include but are not limited to linear classifiers (such as LR), support vector machines (SVM), naive Bayes (NB), K-nearest neighbors (KNN), decision trees (DT), ensemble models (RF/GDBT), Recurrent Neural Networks and Convolutional Neural Networks (CNN) etc.

The recognized head motion types include, but are not limited to, nodding, looking up, left swing, right swing, left turn, right turn, etc. More head motion types can be identified by expanding the training sample set.

Further, after the head action type is identified, the head action type can be associated with the function of the smart device, so as to realize the human-computer interaction function with the smart device. For example, control the headset to play and pause, cut songs, control the volume, etc. In this way, the control buttons or the IMU can be eliminated, thereby reducing the manufacturing cost of the headset and improving the user's human-computer interaction experience. In actual use, the method provided by the present invention can be executed by a chip of a smart device (eg, a headphone chip).

In order to further verify the effect of the present invention, a preliminary experiment was carried out, and the data of 8 people were used for verification, and the recognition accuracy of bowing, raising, shaking left and right, and turning head reached 96%.

To sum up, the present invention can use the PPG sensor to replace the function of the IMU or the control key, realize the head motion recognition, and combine the motion recognition with the function, thereby realizing a new human-computer interaction mode. The present invention can utilize the existing equipment carrying the PPG sensor, is small and portable, and has high popularity. The user can control the electronic equipment with the head without both hands, avoiding the inconvenience caused by the inability to operate the equipment by hand in special circumstances. .

The present invention may be a system, method and/or computer program product. The computer program product may include a computer-readable storage medium having computer-readable program instructions loaded thereon for causing a processor to implement various aspects of the present invention.

A computer-readable storage medium may be a tangible device that can hold and store instructions for use by the instruction execution device. The computer-readable storage medium may be, for example, but not limited to, an electrical storage device, a magnetic storage device, an optical storage device, an electromagnetic storage device, a semiconductor storage device, or any suitable combination of the foregoing. More specific examples (non-exhaustive list) of computer readable storage media include: portable computer disks, hard disks, random access memory (RAM), read only memory (ROM), erasable programmable read only memory (EPROM) or flash memory), static random access memory (SRAM), portable compact disk read only memory (CD-ROM), digital versatile disk (DVD), memory sticks, floppy disks, mechanically coded devices, such as printers with instructions stored thereon Hole cards or raised structures in grooves, and any suitable combination of the above. Computer-readable storage media, as used herein, are not to be construed as transient signals per se, such as radio waves or other freely propagating electromagnetic waves, electromagnetic waves propagating through waveguides or other transmission media (eg, light pulses through fiber optic cables), or through electrical wires transmitted electrical signals.

The computer readable program instructions described herein may be downloaded to various computing/processing devices from a computer readable storage medium, or to an external computer or external storage device over a network such as the Internet, a local area network, a wide area network, and/or a wireless network. The network may include copper transmission cables, fiber optic transmission, wireless transmission, routers, firewalls, switches, gateway computers, and/or edge servers. A network adapter card or network interface in each computing/processing device receives computer-readable program instructions from a network and forwards the computer-readable program instructions for storage in a computer-readable storage medium in each computing/processing device .

The computer program instructions for carrying out the operations of the present invention may be assembly instructions, instruction set architecture (ISA) instructions, machine instructions, machine dependent instructions, microcode, firmware instructions, state setting data, or instructions in one or more programming languages. Source or object code, written in any combination, including object-oriented programming languages, such as Smalltalk, C++, etc., and conventional procedural programming languages, such as the "C" language or similar programming languages. The computer readable program instructions may execute entirely on the user's computer, partly on the user's computer, as a stand-alone software package, partly on the user's computer and partly on a remote computer, or entirely on the remote computer or server implement. In the case of a remote computer, the remote computer may be connected to the user's computer through any kind of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computer (eg, using an Internet service provider through the Internet connect). In some embodiments, custom electronic circuits, such as programmable logic circuits, field programmable gate arrays (FPGAs), or programmable logic arrays (PLAs), can be personalized by utilizing state information of computer readable program instructions. Computer readable program instructions are executed to implement various aspects of the present invention.

Aspects of the present invention are described herein with reference to flowchart illustrations and/or block diagrams of methods, apparatus (systems) and computer program products according to embodiments of the invention. It will be understood that each block of the flowchart illustrations and/or block diagrams, and combinations of blocks in the flowchart illustrations and/or block diagrams, can be implemented by computer readable program instructions.

These computer readable program instructions may be provided to a processor of a general purpose computer, special purpose computer or other programmable data processing apparatus to produce a machine that causes the instructions when executed by the processor of the computer or other programmable data processing apparatus , resulting in means for implementing the functions/acts specified in one or more blocks of the flowchart and/or block diagrams. These computer readable program instructions can also be stored in a computer readable storage medium, these instructions cause a computer, programmable data processing apparatus and/or other equipment to operate in a specific manner, so that the computer readable medium on which the instructions are stored includes An article of manufacture comprising instructions for implementing various aspects of the functions/acts specified in one or more blocks of the flowchart and/or block diagrams.

Computer readable program instructions can also be loaded onto a computer, other programmable data processing apparatus, or other equipment to cause a series of operational steps to be performed on the computer, other programmable data processing apparatus, or other equipment to produce a computer-implemented process , thereby causing instructions executing on a computer, other programmable data processing apparatus, or other device to implement the functions/acts specified in one or more blocks of the flowcharts and/or block diagrams.

The flowchart and block diagrams in the Figures illustrate the architecture, functionality, and operation of possible implementations of systems, methods and computer program products according to various embodiments of the present invention. In this regard, each block in the flowchart or block diagrams may represent a module, segment, or portion of instructions, which comprises one or more functions for implementing the specified logical function(s) executable instructions. It is also noted that each block of the block diagrams and/or flowchart illustrations, and combinations of blocks in the block diagrams and/or flowchart illustrations, can be implemented in dedicated hardware-based systems that perform the specified functions or actions , or can be implemented in a combination of dedicated hardware and computer instructions. It is well known to those skilled in the art that implementation in hardware, implementation in software, and implementation in a combination of software and hardware are all equivalent.

Various embodiments of the present invention have been described above, and the foregoing descriptions are exemplary, not exhaustive, and not limiting of the disclosed embodiments. Numerous modifications and variations will be apparent to those of ordinary skill in the art without departing from the scope and spirit of the described embodiments. The terminology used herein was chosen to best explain the principles of the embodiments, the practical application or technical improvement in the marketplace, or to enable others of ordinary skill in the art to understand the embodiments disclosed herein. The scope of the invention is defined by the appended claims.

Claims

A smart device interaction method based on PPG information, comprising the following steps:

Collect the PPG signal of the target user's ear;

Process the collected PPG signal, and extract the waveform characteristic value of the PPG signal including the head movement;

Input the extracted PPG signal waveform characteristic value into the deep learning model to identify the current head movement type of the target user, wherein the deep learning model takes the waveform characteristic value of the target user's PPG signal as input, and uses the corresponding head Action type is output, obtained through training.
The method according to claim 1, wherein, processing the collected PPG signal, and extracting the waveform characteristic value of the PPG signal including the head movement of the target user comprises:

For the collected PPG signal, use a band-pass filter to remove noise and extract the envelope of the signal;

For the extracted envelope, the envelope segment containing the head motion is identified and cut out.
The method according to claim 2, wherein the envelope of the PPG signal is extracted using an R-wave peak detection algorithm, and an envelope segment containing head movements is cut out by using an endpoint detection method.
The method of claim 1, wherein the deep learning model comprises linear classifiers, support vector machines, naive Bayes, K-nearest neighbors, decision trees, ensemble models, recurrent neural networks, convolutional neural networks.
The method according to claim 1, wherein the smart device is a smart earphone, and a PPG signal in the ear of the target user is collected by using a PPG sensor built in the smart earphone.
The method according to claim 1, wherein the types of head motions include nodding, looking up, swinging left, swinging right, turning left, and turning right.
The method according to claim 1, further comprising: associating the identified types of head movements with functions of the smart device, so as to control the operation of the smart device through different types of head movements.
A computer-readable storage medium having stored thereon a computer program, wherein the program, when executed by a processor, implements the steps of the method of claim 1 .
An electronic device, comprising a memory and a processor, a computer program that can be run on the processor is stored in the memory, and characterized in that, when the processor executes the program, the method of claim 1 is implemented. step.