CN116430981A - Method for associating equipment with user and electronic equipment - Google Patents

Abstract

The application provides a method for associating equipment with a user and electronic equipment, which are applied to first electronic equipment, wherein the method comprises the following steps: determining a first user; displaying first indication information for indicating a first user to execute a first action; acquiring a first image, and determining first posture information of a first user according to the first image, wherein the first image comprises the first user, and the first posture information is used for representing posture characteristics of the first user; receiving first device posture information from a second electronic device, the second electronic device being communicatively connected to the first electronic device; receiving second equipment posture information from third electronic equipment, wherein the third electronic equipment is in communication connection with the first electronic equipment; if the first device posture information is matched with the first posture information, the second device posture information is not matched with the first posture information, and the first user is determined to be associated with the second electronic device. According to the scheme, matching and association between the user and the electronic equipment can be achieved, and further service is provided for the user through the associated electronic equipment.

Description

Method for associating equipment with user and electronic equipment

Technical Field

The present disclosure relates to the field of electronic devices, and in particular, to a method for associating a device with a user and an electronic device.

Background

Currently, with the development of man-machine interaction technology, interaction modes based on human motion recognition are applied to more and more wide scenes. For example, in the process of moving according to a video course of shoulder and neck relaxation played by the terminal equipment, the terminal equipment can evaluate whether the shoulder and neck gesture of the user is standard or not through human body action recognition, and can display actions in the video course and synchronous actions of the user at the same time, so that the user can be guided to adjust the actions according to the actions in the video course, and the standard degree of the actions of the user is improved.

In the above process, the current method for recognizing the gesture of the user commonly adopted by the terminal device is a deep learning method. In the mode, the terminal equipment can respectively acquire the contour image of the head of the user, the gray level image and the depth image corresponding to the contour image, respectively adopt a deep learning model to carry out gesture recognition on the acquired images, and obtain the final gesture characteristics of the head of the user. In this manner, a depth sensor is required to be mounted in the terminal device to acquire the user image, and thus the device cost increases. Meanwhile, the acquisition of user images and the processing of various types of user images also increase the calculation cost of the terminal device. In addition, the processing process of the terminal equipment in the mode is complex, the calculated amount is large, and real-time processing is difficult to achieve in a scene with high real-time requirements, so that the use experience is poor.

Besides the mode, the terminal equipment can finish the recognition of the user gesture by means of the wearable equipment (such as an intelligent helmet and the like), so that the calculated amount of the terminal equipment side is reduced, and the processing process of the terminal equipment side is simplified. However, in a scenario where multiple wearable devices and multiple users exist, the problem that the terminal device cannot distinguish objects of the user gesture recognition of each wearable device, so that the terminal device cannot timely and accurately determine the gesture of each user is caused, and therefore the scheme can be realized only in a specific scenario, and has low feasibility and scenario adaptability.

Disclosure of Invention

The application provides a method for associating equipment with a user and electronic equipment, which are used for realizing matching and association between the user and the electronic equipment, so that the user can be conveniently and rapidly provided with services through the electronic equipment associated with the user.

In a first aspect, the present application provides a method for associating a device with a user, applied to a first electronic device, the method comprising: determining a first user; displaying first indication information, wherein the first indication information is used for indicating a first user to execute a first action; acquiring a first image, and determining first posture information of a first user according to the first image, wherein the first image comprises the first user, and the first posture information is used for representing posture characteristics of the first user; receiving first device pose information from a second electronic device, wherein the second electronic device is communicatively connected to the first electronic device; receiving second device pose information from a third electronic device, wherein the third electronic device is communicatively connected to the first electronic device; if the first device posture information is matched with the first posture information, the second device posture information is not matched with the first posture information, and the first user is determined to be associated with the second electronic device.

In the method, the first electronic device can determine whether the equipment posture information of the other electronic devices indicates the posture characteristics of the first user or not by matching the posture characteristics of the first user recognized by the first electronic device with the posture characteristics indicated by the equipment posture information from the other electronic devices, so that whether the association relationship between the other electronic devices and the first user exists or not can be determined. Based on the method, the first electronic equipment can quickly realize the matching and association between different electronic equipment and different users, so that the scheme can support the first electronic equipment to recognize the gesture of the user by means of other electronic equipment associated with the user, and is not limited by application scenes.

In one possible design, after determining the first user, the method further comprises: determining an identity of a first user; and, the method further comprises: and if the first equipment gesture information and the second equipment gesture information are not matched with the first gesture information, saving the identity characteristics of the first user.

In the method, when the first electronic device determines that the gesture information of the first device and the gesture information of the second device are not matched with the first gesture information, the second electronic device and the third electronic device can be determined to be not matched with the first user, so that the first electronic device can store the identity characteristics of the first user, and the follow-up process can carry out relevant processing on the first user.

In one possible design, the method further comprises: determining a second user, the second user being different from the first user; the first indication information is also used for indicating a second user to execute a second action, and the second action is different from the first action; determining second gesture information of a second user according to the first image, wherein the first image also comprises the second user, and the second gesture information is used for representing gesture characteristics of the second user; if the first equipment gesture information is matched with the second gesture information, the second equipment gesture information is not matched with the second gesture information, and the association between the second user and the second electronic equipment is determined; if the first equipment gesture information is not matched with the second gesture information, the second equipment gesture information is matched with the second gesture information, and the second user is determined to be associated with the third electronic equipment.

In the method, the first electronic device can determine whether the equipment posture information of the other electronic devices indicates the posture characteristics of the second user or not by matching the posture characteristics of the second user identified by the first electronic device with the posture characteristics indicated by the equipment posture information of the other electronic devices, so that the association relationship between the other electronic devices and the second user can be determined. Therefore, the first electronic device can quickly realize the matching and association between the second electronic device or the third electronic device and the user.

In one possible design, after determining whether the user is associated or not associated with the electronic device, the method further comprises: displaying second indication information, wherein the second indication information is used for indicating to execute a second action; if the first user is associated with the second electronic device, receiving third device posture information from the second electronic device; taking the third equipment posture information as the posture information of the first user; and if the first user is not associated with the second electronic device or the third electronic device, acquiring a second image, and determining the gesture information of the first user according to the second image and the identity characteristics of the first user.

In the method, after the first electronic device determines the association relation between the user and the device, the first electronic device can determine the gesture information of the user by means of the association device for the user with the association device after the user is instructed to execute the second action, so that the calculation amount of the gesture information of the user recognized by the first electronic device is transferred to the association device, the complexity and the calculation amount of the processing of the first electronic device side are reduced, and for the user without the association device, the first electronic device can recognize the gesture information of the user according to the image of the user, and the gesture information of the user can be obtained.

In one possible design, determining pose information of the first user from the second image and the identity feature of the first user includes: determining a third image according to the second image and the identity characteristics of the first user, wherein the third image only comprises the first user; pose information of the first user is determined from the third image.

In the method, after the first electronic device acquires the second image containing the first user, the third image containing only the first user can be acquired from the second image, and the gesture recognition is carried out on the first user according to the third image, so that the interference of other irrelevant information in the second image on the recognition process can be reduced, and the speed and accuracy of carrying out the gesture recognition on the first user are improved.

In one possible design, determining that the second device pose information matches the first pose information includes: determining the similarity between the gesture features characterized by the gesture information of the second equipment and the gesture features characterized by the first gesture information; and if the similarity is greater than or equal to the set similarity threshold, determining that the second equipment gesture information is matched with the first gesture information.

In the method, when the similarity between the features indicated by the two feature information is larger than a certain value, the two feature information is determined to be matched, and the accuracy of the feature information matching can be ensured as much as possible while a certain error is allowed.

In one possible design, the first indication information includes identification information and action information; the identification information is used for identifying the first user and/or the second user, and the action information is used for indicating the first action and/or the second action.

In the method, the electronic equipment can instruct the user to execute the corresponding action based on the identity of the user, so that the corresponding appointed action can be instructed to different users at the same time, and the processing speed and the processing efficiency are improved.

In one possible design, capturing a first image includes: and when the target time arrives, acquiring a first image, wherein the target time is preset time, and the target time is later than the time for displaying the first indication information.

In the method, the electronic equipment collects the first image at a set time after the first indication information is displayed, and a certain time for executing the specified action can be reserved for the user, so that the image collected during the process that the first image executes the specified action for the user is ensured as much as possible, and the matching and the association between the user and the equipment are conducted according to the characteristic that the user executes the specified action.

In one possible design, receiving second device pose information from a second electronic device includes: periodically receiving equipment posture information and time information corresponding to the equipment posture information from the second electronic equipment, and determining the equipment posture information corresponding to the time information closest to the target time distance as first equipment posture information; or, sending request information to the second electronic device, where the request information is used to request the second electronic device to send device posture information obtained at the target time; and receiving the equipment posture information from the second electronic equipment, and taking the equipment posture information from the second electronic equipment as the first equipment posture information.

In the method, the second electronic device can acquire the device gesture information acquired by the second electronic device at the time when the target time is nearest, so that the first electronic device can match gesture features identified by the first electronic device and the second electronic device in similar time, and further accuracy of gesture feature matching is improved.

In one possible design, the first indication information includes time information for indicating the target time.

In the method, the first indication information displayed by the first electronic equipment comprises the time for collecting the image, so that a user can determine the time for collecting the image by the first electronic equipment, and further, the state of executing the specified action is kept when the first electronic equipment collects the image, on one hand, the user experience can be improved, and on the other hand, the first electronic equipment can collect the image which can more accurately reflect the characteristics of the user when executing the specified action, and further, the accuracy of the subsequent processing process is improved.

In one possible design, after taking the third device pose information as the pose information of the first user, the method further comprises: and displaying the third equipment posture information and the identity information of the first user.

In the method, the first electronic equipment displays the finally recognized gesture features and the finally recognized identity information of the user to the user, so that the user can timely check the information of the executing action of the user, and the use experience of the user can be improved.

In a second aspect, the present application provides an electronic device comprising a plurality of display screens, a memory, and one or more processors; wherein the memory is for storing computer program code, the computer program code comprising computer instructions; the computer instructions, when executed by one or more processors, cause an electronic device to perform the method described in the first aspect or any of the possible designs of the first aspect.

In a third aspect, the present application provides a computer readable storage medium storing a computer program which, when run on a computer, causes the computer to perform the method described in the first aspect or any one of the possible designs of the first aspect.

In a fourth aspect, the present application provides a computer program product comprising a computer program or instructions which, when run on a computer, cause the computer to perform the method described in the first aspect or any one of the possible designs of the first aspect.

The advantages of the second aspect to the fourth aspect are described with reference to the first aspect, and the detailed description is not repeated here.

Drawings

FIG. 1 is a schematic diagram of a possible system to which the solution provided in the embodiments of the present application is applicable;

fig. 2 is a schematic hardware architecture of an electronic device according to an embodiment of the present application;

fig. 3 is a schematic software architecture of an electronic device according to an embodiment of the present application;

fig. 4 is a schematic diagram of a method for associating a device with a user according to an embodiment of the present application;

fig. 5 is a schematic diagram of an image of an environment where a terminal device is located, which is provided in an embodiment of the present application;

fig. 6 is a schematic diagram of one possible identification information provided in an embodiment of the present application;

fig. 7 is an interface schematic diagram of displaying indication information by a terminal device according to an embodiment of the present application;

fig. 8 is a flowchart of a method for associating a device with a user and performing gesture detection of the user according to an embodiment of the present application;

fig. 9 is a schematic diagram of a method for associating a device with a user according to an embodiment of the present application;

fig. 10 is a schematic structural diagram of an electronic device according to an embodiment of the present application.

Detailed Description

In order to make the objects, technical solutions and advantages of the embodiments of the present application more apparent, the embodiments of the present application will be described in further detail with reference to the accompanying drawings. Wherein in the description of embodiments of the present application, the terms "first," "second," and the like are used for descriptive purposes only and are not to be construed as indicating or implying a relative importance or implicitly indicating the number of technical features indicated. Thus, a feature defining "a first" or "a second" may explicitly or implicitly include one or more such feature.

For ease of understanding, a description of concepts related to the present application is given by way of example for reference.

1) And the electronic equipment is equipment with a wireless connection function. In some embodiments of the present application, the electronic device is a wearable device having a sensing function and a computing function, or a device having an image acquisition and processing function and a display function.

The electronic device in some embodiments of the present application may be a portable device such as a cell phone, tablet computer, wearable device with wireless communication capability (e.g., watch, bracelet, helmet, headset, etc.), vehicle-mounted terminal device, augmented reality (augmented reality, AR)/Virtual Reality (VR) device, notebook computer, ultra-mobile personal computer (UMPC), netbook, personal digital assistant (personal digital assistant, PDA), smart Home device (e.g., smart television, smart speaker, etc.), smart robot, workshop device, wireless terminal in unmanned (Self Driving), wireless terminal in teleoperation (Remote Medical Surgery), wireless terminal in Smart Grid (Smart Grid), wireless terminal in transportation security (Transportation Safety), wireless terminal in Smart City (Smart City), or wireless terminal in Smart Home (Smart Home), flying device (e.g., smart robot, balloon, drone, aircraft), etc.

Wherein the wearable device is a portable device that the user can wear directly on his or her clothing or accessories. The wearable device in the embodiment of the application is a portable device capable of identifying the gesture features of a user wearing the wearable device.

In some embodiments of the present application, the electronic device may also be a portable terminal device that also contains other functions, such as personal digital assistant and/or music player functions. Exemplary embodiments of portable terminal devices include, but are not limited to, piggy-back

Or other operating system. The above-described portable terminal device may also be other portable terminal devices, such as a Laptop computer (Laptop) or the like having a touch-sensitive surface (e.g., a touch panel). It should also be appreciated that in other embodiments of the present application, the electronic device described above may be a desktop computer having a touch-sensitive surface (e.g., a touch panel) instead of a portable terminal device.

2) An inertial measurement unit (inertial measurement unit, IMU) is a device that measures the three-axis attitude angle (or angular rate) and acceleration of an object. Gyroscopes and accelerometers are the main elements of an inertial measurement unit, the accuracy of which directly affects the accuracy of the inertial measurement unit. The inertial measurement unit generally comprises a plurality of accelerometers and gyroscopes, wherein the accelerometers can detect acceleration signals of the object in a carrier coordinate system, and the gyroscopes can detect angular velocity signals of the carrier relative to the carrier coordinate system. By measuring the acceleration and angular velocity of the object in three-dimensional space using an accelerometer and a gyroscope, the inertial measurement unit can calculate the pose of the object therefrom. In order to increase the reliability, more sensors may also be provided in the inertial measurement unit.

It should be understood that in embodiments of the present application, "at least one" means one or more, and "a plurality" means two or more. "and/or", describes an association relationship of an association object, and indicates that there may be three relationships, for example, a and/or B, and may indicate: a alone, a and B together, and B alone, wherein A, B may be singular or plural. The character "/" generally indicates that the context-dependent object is an "or" relationship. "at least one (item) below" or the like, refers to any combination of these items, including any combination of single item(s) or plural items(s). For example, at least one (one) of a, b or c may represent: a, b, c, a and b, a and c, b and c, or a, b and c, wherein a, b and c can be single or multiple.

The embodiment of the application provides a method for associating equipment with a user and electronic equipment. After matching and association between the user and the electronic equipment are achieved, the gesture of the user can be further identified by the electronic equipment associated with the user.

The method provided by the embodiment of the application can be applied to a system formed by the first electronic equipment and at least one second electronic equipment. The method and the device can be particularly applied to various scenes related to man-machine interaction, such as guiding, monitoring, evaluating, correcting and the like, of the user actions by combining the first electronic device and the second electronic device.

In this embodiment of the present application, the first electronic device is a terminal device having an image acquiring and processing function and a display function, and the second electronic device is a wearable device having a sensing function and a computing function. The first electronic device and the second electronic device may communicate based on wired or wireless communication.

The first electronic device is capable of shooting a user image, performing gesture recognition processing on a user in the user image, and displaying the user image or video. For example, the first electronic device may be a smart television, a smart screen, or the like. The second electronic device may identify a gesture of a user wearing the second electronic device. For example, the second electronic device may be an IMU or a wearable device equipped with an IMU, etc.

For convenience of description, the first electronic device will be referred to as a "terminal device" and the second electronic device will be referred to as a "wearable device" hereinafter.

Fig. 1 is a schematic architecture diagram of one possible system to which the solution provided in the embodiment of the present application is applicable. As shown in fig. 1, the system architecture may include: a terminal device and at least one wearable device.

At least one user exists in a scene or environment where the terminal device and the at least one wearable device are located, and the at least one wearable device is a device worn (or worn) by some or all of the at least one user.

Compared with the scheme that the terminal equipment carries out user gesture recognition based on the image recognition method, the wearable equipment of the user can simply and quickly realize the gesture recognition of the user through devices such as a sensor. Therefore, in the scheme, the user gesture is identified through the wearable device, and the identified user gesture features are sent to the terminal device, so that the terminal device can directly use the user gesture features from the wearable device to perform subsequent processing (such as evaluating the gesture standardization of the user, displaying the gesture of the user and the like), thereby avoiding the problems of complex calculation and poor real-time performance existing when the terminal device performs the user gesture feature identification, and further realizing better use experience effect.

In some embodiments of the present application, the number of at least one user may be greater than or equal to the number of at least one wearable device. That is, in one possible scenario, each of the at least one user is wearing at least one wearable device, and in another possible scenario, there may be a user in the at least one user who is not wearing a wearable device.

For users who do not wear wearable equipment, the terminal equipment can obtain the gesture characteristics of the users in a mode of image recognition after shooting the images of the users, and perform subsequent processing according to the gesture characteristics of the users, so that gesture recognition of more users is realized.

In the system, communication can be performed between the terminal device and the wearable device. Optionally, the terminal device and the wearable device may be connected through bluetooth, or may access the same local area network, or may access different local area networks.

In an example where the terminal device and the wearable device are accessed in the same local area network, the specific steps may be: the terminal device and the wearable device establish wireless connection with the same wireless access point. In addition, the terminal device and the wearable device may access the same wireless fidelity (Wireless Fidelity, wi-Fi) hotspot.

For another example, the terminal device and the wearable device may also access the same bluetooth beacon through the bluetooth protocol. For another example, the terminal device and the wearable device may trigger a communication connection through a near field communication (Near Field Communication, NFC) tag, and transmit encrypted information through the bluetooth module to perform identity authentication. After authentication is successful, data transmission is performed by a Point-to-Point (P2P) mode.

Next, referring to fig. 2, a description is given of a structure of an electronic device to which the method provided in the embodiment of the present application is applicable.

As shown in fig. 2, the electronic device 200 may include a processor 210, an external memory interface 220, an internal memory 221, a usb interface 230, a charge management module 240, a power management module 241, a battery 242, an antenna 1, an antenna 2, a mobile communication module 250, a wireless communication module 260, an audio module 270, a speaker 270A, a receiver 270B, a microphone 270C, an earphone interface 270D, a sensor module 280, keys 290, a motor 291, an indicator 292, a camera 293, a display 294, a SIM card interface 295, and the like.

The sensor module 280 may include, among others, a gyroscope sensor, an acceleration sensor, a proximity light sensor, a fingerprint sensor, a touch sensor, a temperature sensor, a pressure sensor, a distance sensor, a magnetic sensor, an ambient light sensor, a barometric pressure sensor, a bone conduction sensor, etc.

It will be appreciated that the electronic device 200 shown in fig. 2 is merely an example and is not limiting of the electronic device, and that the electronic device may have more or fewer components than shown in the figures, may combine two or more components, or may have different configurations of components. The various components shown in fig. 2 may be implemented in hardware, software, or a combination of hardware and software, including one or more signal processing and/or application specific integrated circuits.

Processor 210 may include one or more processing units such as, for example: the processor 210 may include an application processor (application processor, AP), a modem processor, a graphics processor (graphics processing unit, GPU), an image signal processor (image signal processor, ISP), a controller, a memory, a video codec, a digital signal processor (digital signal processor, DSP), a baseband processor, and/or a Neural network processor (Neural-network Processing Unit, NPU), etc. Wherein the different processing units may be separate devices or may be integrated in one or more processors. The controller may be a neural hub and a command center of the electronic device 200, among others. The controller can generate operation control signals according to the instruction operation codes and the time sequence signals to finish the control of instruction fetching and instruction execution.

A memory may also be provided in the processor 210 for storing instructions and data. In some embodiments, the memory in the processor 210 is a cache memory. The memory may hold instructions or data that the processor 210 has just used or recycled. If the processor 210 needs to reuse the instruction or data, it may be called directly from memory. Repeated accesses are avoided and the latency of the processor 210 is reduced, thereby improving the efficiency of the system.

The execution of the method for associating the device with the user provided in the embodiment of the present application may be performed by the processor 210 controlling or calling other components, for example, calling a processing program of the embodiment of the present application stored in the internal memory 221, or calling a processing program of the embodiment of the present application stored in the third party device through the external memory interface 220, to control the wireless communication module 260 to perform data communication with other devices, so as to improve the intelligence and convenience of the electronic device 200 and improve the user experience. The processor 210 may include different devices, for example, when the CPU and the GPU are integrated, the CPU and the GPU may cooperate to execute the method for associating the device with the user provided in the embodiments of the present application, for example, a part of algorithms in the method for associating the device with the user are executed by the CPU, and another part of algorithms are executed by the GPU, so as to obtain a faster processing efficiency.

The display 294 is used to display images, videos, and the like. The display 294 includes a display panel. The display panel may employ a liquid crystal display (liquid crystal display, LCD), an organic light-emitting diode (OLED), an active-matrix organic light-emitting diode (AMOLED) or an active-matrix organic light-emitting diode (matrix organic light emitting diode), a flexible light-emitting diode (flex), a mini, a Micro led, a Micro-OLED, a quantum dot light-emitting diode (quantum dot light emitting diodes, QLED), or the like. In some embodiments, the electronic device 200 may include 1 or N display screens 294, N being a positive integer greater than 1. The display 294 may be used to display information entered by a user or provided to a user as well as various graphical user interfaces (graphical user interface, GUI). For example, the display 294 may display photographs, videos, web pages, or files.

In the embodiment of the present application, the display 294 may be an integral flexible display, or a tiled display formed of two rigid screens and a flexible screen located between the two rigid screens may be used.

The camera 293 (front camera or rear camera, or one camera may be used as either a front camera or a rear camera) is used to capture still images or video. In general, the camera 293 may include a photosensitive element such as a lens group including a plurality of lenses (convex lenses or concave lenses) for collecting optical signals reflected by an object to be photographed and transmitting the collected optical signals to an image sensor. The image sensor generates an original image of the object to be photographed according to the optical signal.

The internal memory 221 may be used to store computer executable program code that includes instructions. The processor 210 executes various functional applications of the electronic device 200 and data processing by executing instructions stored in the internal memory 221. The internal memory 221 may include a storage program area and a storage data area. The storage program area may store, among other things, code for an operating system, application programs (e.g., functions associated with devices and users, etc.), and the like. The storage data area may store data created during use of the electronic device 200, etc.

The internal memory 221 may also store one or more computer programs corresponding to algorithms of a user for the associated device provided in the embodiments of the present application. The one or more computer programs are stored in the internal memory 221 and configured to be executed by the one or more processors 210, the one or more computer programs including instructions that can be used to perform the various steps in the following embodiments.

In addition, the internal memory 221 may include a high-speed random access memory, and may further include a nonvolatile memory such as at least one magnetic disk storage device, a flash memory device, a universal flash memory (universal flash storage, UFS), and the like.

Of course, the codes of the algorithms for associating the device with the user provided in the embodiments of the present application may also be stored in the external memory. In this case, the processor 210 may run code of an algorithm of the associated device and the user stored in the external memory through the external memory interface 220.

The sensor module 280 may include a gyroscope sensor, an acceleration sensor, a proximity light sensor, a fingerprint sensor, a touch sensor, and the like.

Touch sensors, also known as "touch panels". The touch sensor may be disposed on the display screen 294, and the touch sensor and the display screen 294 form a touch display screen, which is also referred to as a "touch screen". The touch sensor is used to detect a touch operation acting on or near it. The touch sensor may communicate the detected touch operation to the application processor to determine the touch event type. Visual output related to touch operations may be provided through the display 294. In other embodiments, the touch sensor may also be disposed on a surface of the electronic device 200 at a different location than the display 294.

Illustratively, the display 294 of the electronic device 200 displays a main interface that includes icons of a plurality of applications (e.g., camera applications, weChat applications, etc.). The user clicks an icon of the camera application in the main interface by touching the sensor, triggers the processor 210 to start the camera application, and opens the camera 293. The display 294 displays an interface of the camera application, such as a viewfinder interface.

The wireless communication function of the electronic device 200 can be implemented by the antenna 1, the antenna 2, the mobile communication module 250, the wireless communication module 260, a modem processor, a baseband processor, and the like.

The antennas 1 and 2 are used for transmitting and receiving electromagnetic wave signals. Each antenna in the electronic device 200 may be used to cover a single or multiple communication bands. Different antennas may also be multiplexed to improve the utilization of the antennas. For example: the antenna 1 may be multiplexed into a diversity antenna of a wireless local area network. In other embodiments, the antenna may be used in conjunction with a tuning switch.

The mobile communication module 250 may provide a solution for wireless communication including 2G/3G/4G/5G, etc., applied on the electronic device 200. The mobile communication module 250 may include at least one filter, switch, power amplifier, low noise amplifier (low noise amplifier, LNA), etc. The mobile communication module 250 may receive electromagnetic waves from the antenna 1, perform processes such as filtering, amplifying, and the like on the received electromagnetic waves, and transmit the processed electromagnetic waves to the modem processor for demodulation. The mobile communication module 250 can amplify the signal modulated by the modem processor, and convert the signal into electromagnetic waves through the antenna 1 to radiate. In some embodiments, at least some of the functional modules of the mobile communication module 250 may be disposed in the processor 210. In some embodiments, at least some of the functional modules of the mobile communication module 250 may be provided in the same device as at least some of the modules of the processor 210. In the embodiment of the present application, the mobile communication module 250 may also be used for information interaction with other devices.

The modem processor may include a modulator and a demodulator. The modulator is used for modulating the low-frequency baseband signal to be transmitted into a medium-high frequency signal. The demodulator is used for demodulating the received electromagnetic wave signal into a low-frequency baseband signal. The demodulator then transmits the demodulated low frequency baseband signal to the baseband processor for processing. The low frequency baseband signal is processed by the baseband processor and then transferred to the application processor. The application processor outputs sound signals through an audio device (not limited to speaker 270A, receiver 270B, etc.), or displays images or video through display screen 294. In some embodiments, the modem processor may be a stand-alone device. In other embodiments, the modem processor may be provided in the same device as the mobile communication module 250 or other functional module, independent of the processor 210.

The wireless communication module 260 may provide solutions for wireless communication including wireless local area network (wireless local area networks, WLAN) (e.g., wireless fidelity (wireless fidelity, wi-Fi) network), bluetooth (BT), global navigation satellite system (global navigation satellite system, GNSS), frequency modulation (frequency modulation, FM), near field wireless communication technology (near field communication, NFC), infrared technology (IR), etc., as applied on the electronic device 200. The wireless communication module 260 may be one or more devices that integrate at least one communication processing module. The wireless communication module 260 receives electromagnetic waves via the antenna 2, modulates the electromagnetic wave signals, filters the electromagnetic wave signals, and transmits the processed signals to the processor 210. The wireless communication module 260 may also receive a signal to be transmitted from the processor 210, frequency modulate it, amplify it, and convert it to electromagnetic waves for radiation via the antenna 2. In this embodiment, the wireless communication module 260 is configured to establish a connection with another electronic device for data interaction. Or the wireless communication module 260 may be used to access the access point device, send control instructions to other electronic devices, or receive data sent from other electronic devices.

For example, the terminal device and the wearable device as shown in fig. 1 may perform reception or transmission of instructions and data through the mobile communication module 250 or the wireless communication module 260, thereby implementing functions of establishing a communication connection, detecting a gesture of a user, and the like.

In addition, the electronic device 200 may implement audio functions through an audio module 270, a speaker 270A, a receiver 270B, a microphone 270C, an earphone interface 270D, an application processor, and the like. Such as music playing, recording, etc. The electronic device 200 may receive key 290 inputs, generating key signal inputs related to user settings and function control of the electronic device 200. The electronic device 200 may generate a vibration alert (such as an incoming call vibration alert) using the motor 291. The indicator 292 in the electronic device 200 may be an indicator light, which may be used to indicate a state of charge, a change in power, an indication message, a missed call, a notification, etc. The SIM card interface 295 in the electronic device 200 is used to connect to a SIM card. The SIM card may be inserted into the SIM card interface 295 or removed from the SIM card interface 295 to enable contact and separation from the electronic device 200.

It should be understood that in practical applications, the electronic device 200 may include more or fewer components than shown in fig. 2, and embodiments of the present application are not limited. The illustrated electronic device 200 is only one example, and the electronic device 200 may have more or fewer components than shown in the figures, may combine two or more components, or may have a different configuration of components. The various components shown in the figures may be implemented in hardware, software, or a combination of hardware and software, including one or more signal processing and/or application specific integrated circuits.

The software system of the electronic device 200 may employ a layered architecture, an event driven architecture, a microkernel architecture, a microservice architecture, or a cloud architecture. In the embodiment of the application, taking an Android system with a layered architecture as an example, a software structure of an electronic device is illustrated.

The layered architecture divides the software into several layers, each with distinct roles and branches. The layers communicate with each other through a software interface. As shown in fig. 3, the software architecture can be divided into four layers, from top to bottom, an application layer, an application framework layer (FWK), an Zhuoyun row and system libraries, and a Linux kernel layer.

The application layer is the top layer of the operating system, including native applications of the operating system, such as cameras, gallery, calendar, bluetooth, music, video, information, and so forth. An application program referred to in the embodiments of the present application is simply referred to as Application (APP), which is a software program capable of implementing one or more specific functions. Typically, a plurality of applications may be installed in an electronic device. Such as camera applications, mailbox applications, smart home control applications, and the like. The application mentioned below may be a system application installed when the electronic device leaves the factory, or may be a third party application downloaded from a network or acquired from other electronic devices by a user during the process of using the electronic device.

Of course, for a developer, the developer may write an application and install it to that layer. In one possible implementation, the application may be developed using Java language, by calling an application programming interface (Application Programming Interface, API) provided by the application framework layer, through which a developer may interact with the underlying layers of the operating system (e.g., kernel layer, etc.) to develop his own application.

The application framework layer provides an application programming interface (application programming interface, API) and programming framework for application programs of the application layer. The application framework layer may include some predefined functions. The application framework layer may include a window manager, a content provider, a view system, a telephony manager, a resource manager, a notification manager, and the like.

The window manager is used for managing window programs. The window manager can acquire the size of the display screen, judge whether a status bar exists, lock the screen, intercept the screen and the like.

The content provider is used to store and retrieve data and make such data accessible to applications. The data may include information such as files (e.g., documents, video, images, audio), text, etc.

The view system includes visual controls, such as controls that display text, pictures, documents, and the like. The view system may be used to build applications. The interface in the display window may be composed of one or more views. For example, a display interface including a text message notification icon may include a view displaying text and a view displaying a picture.

The telephony manager is for providing communication functions of the electronic device. The notification manager allows the application to display notification information in a status bar, can be used to communicate notification type messages, can automatically disappear after a short dwell, and does not require user interaction.

The android runtime includes a core library and virtual machines. And the android running time is responsible for scheduling and managing an android system.

The core library of the android system comprises two parts: one part is a function which needs to be called by Java language, and the other part is a core library of the android system. The application layer and the application framework layer run in a virtual machine. Taking Java as an example, the virtual machine executes Java files of the application layer and the application framework layer as binary files. The virtual machine is used for executing the functions of object life cycle management, stack management, thread management, security and exception management, garbage collection and the like.

The system library may include a plurality of functional modules. For example: a surface manager, a media library, a three-dimensional graphics processing library (e.g., openGL ES), a two-dimensional graphics engine (e.g., SGL), etc. The surface manager is used to manage the display subsystem and provides a fusion of two-dimensional and three-dimensional layers for multiple applications. Media libraries support a variety of commonly used audio, video format playback and recording, still image files, and the like. The media library may support a variety of audio video encoding formats, such as: MPEG4, h.564, MP3, AAC, AMR, JPG, PNG, etc. The three-dimensional graphic processing library is used for realizing three-dimensional graphic drawing, image rendering, synthesis, layer processing and the like. A two-dimensional graphics engine is a drawing engine that draws two-dimensional drawings.

The Kernel (Kernel) layer provides core system services of the operating system, such as security, memory management, process management, network protocol stacks, driving models, and the like, which are implemented based on the Kernel layer. The kernel layer also acts as an abstraction layer between the hardware and software stacks. This layer has many drivers associated with the electronic device, the main drivers being: a display drive; a keyboard driver as an input device; flash driving based on memory technology equipment; a camera drive; an audio drive; bluetooth driving; wiFi drive, etc.

It should be understood that the functional services described above are only examples, and in practical applications, the electronic device may be divided into more or fewer functional services according to other factors, or the functions of the respective services may be divided in other manners, or the functional services may not be divided, but may operate as a whole.

It should be noted that, in the system shown in fig. 1, both the terminal device and the wearable device may be implemented by the above hardware architecture and software architecture.

The method provided in the present application will be described in detail with reference to specific examples.

Referring to fig. 4, a method for associating a device with a user according to an embodiment of the present application includes:

s401: the terminal equipment establishes communication connection with at least one wearable equipment in the environment where the terminal equipment is located, wherein any one wearable equipment is used for detecting gesture characteristics of a user wearing the wearable equipment.

In the embodiment of the present application, the process of establishing the communication connection between the terminal device and the wearable device may be initiated by the terminal device or may be initiated by the wearable device.

For example, the wearable device initiated process of establishing a communication connection may be: after the user wears and starts the wearable device, the wearable device can search whether connectable devices exist in the environment where the wearable device is located according to control of the user or automatically, and when the terminal device is searched, the wearable device requests to establish communication connection with the terminal device. After receiving the request of the wearable device, the terminal device responds to the request and establishes communication connection with the wearable device. The procedure for setting up a communication connection initiated by the terminal device is similar to the procedure described above and will not be described in detail here.

In this embodiment of the present application, the terminal device and the wearable device may establish communication connection in a wired communication manner or a wireless communication manner.

In some embodiments of the present application, the terminal device may use different communication manners to establish communication connection with different wearable devices. For example, when the at least one wearable device includes a bracelet and Virtual Reality (VR) glasses, the terminal device may establish a communication connection with the bracelet using a bluetooth communication mode and establish a communication connection with the VR glasses using a WiFi communication mode.

In this embodiment of the present application, after a communication connection is established between a terminal device and at least one wearable device, identification information of the wearable device is obtained from each wearable device, so as to distinguish each wearable device. The identification information of each wearable device may be information, such as a device fingerprint, capable of uniquely identifying the wearable device, for example, a unique identification code or serial number of the device, a registered device name, and the like. Wherein a device fingerprint is a device characteristic or unique device identification that can be used to uniquely identify a device.

S402: the terminal equipment displays first indication information, wherein the first indication information is used for indicating a first user in an environment where the terminal equipment is located to execute a first action; wherein, at least one user exists in the environment where the terminal equipment is located.

The first user is any user in at least one user in the environment where the terminal equipment is located.

In some embodiments of the present application, the first indication information is further used to instruct the second user in the environment to perform the second action, that is, the terminal device may simultaneously instruct, through the first indication information, the first user and the second user to perform different actions respectively, and based on this, the terminal device may simultaneously instruct a plurality of users in the environment to perform different actions respectively.

In some embodiments of the present application, the terminal device has a camera, and may capture an image of an environment (or a scene) in which the terminal device is located, and perform processing such as human body target detection and face recognition on the image to identify users existing in the image, so as to determine the number of users existing in the environment in which the terminal device is located. After the terminal equipment identifies the users in the image, the image of each user can be obtained by cutting, dividing, scaling and the like the image. Wherein, the image of each user only contains the image information of the user, and the image information of other users is not present. After the terminal device obtains the image of each user, the image of each user can be used as the identity information of the corresponding user. Wherein the user's identification information may be used to indicate the identity of the user, uniquely identify the user, etc.

For example, the terminal device shoots an image of the environment where the terminal device is located, as shown in fig. 5, by performing human body target detection and human face recognition, each target object marked by a dotted rectangular frame in the image can be determined, and the human face area of each target object is positioned, so that it can be determined that two users, namely, user 1 and user 2, exist. Further, the image of each user obtained by capturing the image shown in fig. 5 is shown in fig. 6, and the terminal device may store the image of each user shown in fig. 6 as the identification information of the corresponding user, so as to confirm the corresponding user in the environment based on the stored identification information in a subsequent process.

After the terminal device identifies the users existing in the environment according to the shot images, an action can be allocated to each user, and the action allocated to each user is indicated to the user, so that the user can execute the action according to the indication. Specifically, the terminal device may instruct the users in the environment to perform the corresponding actions respectively at the same time, or the terminal device may instruct each user to perform the corresponding actions individually according to the order of the users in the environment. The sequence of the users in the environment where the terminal device is located may be obtained after the terminal device sequences the users, or may be obtained by the terminal device according to the setting or the instruction of the users. The terminal device may adopt a random ordering mode or a mode of ordering according to the user position when ordering the users.

If at least one user exists in the environment where the terminal equipment is located, the terminal equipment allocates at least one action for the at least one user. Wherein the number of at least one action is the same as the number of at least one user. When the actions are allocated, the terminal device may allocate different actions to each user, instruct the user to do the allocated actions at the same time, or allocate at least one action to the corresponding user in sequence according to the sequence set for at least one user.

Optionally, when there is a user of the users who is not wearing the wearable device, the number of at least one action or the number of at least one user is greater than or equal to the number of at least one wearable device.

The above-mentioned actions may be selected randomly by the terminal device in a plurality of preset actions, or may be actions that the terminal device randomly generates and can be completed by a human body. When the terminal device simultaneously instructs at least one user to perform the corresponding action, the at least one action is different from each other. When the terminal devices individually instruct at least one user to perform the corresponding actions, the at least one action may be different from each other, or the same action may be included in the at least one action.

In this embodiment of the present application, when at least one user is instructed to perform a corresponding action, the terminal device may use a display screen to display the instruction information and/or use a voice broadcast mode to broadcast the instruction information. For example, in the process that the terminal device instructs the user to perform the corresponding action, for each user, the terminal device may display the instruction information on a display screen of the terminal device, or use a player to play the instruction information.

The indication information comprises identification information and action information, wherein the identification information is used for indicating a user needing to execute the specified action, and the action information is used for indicating information of the specified action which the user needs to execute. The user can determine and execute the corresponding action after seeing the self identity information and the corresponding action information displayed by the terminal equipment.

In some embodiments of the present application, the identification information of the user may be a name of the user, a network account number registered by the user in the terminal device, an image of the user captured by the terminal device, user characteristic information identified by the terminal device (for example, body type clothing characteristics of the user identified by the terminal device, face characteristics of the user obtained by face recognition of the terminal device, and the like), a location where the user is located, and the like, which can uniquely distinguish the information of the user.

Alternatively, the identification information of the user may be pre-entered into the terminal device by the user. For example, before the gesture detection of the user is performed, the terminal may instruct the user to input face information, identify the face feature of the user by face recognition, establish an association between the face feature and a personal identifier (such as a name of the user) as identity information, which is input by the user, and store the association.

When the indication information is displayed, the terminal equipment can directly take the shot image of the user as the identity information of the user, can also recognize the face characteristics of the user from the shot image of the user in a face recognition mode, and takes the personal identification associated with the face characteristics as the identity information of the user.

In some embodiments of the present application, when the terminal device indicates the indication information to the user in a display manner, the identification information or the action information may be at least one of the following formats:

1) Text format

In one possible implementation, the terminal device may perform the presentation format of the identification information and the action information in a text format. In the mode, the terminal equipment can summarize and indicate the identity information or the action information through the text description, and display the corresponding text.

For example, when the first action assigned to the first user by the terminal device is "head-to-right" the terminal device may display the following text contents as indication information on the display screen: please user XXX perform the "head to right" action. The text content "XXX (XXX may be the name of the first user)" is identity information of the first user, and the text content "the head of which bends right" is action information of the first action that the first user needs to execute.

2) Image or video formats

In another possible implementation, the terminal device may perform the presentation format of the identification information and the action information in an image or video format. In this way, the terminal device may use a pre-shot or real-time shot user image as the user identification information, and may use a pre-recorded or generated image or video of the human body execution action as the action information.

Optionally, the terminal device may perform some optimization processing on the image and then display the image. For example, the obtained original image may be subjected to post-processing such as cutting, adding a filter, and cartoon processing, and then displayed.

For example, when the first action assigned to the first user by the terminal device is "head-to-right" the terminal device may display an image of the first user and an image of the first action (i.e., an image in which the action of "head-to-right" is performed) on the display screen.

Of course, the terminal device may also display the identification information and the action information by adopting a mode of combining the two formats.

For example, when the terminal device is "head-to-right" at the first action assigned to the first user, the terminal device may display a text prompt prompting the first user to perform the action of head-to-right on the display screen, and simultaneously display an image of the first action. Wherein, the text content can be: please user XXX perform the "head to right" action as illustrated below.

For another example, the terminal device may display the image of the first user and the image of the first action simultaneously on the display screen, and display text content for prompting. The text content may be: the following actions are performed by the following user.

When the terminal device indicates the indication information to the user in a voice manner, the indication information in the text format described in the above format 1) may be generated first, and then the indication information in the text format may be played by the player.

For a first user of the at least one user in which the environment of the terminal device exists, the terminal device may output the indication information for the first user in the above manner, so as to instruct the first user to perform the first action allocated to the first user.

In some embodiments of the present application, besides the above manner of individually indicating the indication information corresponding to the first user, the terminal device may also use a manner of displaying the indication information corresponding to the first user and the indication information corresponding to the other users at the same time, that is, the terminal device may instruct multiple users in the environment to execute the specified actions respectively at the same time.

For example, when the terminal device is a smart large screen, for a plurality of users shown in fig. 5, the terminal device may display indication information in the interface shown in fig. 7 to indicate the plurality of users to perform specified actions, respectively.

S403: the terminal equipment collects a first image and identifies first gesture information of a first user from the first image, wherein the first gesture information is used for indicating gesture characteristics of the first user.

The first image may be at least one image shot by the terminal device, or the first image may be at least one frame image in a video shot by the terminal device.

In some embodiments of the present application, after displaying the indication information in the above manner to indicate the user in the environment to perform the corresponding specified action, the terminal device collects an image or video of the user performing the corresponding specified action, and identifies the gesture feature of the corresponding user from the image included in the collected image or video.

When the terminal equipment collects images, the terminal equipment can shoot the images of the environment where the terminal equipment is located, and the images of all users are respectively obtained by cutting the images.

As an optional implementation manner, after displaying the indication information, the terminal device may wait for a set time and then collect the image of the user, so as to reserve time required for the user to view the indication information and execute the action, and further collect the image of the user including the execution of the specified action more accurately.

As another alternative embodiment, when the indication information is displayed, the terminal device may simultaneously display information for indicating the time when the user performs the specified action, so that the user performs the specified action at the specified time, and further the terminal device may collect the image of the user performing the specified action at the specified time, thereby accurately collecting the image including the user performing the specified action.

For example, the terminal device may display real-time countdown information on the basis of the interface shown in fig. 7, so as to prompt the user to capture an image after the countdown is finished, so that the user can confirm the time for executing the specified action, and thus the specified action can be maintained to be executed when the countdown is finished, so that the terminal device is convenient to collect the image.

In some embodiments of the present application, after the terminal device collects the image of the user performing the specified action, the gesture feature recognition may be performed on the user by using an image-based gesture recognition algorithm or a gesture recognition model. The gesture recognition model may be a model obtained by training a network model using a gesture recognition algorithm. The posture recognition algorithm is classified into a head posture recognition algorithm for performing head posture estimation and a body posture recognition algorithm for performing body posture estimation.

Illustratively, the head pose recognition algorithm may employ Hopenet, or the like. The body posture recognition algorithm may employ Openpose, HRnet (high resolution net) or the like. The training data and test data employed in training the network model employing the head pose recognition algorithm may be images from the dataset 300W-LP. The training data and test data employed in training the network model employing the body position recognition algorithm may be images from data sets such as MPII body position data sets (MPII human pose dataset), COCO (common objects in context), crowdPose, poseTrack, etc.

Optionally, the terminal device may obtain an image of the user executing the specified action by shooting with a built-in camera or an external camera.

In some embodiments of the present application, when the terminal device collects images of the respective users that perform the corresponding actions, the time of collecting the images of the respective users may also be recorded separately.

For example, for a first user existing in an environment where the terminal device is located, after the terminal device instructs the first user to perform a first action allocated to the first user, during the period when the first user performs the first action, the terminal device may obtain, through capturing by a camera, a first image when the first user performs the first action. And then, carrying out gesture recognition on the first image to obtain first gesture information, wherein the first gesture information is used for representing the body gesture characteristics of the first user and/or the characteristics of a first action executed by the first user. Further, when the terminal device shoots and obtains the first image, the terminal device records the time of shooting and obtaining the first image at the same time.

S404: the terminal equipment receives second gesture information from at least one wearable equipment respectively, and determines the situation that the first user wears the wearable equipment according to the first gesture information and the second gesture information of the at least one wearable equipment, wherein the second gesture information of any one wearable equipment is used for representing gesture characteristics of the user wearing the wearable equipment.

Wherein the case that the first user wears the wearable device includes whether the first user wears the wearable device and which of the at least one wearable device the first user wears when the first user wears the wearable device.

In some embodiments of the present application, a wearable device may identify gesture features of a user wearing the wearable device. After the wearable device and the terminal device are in communication connection, the gesture features of the user can be recognized in real time, and corresponding device gesture information can be reported to the terminal device in real time. The terminal equipment can receive equipment posture information from each wearable equipment in real time and store the equipment posture information. The device gesture information reported by the wearable device is used for indicating gesture information of a user wearing the wearable device.

In some embodiments of the present application, for each wearable device that establishes communication connection with a terminal device, the terminal device may also periodically receive and store device posture information reported by the wearable device and time information corresponding to the device posture information, where the time information corresponding to the device posture information is used to indicate a time when the wearable device acquires the device posture information. The terminal device may also send, after the first image of the first user is collected, request information for reporting device posture information to the wearable device, where the request information carries time for the terminal device to collect the first image and is used to request the wearable device to send the device posture information collected at the time to the terminal device.

For each user, after acquiring an image of the user, identifying gesture features in the image, determining gesture features from each wearable device according to device gesture information from each wearable device, comparing the identified gesture features with gesture features of each wearable device, when the similarity between the gesture features from a certain wearable device and the gesture features of a user identified by the terminal device through the image is greater than a set similarity threshold, the terminal device can determine that the user wears the wearable device, can establish an association relationship between the user and the wearable device, and takes the gesture features detected by the wearable device as gesture features of the user; otherwise, if the similarity between the gesture features of a user identified by the image and the gesture features detected by all the wearable devices connected with the terminal device is smaller than the set similarity threshold, the terminal device may determine that the user does not wear the wearable device.

For example, for a first user existing in the environment where the terminal device is located, the terminal device may intercept a first image including the first user from the captured image according to the method of the above step, and identify a first gesture feature of the first user from the first image, and then determine a situation that the first user wears the wearable device according to the following steps 1 to 4.

Step 1: after the terminal equipment identifies the first gesture feature of the first user, judging whether wearable equipment which is not associated with other users exists in at least one wearable equipment which is in communication connection with the terminal equipment; if yes, executing the step 2, otherwise, executing the step 4.

Step 2: the terminal equipment determines whether a first target electronic device exists in the wearable equipment which is not related to other users according to the first gesture characteristics and gesture characteristics reported by the wearable equipment which is not related to other users, wherein the first target electronic device is the wearable equipment of which the similarity between the reported gesture characteristics and the first gesture characteristics is greater than or equal to a set similarity threshold; if yes, executing the step 3, otherwise, executing the step 4.

The specific steps for determining whether the first target electronic device exists are as follows:

the terminal equipment sequentially calculates the similarity between the first gesture feature and at least one wearable equipment which is not related to other users, compares the similarity with a set similarity threshold, and when determining that the similarity between a certain gesture feature and the first gesture feature is higher than the set similarity threshold, considers that the gesture feature and the first gesture feature belong to the gesture feature of the same user, and determines that the wearable equipment reporting the gesture feature is the first target electronic equipment.

As an optional implementation manner, when the terminal device calculates the similarity between the gesture features, the gesture feature of the wearable device used is the gesture feature received by the terminal device at the target time and indicated by the device gesture information from the wearable device, and the target time is the time when the terminal device shoots to obtain the first image.

As another optional implementation manner, when the terminal device calculates the similarity between the gesture features, the gesture features from the wearable device are gesture features that are recognized by the wearable device at a specific time, where the specific time may be a time when the terminal device captures a first image, or a time interval between the specific time and a time when the terminal device captures the first image is less than or equal to a set duration threshold.

In the mode, when the wearable device reports the gesture information, the time information corresponding to the gesture information can be reported at the same time, and the time information is used for indicating the time when the wearable device recognizes the gesture feature indicated by the gesture information, so that the terminal device can more accurately determine the time when the wearable device recognizes the gesture feature.

Step 3: the terminal device determines that the wearable device worn by the first user is the first target electronic device, and associates the first target electronic device with the first user.

After the terminal equipment determines that the first user wears the first target electronic equipment, the association relationship between the first target electronic equipment and the first user can be established and stored. In this way, by associating the first target electronic device with the first user, the terminal device can directly take the gesture feature reported by the first target electronic device as the gesture feature of the first user in the subsequent processing process, so as to evaluate the gesture of the first user according to the feature and perform subsequent processing.

Step 4: the terminal device determines that the first user does not wear the wearable device.

When the terminal device determines that the first user is not wearing the wearable device, the first user may be marked as a user who is not wearing the wearable device. In the subsequent processing process, the terminal equipment can collect the image of the first user, identify the gesture feature of the first user from the image of the first user, and perform subsequent processing such as evaluation according to the gesture feature of the first user identified by the terminal equipment.

Optionally, when the terminal device determines that the first user does not wear the wearable device, the terminal device may automatically save the identity information of the first user (such as a face image of the first user) or prompt the first user to enter the identity information, and save the identity information after establishing an association relationship with the face image of the first user. The face image of the first user can be obtained by cutting out from the first image of the first user. When the subsequent terminal equipment identifies the first user according to the shot image, the gesture feature of the first user can be directly identified from the shot image of the first user, and the process of comparing and matching the identified gesture feature of the first user with the gesture feature reported by the wearable equipment is omitted.

In some embodiments of the present application, the terminal device may ensure accuracy in determining a situation where the first user wears the wearable device by repeating the method provided in the above embodiments a plurality of times. And finally determining that the first user does not wear the wearable device by the terminal device under the condition that the wearable device is not worn by the first user after multiple times of judgment.

Specifically, in step 4 above, after determining that the first user does not wear the wearable device, the terminal device may instruct the first user to perform the first action again, capture an image of the first user, identify the gesture feature of the first user according to the image of the first user, and perform steps 1 to 4 above again to determine the situation that the first user wears the wearable device. And when the number of times that the terminal equipment repeatedly executes the process is larger than or equal to the set threshold value, the first user still determines that the wearable equipment is not worn by the first user, and finally the first user is determined to not wear the wearable equipment.

After the terminal device determines that the first user wears the wearable device according to the above manner, the terminal device may continuously confirm the situation that the other users wear the wearable device according to the method provided in the above embodiment, which is not described herein.

S405: after the terminal device determines that all users wear the wearable device, a user interface may be displayed, where the user interface may instruct the user to perform a movement. When the wearable device is worn by the first user, the terminal device determines the gesture of the first user according to gesture features indicated by the device gesture information of the wearable device worn by the first user; when the first user is determined not to wear the wearable device, the terminal device collects a second image containing the first user, and the gesture of the first user is identified from the second image.

After the terminal device determines that the first user wears the wearable device, a mode of identifying the gesture of the first user in a subsequent process can be determined according to the condition that the first user wears the wearable device. Specifically, if the first user wears the wearable device, the terminal device may acquire, from the wearable device, gesture information of the first user identified by the wearable device, and determine a gesture of the first user according to the gesture information. If the first user does not wear the wearable device, the terminal device can shoot an image of the environment where the first user is located, acquire the image of the first user from the image, and recognize the gesture of the first user according to the image.

In some embodiments of the present application, after determining the gesture of the first user, the terminal device may display the gesture information of the first user and the identity information of the first user to the first user, so that the first user may timely view the information of the executing action of the first user, thereby improving the user experience. For a plurality of users in the environment, the terminal device can also display the gesture information and the identity information of each user simultaneously after determining the gesture information of each user respectively.

In the above embodiment, for a user wearing the wearable device, the terminal device may adaptively complete matching and association between the user and the wearable device worn by the user, and the associated terminal device may use the gesture features of the user identified by the wearable device. For users who do not wear wearable devices, the terminal device may identify gesture features of the user. Based on the method, the user gesture features are identified by combining the wearable equipment and the terminal equipment, on one hand, the calculation amount of identifying the user gesture features at the terminal equipment side is reduced by using the user gesture features identified by the wearable equipment, and meanwhile, the real-time performance of the user gesture identification is improved by utilizing the rapid identification performance of the wearable equipment, and on the other hand, the gesture identification of various users can be guaranteed, so that the use experience is improved. Therefore, the scheme provided by the embodiment has stronger scene adaptability, and can meet the requirements of different practical application scenes.

The method for detecting the gesture of the user and the associated device provided by the embodiment of the application are described in the following with reference to specific examples.

The method provided by the embodiment of the application is described below by taking the wearable device as an IMU device (i.e., the wearable device integrated with the IMU function) as an example in the scenario that the scheme provided by the embodiment of the application is applied to the study of the shoulder and neck relaxing course of the user.

Under the above scenario, the scheme provided by the embodiment of the application adopts a multi-source data fusion mode to perform the processing procedures of recognizing and displaying the gesture features of the user's shoulder, neck and head. The terminal device can play the shoulder and neck relaxing course, and the user can execute corresponding shoulder and neck relaxing actions along with the instructions of the shoulder and neck relaxing course. If the user wears the IMU device, the IMU device can recognize the shoulder and neck and head gestures of the user in the process of the user executing the shoulder and neck relaxing action, and the recognized gesture features of the user are sent to the terminal device. If the user does not wear the IMU equipment, the terminal equipment can shoot the user image and identify the gesture characteristics of the user according to the user image. The terminal equipment can display the synchronous action of the user determined according to the gesture characteristics from the IMU equipment or the synchronous action of the user determined according to the gesture characteristics recognized by the terminal equipment while displaying the standard action in the shoulder and neck relaxing course by utilizing the calculation and display capabilities of the terminal equipment, so that the user can intuitively correct the action of the terminal equipment according to the standard action, and a better learning effect is realized.

The following describes the association device and the user in the above process, and the specific embodiment of user gesture recognition after the association device and the user.

Fig. 8 is a flowchart of a method for associating a device with a user and performing gesture detection of the user according to an embodiment of the present application. As shown in fig. 8, the method flow includes:

s801: the terminal device establishes a communication connection with an IMU device in the environment.

S802: the terminal equipment respectively acquires the identification information of each IMU equipment. In some embodiments, the terminal device may also determine a number N of IMU devices that establish a communication connection with itself, where N is a positive integer.

In this scenario, the IMU device and the terminal device may establish a communication connection through automatic search pairing.

After each IMU device establishes communication connection with the terminal device, the gesture characteristics of the user wearing the IMU device are monitored in real time and reported to the terminal device, and the terminal device receives and stores the information of the gesture characteristics reported by the IMU devices. The IMU device can measure acceleration signals and angular velocity signals of a user wearing the IMU device through the sensor, and can calculate the gesture features of the user such as head gesture angles and the like according to the acceleration signals and the angular velocity signals, so that the head gesture of the user can be determined according to the gesture features.

S803: the terminal equipment shoots an image of the environment, identifies users existing in the image, determines the number n of the users, and respectively intercepts face images of each user. Wherein n is a positive integer.

After the terminal device intercepts the face image of each user, the face image of each user and/or the face features identified according to the face image can be stored as the identity information of the user.

S804: the terminal device randomly allocates n different actions to n users.

S805: the terminal device displays instruction information to instruct the n users to perform actions assigned to the users, respectively.

The indication information comprises the identity information of each user in the n users and corresponding action information, so that each user executes the action indicated by the action information corresponding to the identity information.

For example, in the scenario shown in fig. 5, n=2, the terminal device may display the interface of the instruction information shown in fig. 7, according to the instruction information shown in fig. 7, the user 1 in the environment where the terminal device is located may determine that the user himself performs the action of bending the head to the right side, and the user 2 may determine that the user himself performs the action of bending the head to the left side.

S806: the terminal equipment shoots an image of the environment, and images of each user are respectively intercepted from the image.

S807: and the terminal equipment respectively identifies the gesture characteristics of each user according to the images of the users.

In this scenario, the terminal device is mainly based on the head posture angle information of the user when evaluating the shoulder and neck posture of the user, and therefore, the terminal device can use the head posture angle of the user as the main posture feature of the user.

When the head attitude angle of the user is identified based on the image, the terminal equipment can intercept the face image of the user from the image of the user, then detect the head attitude angle of the face image by utilizing a head attitude identification algorithm or model, and the specific process can be that the two-dimensional key point detection is firstly carried out on the face image, then the detected key point is matched with a preset or fitted three-dimensional face standard model, the conversion relation between the key point of the three-dimensional face model and the detected key point is solved, a rotation matrix is obtained, finally Euler angle parameters for representing the head attitude angle of the user are solved according to the rotation matrix, and thus the attitude characteristics of the user are obtained.

S808: for each user, the terminal equipment judges whether the reported IMU equipment with the gesture characteristics matched with the gesture characteristics of the user exists in the N IMU equipment; if yes, step S809 is performed, otherwise, step S810 is performed until it is determined that each of the n users wears the IMU device, and step S811 is performed.

Wherein, matching means that the similarity between two gesture features is greater than or equal to a set similarity threshold.

S809: the terminal equipment determines that the user wears the IMU equipment, establishes the association relation between the identity information of the user and the identity information of the IMU equipment and stores the association relation.

S810: and the terminal equipment determines that the user does not wear the IMU equipment, only saves the identity information of the user and marks the user as the unworn IMU equipment.

S811: after determining that each user wears the IMU device, the terminal device plays the shoulder and neck relaxing course.

S812: the terminal equipment shoots an image of the environment, and images of each user are respectively intercepted from the image.

S813: and the terminal equipment respectively determines the identity information of each user according to the image of the user.

The terminal equipment can identify the face characteristics of the user and determine the stored identity information matched with the face characteristics as the identity information of the user.

S814: when the identity information has the identification information of the associated IMU equipment, the terminal equipment evaluates and displays the gesture of the user according to the information of the gesture characteristics of the IMU equipment indicated by the identification information; when the identity information does not have the identification information of the associated IMU equipment, the terminal equipment identifies the gesture characteristics of the user according to the image of the user, and evaluates and displays the gesture of the user according to the identified gesture characteristics.

In other embodiments, the terminal device may determine only the user who is not wearing the IMU device from the environmental image and identify the gesture features of the user from the image of the user.

S815: the terminal equipment displays the identity information and the gesture evaluation result of each user respectively.

It should be noted that, the specific implementation process provided by each example is merely an example of a process flow applicable to the embodiment of the present application, where the execution sequence of each step in each example may be adjusted accordingly according to actual needs, and other steps may be added or some steps may be reduced. The method flows provided by the above examples may also be performed in combination with the method flows provided by other examples to implement the methods provided by the embodiments of the present application.

In the above example, the terminal device detects the user gesture by using the gesture feature detected by the IMU device worn by the user or the terminal device itself, so that the calculation amount caused by the detection of the user gesture by the terminal device side based on the user image can be reduced. Meanwhile, for a user wearing the IMU equipment, the terminal equipment can improve the speed of detecting the gesture of the user by means of the faster detection speed of the IMU equipment, so that the real-time performance of detection is improved. For users who do not wear IMU equipment, the terminal equipment can also utilize the self-capability to realize gesture detection of the users.

In addition, in the above example, when a plurality of users and a plurality of IMU devices exist in a scene where the terminal device is located, the terminal device may also determine the specific situation that each user wears the IMU device faster, so as to perform gesture detection in different manners for different users.

Based on the above embodiments and the same concept, the embodiments of the present application further provide a method for associating a device with a user, as shown in fig. 9, the method includes:

s901: the first electronic device determines a first user.

S902: the first electronic device displays first indication information, wherein the first indication information is used for indicating a first user to execute a first action.

S903: the first electronic device acquires a first image and determines first gesture information of a first user according to the first image, wherein the first image comprises the first user, and the first gesture information is used for representing gesture features of the first user.

S904: the first electronic device receives first device pose information from a second electronic device, wherein the second electronic device is communicatively coupled to the first electronic device.

S905: the first electronic device receives second device pose information from a third electronic device, wherein the third electronic device is communicatively connected to the first electronic device.

S906: if the first device pose information is matched with the first pose information, the second device pose information is not matched with the first pose information, and the first electronic device determines that the first user is associated with the second electronic device.

The first electronic device may be a terminal device provided in the foregoing embodiment, and the second electronic device and the third electronic device may be wearable devices provided in the foregoing embodiment.

Specifically, the specific steps executed by the electronic device in the method may refer to the foregoing embodiments, and are not repeated herein.

Based on the above embodiments and the same concept, the embodiments of the present application further provide an electronic device, where the electronic device is configured to implement the method for associating the device with the user provided in the embodiments of the present application. As shown in fig. 10, the electronic device 1000 may include: a display 1001, a memory 1002, one or more processors 1003, and one or more computer programs (not shown). The devices described above may be coupled by one or more communication buses 1004.

The display 1001 is used for displaying related user interfaces such as images, videos, application interfaces, and the like. The memory 1002 has stored therein one or more computer programs (code), including computer instructions; the one or more processors 1003 invoke computer instructions stored in the memory 1002 to cause the electronic device 1000 to perform the methods of associating devices with users provided in embodiments of the present application.

In particular implementations, memory 1002 may include high-speed random access memory, and may also include non-volatile memory, such as one or more magnetic disk storage devices, flash memory devices, or other non-volatile solid-state storage devices. The memory 1002 may store an operating system (hereinafter referred to as a system), such as ANDROID, IOS, WINDOWS, or an embedded operating system such as LINUX. The memory 1002 may be used to store implementation programs for embodiments of the present application. Memory 1002 may also store network communication programs that may be used to communicate with one or more additional devices, one or more user devices, and one or more network devices. The one or more processors 1003 may be a general purpose central processing unit (Central Processing Unit, CPU), microprocessor, application-specific integrated circuit (ASIC), or one or more integrated circuits for controlling the execution of programs in accordance with aspects of the present Application.

It should be noted that fig. 10 is merely an implementation of the electronic device 1000 provided in the embodiments of the present application, and in practical application, the electronic device 1000 may further include more or fewer components, which is not limited herein.

Based on the above embodiments and the same conception, the present application embodiment also provides a computer-readable storage medium storing a computer program, which when run on a computer, causes the computer to perform the method provided by the above embodiments.

Based on the above embodiments and the same conception, the present application embodiment also provides a computer program product comprising a computer program or instructions for causing a computer to perform the method provided by the above embodiments when the computer program or instructions are run on the computer.

The method provided in the embodiments of the present application may be implemented in whole or in part by software, hardware, firmware, or any combination thereof. When implemented in software, may be implemented in whole or in part in the form of a computer program product. The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, the processes or functions in accordance with embodiments of the present invention are produced in whole or in part. The computer may be a general purpose computer, a special purpose computer, a computer network, a network device, a user device, or other programmable apparatus. The computer instructions may be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another computer-readable storage medium, for example, the computer instructions may be transmitted from one website, computer, server, or data center to another website, computer, server, or data center by a wired (e.g., coaxial cable, optical fiber, digital subscriber line (digital subscriber line, abbreviated DSL), or wireless (e.g., infrared, wireless, microwave, etc.) medium, for example, the usable medium may be any available medium that the computer can access or a data storage device such as a server, data center, etc., that contains an integration of one or more usable mediums, for example, a magnetic medium (e.g., floppy disk, hard disk, tape), an optical medium (e.g., digital video disc (digital video disc, abbreviated DVD), or a semiconductor medium (e.g., SSD), etc.

It will be apparent to those skilled in the art that various modifications and variations can be made in the present application without departing from the scope of the application. Thus, if such modifications and variations of the present application fall within the scope of the claims and the equivalents thereof, the present application is intended to cover such modifications and variations.

Claims (12)

1. A method of associating a device with a user, applied to a first electronic device, the method comprising:

determining a first user;

displaying first indication information, wherein the first indication information is used for indicating the first user to execute a first action;

acquiring a first image, and determining first gesture information of the first user according to the first image, wherein the first image comprises the first user, and the first gesture information is used for representing gesture features of the first user;

receiving first device pose information from a second electronic device, wherein the second electronic device is communicatively connected to the first electronic device;

receiving second device pose information from a third electronic device, wherein the third electronic device is communicatively connected to the first electronic device;

and if the first equipment gesture information is matched with the first gesture information, the second equipment gesture information is not matched with the first gesture information, and the first user is determined to be associated with the second electronic equipment.

2. The method of claim 1, wherein after the determining the first user, the method further comprises:

determining an identity of the first user; and

the method further comprises the steps of:

and if the first equipment gesture information and the second equipment gesture information are not matched with the first gesture information, storing the identity characteristic of the first user.

3. The method of claim 1 or 2, wherein the method further comprises:

determining a second user, the second user being different from the first user;

the first indication information is further used for indicating the second user to execute a second action, and the second action is different from the first action;

determining second gesture information of the second user according to the first image, wherein the first image further comprises the second user, and the second gesture information is used for representing gesture features of the second user;

if the first equipment gesture information is matched with the second gesture information, the second equipment gesture information is not matched with the second gesture information, and the second user is determined to be associated with the second electronic equipment;

and if the first equipment gesture information is not matched with the second gesture information, the second equipment gesture information is matched with the second gesture information, and the second user is determined to be associated with the third electronic equipment.

4. A method as claimed in claim 2 or 3, wherein after determining whether the user is associated or not with the electronic device, the method further comprises:

displaying second indication information, wherein the second indication information is used for indicating to execute a second action;

if the first user is associated with the second electronic device, third device gesture information is received from the second electronic device; taking the third equipment gesture information as gesture information of the first user;

and if the first user is not associated with the second electronic device or the third electronic device, acquiring a second image, and determining the gesture information of the first user according to the second image and the identity characteristics of the first user.

5. The method of claim 4, wherein the determining pose information of the first user from the second image and the identity feature of the first user comprises:

determining a third image according to the second image and the identity characteristics of the first user, wherein the third image only comprises the first user;

and determining the gesture information of the first user according to the third image.

6. The method of any of claims 1-5, wherein determining that the second device pose information matches the first pose information comprises:

Determining the similarity between the gesture features characterized by the gesture information of the second equipment and the gesture features characterized by the first gesture information;

and if the similarity is greater than or equal to a set similarity threshold, determining that the second equipment gesture information is matched with the first gesture information.

7. The method of any of claims 1-6, wherein the first indication information includes identification information and action information;

the identification information is used for identifying the first user and/or the second user, and the action information is used for indicating the first action and/or the second action.

8. The method of any of claims 1-7, wherein the acquiring the first image comprises:

and when the target time arrives, acquiring the first image, wherein the target time is preset time, and the target time is later than the time for displaying the first indication information.

9. The method of claim 8, wherein the receiving second device pose information from the second electronic device comprises:

periodically receiving equipment posture information and time information corresponding to the equipment posture information from the second electronic equipment, and determining the equipment posture information corresponding to the time information closest to the target time distance as the first equipment posture information; or alternatively

Transmitting request information to the second electronic device, wherein the request information is used for requesting the second electronic device to transmit device posture information obtained at the target time; and receiving the equipment posture information from the second electronic equipment, and taking the equipment posture information from the second electronic equipment as the first equipment posture information.

10. The method of any of claims 4-9, wherein after taking the third device pose information as the pose information of the first user, the method further comprises:

and displaying the third equipment gesture information and the identity characteristic of the first user.

11. An electronic device comprising a display screen, a memory, and one or more processors;

wherein the memory is for storing computer program code, the computer program code comprising computer instructions; the computer instructions, when executed by the one or more processors, cause the electronic device to perform the method of any of claims 1-10.

12. A computer readable storage medium, characterized in that the computer readable storage medium stores a computer program which, when run on an electronic device, causes the electronic device to perform the method according to any one of claims 1-10.

Images