CN115576417A

CN115576417A - Interaction control method, device and equipment based on image recognition

Info

Publication number: CN115576417A
Application number: CN202211179850.3A
Authority: CN
Inventors: 许康太
Original assignee: Guangzhou Shikun Electronic Technology Co Ltd
Current assignee: Guangzhou Shikun Electronic Technology Co Ltd
Priority date: 2022-09-27
Filing date: 2022-09-27
Publication date: 2023-01-06
Also published as: WO2024067468A1

Abstract

The application provides an interaction control method, device and equipment based on image recognition, and relates to the technology of Internet of things, wherein the method comprises the following steps: and if the electronic equipment is determined to be in the gesture image conversion mode, shooting the indication action of the user through a camera of the electronic equipment to obtain a plurality of frames of indication images. Carrying out identification processing on the multi-frame indication image to obtain a bone point diagram corresponding to the indication image; wherein the bone point diagram comprises indicated bone information corresponding to the indicated action in the indicated image. And sending the skeleton point diagram to a display device so that the display device identifies the skeleton point diagram based on the corresponding relation between the preset skeleton point diagram and the control instruction, determines a target control instruction corresponding to the skeleton point diagram, and executes the target control instruction. According to the method, the display equipment without the camera has the function and the capability of controlling operation through the air gesture, and the problem that the cost of human-computer interaction of the display equipment is high is solved.

Description

Interaction control method, device and equipment based on image recognition

Technical Field

The application relates to the technology of the Internet of things, in particular to an interaction control method, device and equipment based on image recognition.

Background

Currently, in order to make it more convenient for a user to use a display device, high-end display devices are equipped with a camera for human-computer interaction.

In the prior art, an artificial intelligence limb recognition algorithm is deployed in display equipment, and when a user makes a defined limb action within a camera range arranged on the display equipment, the display equipment converts the defined limb action into a control instruction through the limb recognition algorithm and executes the control instruction to complete human-computer interaction.

However, in the prior art, since the display device needs to be provided with a camera hardware module and also needs to deploy and operate a limb recognition algorithm, the requirements on the performance of a processor, a memory and the like of the display device are high, the cost of the display device can be increased, the existing human-computer interaction can only be realized in a high-end display device in a small range, and the popularity of the existing human-computer interaction is low.

Disclosure of Invention

The application provides an interaction control method, device and equipment based on image recognition, which are used for solving the technical problem of high cost of man-machine interaction of display equipment.

In a first aspect, the present application provides an interactive control method based on image recognition, which is applied to an electronic device, where the electronic device is in communication connection with a display device; the method comprises the following steps:

if the electronic equipment is determined to be in the gesture image conversion mode, shooting an indication action of a user through a camera of the electronic equipment to obtain a multi-frame indication image;

carrying out identification processing on a plurality of frames of the indication images to obtain a bone point diagram corresponding to the indication images; wherein the bone point map comprises indicated bone information corresponding to an indicated action in the indicated image;

and sending the bone point diagram to the display equipment so that the display equipment identifies the bone point diagram based on the corresponding relation between a preset bone point diagram and a control instruction, determines a target control instruction corresponding to the bone point diagram and executes the target control instruction.

Further, the identifying process is performed on a plurality of frames of the indication images to obtain a bone point diagram corresponding to the indication images, and the identifying process includes:

carrying out identification processing on a plurality of frames of indication images to obtain an initial bone point diagram corresponding to each frame of indication image;

if the initial bone point diagram corresponding to each frame of indication image is the same, determining the initial bone point diagram as the bone point diagram corresponding to the indication image;

and if the initial bone point diagram corresponding to each frame of the indication image is different, determining that the initial bone point diagram of the first frame of the indication image and the initial bone point diagram of the last frame of the indication image are the bone point diagrams corresponding to the indication images.

Furthermore, the target control instruction corresponding to the bone point diagram is obtained by performing recognition processing on the bone point diagram through a bone point diagram recognition mode which is opened in the display device based on a preset corresponding relation between the bone point diagram and the control instruction.

Further, sending the bone spot map to the display device includes:

and if a plurality of the bone point maps exist, sequentially sending the bone point maps to the display equipment according to the generation time of the bone point maps.

Further, the pointing motion comprises a gesture motion and/or a limb motion.

Furthermore, the connection mode between the electronic device and the display device comprises wired connection and wireless connection, wherein the wired connection is line connection between a charging interface of the electronic device and a universal serial bus interface of the display device; the wireless connection may include a bluetooth communication technology, a local area network protocol, a near field communication, or a wide area network server.

Further, the method further comprises:

and in response to the selection operation for the gesture image conversion mode, disconnecting the communication connection between the electronic equipment and the display equipment.

In a second aspect, the present application provides an interaction control method based on image recognition, which is applied to a display device, where the display device is in communication connection with an electronic device; the method comprises the following steps:

receiving a skeleton point diagram sent by the electronic equipment; the skeleton point diagram is obtained by identifying and processing an indication image, and the indication image is obtained by shooting an indication action of a user through a camera of the electronic equipment when the electronic equipment is in a gesture image conversion mode;

the method comprises the steps of identifying a bone point diagram based on a corresponding relation between a preset bone point diagram and a control instruction, determining a target control instruction corresponding to the bone point diagram, and executing the target control instruction.

Further, the identification processing is carried out on the bone point diagram, and the identification processing comprises the following steps:

and based on the corresponding relation between a preset bone point diagram and the control instruction, carrying out recognition processing on the bone point diagram through a started bone point diagram recognition mode in the display equipment, and determining a target control instruction corresponding to the bone point diagram.

Further, a bone point diagram corresponding to the indication image is determined according to an initial bone point diagram of a first frame indication image and an initial bone point diagram of a last frame indication image, or the bone point diagram corresponding to the indication image is determined according to the initial bone point diagram, wherein the initial bone point diagram is obtained by performing identification processing on each frame of indication image.

Further, the transmission order of the bone point map is determined according to the generation time of the bone point map.

Further, the pointing motion comprises a gesture motion and/or a limb motion.

Furthermore, the connection mode between the electronic equipment and the display equipment comprises wired connection and wireless connection, wherein the wired connection is determined by a charging interface of the electronic equipment and a universal serial bus interface of the display equipment; the wireless connection comprises bluetooth wireless technology, a local area network protocol, near field communication, or a wide area network server.

Further, the communication connection between the electronic device and the display device is disconnected according to the selection operation of the gesture image conversion mode.

In a third aspect, the present application provides an interactive control apparatus based on image recognition, which is applied to an electronic device, where the electronic device is in communication connection with a display device; the device comprises:

the shooting unit is used for shooting the indication action of the user through a camera of the electronic equipment to obtain a plurality of frames of indication images if the electronic equipment is determined to be in the gesture image conversion mode;

the identification unit is used for carrying out identification processing on a plurality of frames of the indication images to obtain a bone point diagram corresponding to the indication images; wherein the bone point map comprises indicated bone information corresponding to an indicated action in the indicated image;

and the sending unit is used for sending the bone point diagram to the display equipment so as to enable the display equipment to identify and process the bone point diagram based on the corresponding relation between a preset bone point diagram and a control instruction, determine a target control instruction corresponding to the bone point diagram and execute the target control instruction.

Further, the identification unit includes:

the identification module is used for identifying a plurality of frames of indication images to obtain an initial bone point diagram corresponding to each frame of indication image;

the first determining module is used for determining that the initial bone point diagram corresponding to each frame of indication image is the same as the initial bone point diagram corresponding to the indication image;

and the second determining module is used for determining that the initial bone point diagram of the first frame of indication image and the initial bone point diagram of the last frame of indication image are the bone point diagrams corresponding to the indication images if the initial bone point diagrams corresponding to the indication images are determined to be different.

Further, the sending unit is specifically configured to:

Further, the pointing motion comprises a gesture motion and/or a limb motion.

Further, the apparatus further comprises:

a disconnection unit configured to disconnect a communication connection between the electronic apparatus and the display apparatus in response to a selection operation for the gesture image conversion mode.

In a fourth aspect, the present application provides an interactive control apparatus based on image recognition, which is applied to a display device, where the display device is in communication connection with an electronic device; the device comprises:

the receiving unit is used for receiving the bone point diagram sent by the electronic equipment; the skeleton point diagram is obtained by identifying and processing an indication image, and the indication image is obtained by shooting an indication action of a user through a camera of the electronic equipment when the electronic equipment is in a gesture image conversion mode;

the determining unit is used for identifying the bone point diagram based on the corresponding relation between a preset bone point diagram and a control instruction, and determining a target control instruction corresponding to the bone point diagram;

and the execution unit is used for executing the target control instruction.

Further, the determining unit is specifically configured to:

and based on the corresponding relation between a preset bone point diagram and the control instruction, carrying out recognition processing on the bone point diagram through an opened bone point diagram recognition mode in the display equipment, and determining a target control instruction corresponding to the bone point diagram.

Further, the pointing motion comprises a gesture motion and/or a limb motion.

Furthermore, the connection mode between the electronic equipment and the display equipment comprises wired connection and wireless connection, wherein the wired connection is determined by a charging interface of the electronic equipment and a universal serial bus interface of the display equipment; the wireless connection includes bluetooth wireless technology, a local area network protocol, near field communication, or a wide area network server.

In a fifth aspect, the present application provides an electronic device, comprising a memory and a processor, wherein the memory stores a computer program operable on the processor, and the processor implements the method of the first aspect when executing the computer program.

In a sixth aspect, the present application provides a display device, which includes a memory and a processor, wherein the memory stores a computer program operable on the processor, and the processor executes the computer program to implement the method of the second aspect.

In a seventh aspect, the present application provides a computer-readable storage medium having stored thereon computer-executable instructions for implementing the method of the first aspect or the method of the second aspect when executed by a processor.

In an eighth aspect, the present application provides a computer program product comprising a computer program that, when executed by a processor, performs the method of the first aspect or the method of the second aspect.

According to the interaction control method, the interaction control device and the interaction control equipment based on the image recognition, if the electronic equipment is determined to be in the gesture image conversion mode, the indication action of a user is shot through a camera of the electronic equipment, and a multi-frame indication image is obtained. Identifying and processing the multiple frames of indication images to obtain a bone point diagram corresponding to the indication images; wherein the bone point map includes indicated bone information corresponding to the indicated action in the indicated image. And sending the bone point diagram to a display device, so that the display device identifies the bone point diagram based on the corresponding relation between the preset bone point diagram and the control instruction, determines a target control instruction corresponding to the bone point diagram, and executes the target control instruction. In the scheme, when the electronic equipment is in a gesture image conversion mode, the camera can be started to work, if a user makes an indication action within the range of the camera, the camera can automatically shoot the indication action to obtain a multi-frame indication image. The electronic equipment carries out recognition processing on a plurality of frames of indication images to obtain a skeleton point diagram corresponding to the indication images, wherein the skeleton point diagram comprises indication skeleton information corresponding to indication actions in the indication images. And finally, sending the bone point diagram to a display device, and when the display device receives the bone point diagram sent by the electronic device, the display device identifies the bone point diagram based on the preset corresponding relation between the bone point diagram and the control command, determines a target control command corresponding to the bone point diagram, and executes the target control command.

Therefore, the bone point diagram is obtained through the camera of the electronic equipment and is sent to the interactive process of the display equipment, the advantages of the shooting capability and the processor computing capability of the camera of the electronic equipment are fully utilized, the image conversion of the indication action is completed and is converted into the bone point diagram which can be identified by the display equipment, the camera cost and the processor and memory performance cost of the display equipment are saved, the resources are greatly saved, the cost of the display equipment is greatly reduced, the display equipment without the camera has the function and capability of controlling operation through the air gesture, and the technical problem of high cost of human-computer interaction of the display equipment is solved.

Drawings

The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the present disclosure and together with the description, serve to explain the principles of the disclosure.

Fig. 1 is a schematic flowchart of an interaction control method based on image recognition according to an embodiment of the present disclosure;

fig. 2 is a schematic view of a scene of an indication action according to an embodiment of the present application;

fig. 3 is a schematic view of a scene of a gesture action according to an embodiment of the present disclosure;

FIG. 4 is a scene schematic diagram of a skeleton point diagram provided in the embodiment of the present application;

fig. 5 is a schematic flowchart of another interaction control method based on image recognition according to an embodiment of the present application;

fig. 6 is a schematic flowchart of another interaction control method based on image recognition according to an embodiment of the present application;

fig. 7 is a schematic flowchart of another interaction control method based on image recognition according to an embodiment of the present application;

fig. 8 is a schematic flowchart of another interaction control method based on image recognition according to an embodiment of the present application;

fig. 9 is a schematic flowchart of another interaction control method based on image recognition according to an embodiment of the present application;

fig. 10 is a schematic structural diagram of an interaction control device based on image recognition according to an embodiment of the present application;

fig. 11 is a schematic structural diagram of another interactive control device based on image recognition according to an embodiment of the present application;

fig. 12 is a schematic structural diagram of another interaction control device based on image recognition according to an embodiment of the present application;

fig. 13 is a schematic structural diagram of an electronic device according to an embodiment of the present application;

fig. 14 is a schematic structural diagram of a display device according to an embodiment of the present application;

fig. 15 is a block diagram of an electronic device according to an embodiment of the present application.

With the foregoing drawings in mind, certain embodiments of the disclosure have been shown and described in more detail below. These drawings and written description are not intended to limit the scope of the disclosed concepts in any way, but rather to illustrate the concepts of the disclosure to those skilled in the art by reference to specific embodiments.

Detailed Description

Reference will now be made in detail to the exemplary embodiments, examples of which are illustrated in the accompanying drawings. When the following description refers to the accompanying drawings, like numbers in different drawings represent the same or similar elements unless otherwise indicated. The implementations described in the exemplary embodiments below are not intended to represent all implementations consistent with the present disclosure.

At present, in order to make a user more convenient to use a display device, high-end display devices are all provided with cameras for human-computer interaction, but a plurality of problems exist in the human-computer interaction process.

In one example, an artificial intelligence limb recognition algorithm is deployed in the display device, when a user makes a defined limb action within a camera range set on the display device, the display device converts the defined limb action into a control instruction through the limb recognition algorithm, and executes the control instruction to complete human-computer interaction. However, in the prior art, since the display device needs to be provided with a camera hardware module and also needs to deploy and operate a limb recognition algorithm, the requirements on the performance of a processor, a memory and the like of the display device are high, the cost of the display device can be increased, the existing human-computer interaction can only be realized in a high-end display device in a small range, and the popularity of the existing human-computer interaction is low.

In one example, if the recognition rate and the control accuracy are good, the image quality, the resolution, the frame rate, and the like of the picture taken by the camera are also required to be high, so that the cost of the camera hardware module is also high.

In one example, if the display device is a self-contained camera module, particularly a display device located in a home environment, privacy and security concerns may be raised by the user.

The application provides an interaction control method, device and equipment based on image recognition, and aims to solve the technical problems in the prior art.

The following describes the technical solutions of the present application and how to solve the above technical problems with specific embodiments. These several specific embodiments may be combined with each other below, and details of the same or similar concepts or processes may not be repeated in some embodiments. Embodiments of the present application will be described below with reference to the accompanying drawings.

Fig. 1 is a schematic flowchart of an interaction control method based on image recognition according to an embodiment of the present disclosure, and is applied to an electronic device, where the electronic device is connected to a display device; as shown in fig. 1, the method includes:

101. if the electronic equipment is determined to be in the gesture image conversion mode, shooting the indication action of the user through a camera of the electronic equipment to obtain a plurality of frames of indication images.

For example, the execution subject of this embodiment may be an electronic device, or a terminal device, or an interactive control device or device based on image recognition, or other devices or devices that may execute this embodiment, which is not limited in this respect. In this embodiment, an execution main body is described as an electronic device.

Firstly, the gesture image conversion mode is used for performing gesture image conversion on the indication image, and the electronic device is in communication connection with the display device, wherein the communication connection can be wired connection or wireless connection. The wired connection can be connected with a Universal Serial Bus (USB) interface of the display device through a charging interface of the electronic device; the wireless connection includes, but is not limited to, establishing a connection through a local area network protocol, bluetooth Communication technology, near Field Communication (NFC), or a wide area network server. When the user starts the gesture image conversion mode, the electronic device is in the gesture image conversion mode, and the gesture image conversion mode requests image data of a camera of the electronic device, the display device also needs to be in the skeleton point diagram recognition mode. The starting and closing of the gesture image conversion mode of the electronic device are achieved according to the selection operation control of the user on the APP, the skeleton point diagram identification mode of the display device can be that the display device is always kept to automatically run in the background after being started, and the gesture image conversion mode and the skeleton point diagram identification mode can be factory pre-deployment or post-deployment without limitation. The indication action is a meaningful action made by the user according to a predefined action instruction library, for example, the indication action includes a gesture action and a limb action, fig. 2 is a scene schematic diagram of the indication action provided in the embodiment of the present application, as shown in fig. 2, the gesture action includes a wake-up gesture, a confirm key, a directional up key \8230, and the like, and the limb action includes a head shaking, a two-hand high lift, a two-hand flat lift, and the like. The indication actions include static actions and dynamic actions, the static actions are various stable and continuous gesture actions or limb actions in fig. 2, and such action characteristic information is generally determined according to any frame data by using single frame data acquisition or multiple frames of data at the same time. Dynamic motion such as hand movement from point a to point B, etc., such motion characteristic information is typically obtained over multiple frames of data.

In the step, if the electronic equipment is determined to be in the gesture image conversion mode, when the user makes an indication action within the shooting range of the camera of the electronic equipment, the indication action of the user is shot through the camera of the electronic equipment, and a plurality of frames of indication images are obtained.

102. Carrying out identification processing on the multi-frame indication image to obtain a bone point diagram corresponding to the indication image; wherein the bone point diagram comprises indicated bone information corresponding to the indicated action in the indicated image.

Fig. 3 is a schematic view of a scene of a gesture action according to an embodiment of the present disclosure, as shown in fig. 3, fig. 3 is a schematic view of an indication image of a palm including extended five fingers, fig. 4 is a schematic view of a skeleton point diagram according to an embodiment of the present disclosure, as shown in fig. 4, and fig. 4 is a schematic view of a skeleton point diagram corresponding to a palm including extended five fingers in fig. 3. If the indication movement is a gesture movement, the electronic equipment identifies the hand which makes the gesture movement in each frame of indication images in the plurality of frames of indication images in a gesture image conversion mode, and converts the hand into a skeleton point diagram containing the spatial distribution of finger bones of the hand, wherein the skeleton point diagram comprises indication skeleton information corresponding to the indication movement in the indication images. Or if the indication action is limb action, the electronic equipment identifies and processes the limb making the limb action in each indication image of a plurality of indication images, and converts the limb into a skeleton point diagram containing the spatial distribution of the phalanges of the limb, wherein the skeleton point diagram comprises indication skeleton information corresponding to the indication action in the indication images.

103. And sending the bone point diagram to a display device, so that the display device identifies the bone point diagram based on the corresponding relation between the preset bone point diagram and the control instruction, determines a target control instruction corresponding to the bone point diagram, and executes the target control instruction.

Illustratively, the electronic device sends a bone point diagram which is converted in real time to the display device in real time through a stable communication connection established between the electronic device and the display device, when the display device receives the bone point diagram, the display device identifies the bone point diagram through an opened bone point diagram identification mode based on a corresponding relation between a preset bone point diagram and a control instruction, a target control instruction corresponding to the bone point diagram is determined through the corresponding relation between the preset bone point diagram and the control instruction, the display device sends the target control instruction to a control system of the display device in real time, and the display device executes the target control instruction through the control system.

In the embodiment of the application, if it is determined that the electronic device is in the gesture image conversion mode, the indication action of the user is shot through a camera of the electronic device, and a plurality of frames of indication images are obtained. Carrying out identification processing on the multi-frame indication image to obtain a bone point diagram corresponding to the indication image; wherein the bone point map includes indicated bone information corresponding to the indicated action in the indicated image. And sending the bone point diagram to a display device, so that the display device identifies the bone point diagram based on the corresponding relation between the preset bone point diagram and the control instruction, determines a target control instruction corresponding to the bone point diagram, and executes the target control instruction. In the scheme, when the electronic equipment is in a gesture image conversion mode, the camera can be started to work, if a user makes an indication action within the range of the camera, the camera can automatically shoot the indication action to obtain a multi-frame indication image. The electronic equipment carries out recognition processing on a plurality of frames of indication images to obtain a skeleton point diagram corresponding to the indication images, wherein the skeleton point diagram comprises indication skeleton information corresponding to indication actions in the indication images. And finally, sending the bone point diagram to a display device, and when the display device receives the bone point diagram sent by the electronic device, the display device identifies the bone point diagram based on the preset corresponding relation between the bone point diagram and the control command, determines a target control command corresponding to the bone point diagram, and executes the target control command. Therefore, the bone point diagram is obtained through the camera of the electronic equipment and is sent to the interactive process of the display equipment, the advantages of the shooting capability and the processor computing capability of the camera of the electronic equipment are fully utilized, the image conversion of the indication action is completed and is converted into the bone point diagram which can be identified by the display equipment, the camera cost and the processor and memory performance cost of the display equipment are saved, the resources are greatly saved, the cost of the display equipment is greatly reduced, the display equipment without the camera has the function and capability of controlling operation through the air gesture, and the technical problem of high cost of human-computer interaction of the display equipment is solved.

Fig. 5 is a schematic flowchart of another interaction control method based on image recognition according to an embodiment of the present application, and is applied to an electronic device and a display device for communication connection; as shown in fig. 5, the method includes:

201. and if the electronic equipment is determined to be in the gesture image conversion mode, shooting the indication action of the user through a camera of the electronic equipment to obtain a plurality of frames of indication images.

In one example, the pointing motion includes a gesture motion and/or a limb motion.

In one example, the connection mode between the electronic device and the display device includes a wired connection and a wireless connection, wherein the wired connection is a line connection between a charging interface of the electronic device and a universal serial bus interface of the display device; the wireless connection may include bluetooth communication technology, a local area network protocol, near field communication, or a wide area network server.

For example, this step may refer to step 101 in fig. 1, and is not described again.

202. And carrying out identification processing on the multiple frames of indication images to obtain an initial bone point diagram corresponding to each frame of indication image.

Illustratively, the electronic equipment carries out recognition processing on a plurality of frames of indication images to obtain an initial bone point diagram corresponding to each frame of indication image.

203. And if the initial bone point diagram corresponding to each frame of the indication image is the same, determining the initial bone point diagram as the bone point diagram corresponding to the indication image.

Illustratively, the electronic device compares the initial skeleton point maps corresponding to the indication images of each frame, and if the initial skeleton point maps corresponding to the indication images of each frame are determined to be the same and indicate that the indication movement is a static movement, the initial skeleton point map is determined to be the skeleton point map corresponding to the indication images.

204. And if the initial bone point diagram corresponding to each frame of the indication image is different, determining that the initial bone point diagram of the first frame of the indication image and the initial bone point diagram of the last frame of the indication image are the bone point diagrams corresponding to the indication images.

Illustratively, the electronic device compares the initial bone point maps corresponding to the indication images of each frame, and if the initial bone point maps corresponding to the indication images of each frame are determined to be different and indicate the indication movement as the dynamic action, the electronic device determines that the initial bone point map of the indication image of the first frame and the initial bone point map of the indication image of the last frame are both the bone point maps corresponding to the indication images.

For example, when the pointing movement moves from point a to point B as a hand, the initial skeleton point map of the first frame pointing image including the starting position point a of the hand and the initial skeleton point map of the last frame pointing image including the ending position point B of the hand are both determined to be the skeleton point maps corresponding to the pointing images.

205. If a plurality of bone point diagrams exist, the bone point diagrams are sequentially sent to the display device according to the generation time of the bone point diagrams, so that the display device identifies the bone point diagrams based on the corresponding relation between the preset bone point diagrams and the control commands, determines the target control commands corresponding to the bone point diagrams, and executes the target control commands.

In one example, the target control instruction corresponding to the bone point diagram is obtained by performing recognition processing on the bone point diagram through a bone point diagram recognition mode which is opened in the display device based on a preset corresponding relation between the bone point diagram and the control instruction.

Illustratively, if a plurality of bone point maps exist, the bone point maps are sequentially transmitted to the display device in real time according to the generation time of the bone point maps, namely, the bone point map generated firstly is transmitted firstly. When the display device receives the bone point diagram, the display device identifies the bone point diagram through a bone point diagram identification mode running in a background based on the corresponding relation between the preset bone point diagram and the control instructions, determines target control instructions corresponding to the bone point diagram through the corresponding relation between the preset bone point diagram and the control instructions, and executes the target control instructions.

206. And in response to the selection operation of the gesture image conversion mode, disconnecting the communication connection between the electronic equipment and the display equipment.

Illustratively, when the user makes a selection operation for the gesture image conversion mode, the communication connection between the electronic device and the display device is disconnected, wherein the selection operation may be a single click, a double click, or the like, which is not limited.

In the embodiment of the application, if it is determined that the electronic device is in the gesture image conversion mode, the indication action of the user is shot through a camera of the electronic device, and a plurality of frames of indication images are obtained. And identifying the multiple frames of indication images to obtain an initial skeleton point diagram corresponding to each frame of indication image. And if the initial bone point diagram corresponding to each frame of the indication image is the same, determining the initial bone point diagram as the bone point diagram corresponding to the indication image. And if the initial bone point diagram corresponding to each frame of the indication image is different, determining that the initial bone point diagram of the first frame of the indication image and the initial bone point diagram of the last frame of the indication image are the bone point diagrams corresponding to the indication images. And if a plurality of bone point diagrams exist, sequentially transmitting the bone point diagrams to the display device according to the generation time of the bone point diagrams, so that the display device identifies the bone point diagrams based on the preset corresponding relation between the bone point diagrams and the control commands, determines target control commands corresponding to the bone point diagrams, and executes the target control commands. And in response to the selection operation for the gesture image conversion mode, disconnecting the communication connection between the electronic equipment and the display equipment. Therefore, the bone point diagram is obtained through the camera of the electronic equipment and is sent to the interactive process of the display equipment, the advantages of the shooting capability and the processor computing capability of the camera of the electronic equipment are fully utilized, the image conversion of the indication action is completed and is converted into the bone point diagram which can be identified by the display equipment, the camera cost and the processor and memory performance cost of the display equipment are saved, the resources are greatly saved, the cost of the display equipment is greatly reduced, the display equipment without the camera has the function and capability of controlling operation through the air gesture, and the technical problem of high cost of human-computer interaction of the display equipment is solved. And when the electronic equipment is matched with the display equipment to carry out the spaced gesture control, the placing position of the electronic equipment can be moved and the angle can be adjusted at any time according to the position of the user, the electronic equipment can be closer to the user, the limb actions and the gesture actions can be shot more clearly, and the recognition accuracy is higher. Because the electronic equipment sends the skeleton point diagram to the display equipment after being identified, but not the image picture of the user, and the connection between the electronic equipment and the display equipment can be disconnected after the electronic equipment is used up, the concerns of privacy and safety are effectively avoided.

Exemplarily, fig. 6 is a schematic flowchart of another interactive control method based on image recognition provided by the embodiment of the present application, and fig. 7 is a schematic flowchart of another interactive control method based on image recognition provided by the embodiment of the present application, where the control instruction database includes a plurality of control instructions and a corresponding relationship between a preset bone point diagram and a control instruction.

Fig. 8 is a schematic flowchart of another interaction control method based on image recognition according to an embodiment of the present application, where the method is applied to a display device, and the display device is in communication with an electronic device; as shown in fig. 8, the method includes:

401. receiving a skeleton point diagram sent by the electronic equipment; the skeleton point diagram is obtained by carrying out recognition processing on an indication image, and the indication image is obtained by shooting an indication action of a user through a camera of the electronic equipment when the electronic equipment is in a gesture image conversion mode.

402. And identifying the bone point diagram based on the corresponding relation between the preset bone point diagram and the control instruction, determining a target control instruction corresponding to the bone point diagram, and executing the target control instruction.

Fig. 9 is a schematic flowchart of another interaction control method based on image recognition according to an embodiment of the present disclosure, which is applied to a display device, where the display device is communicatively connected to an electronic device; as shown in fig. 9, the method includes:

501. receiving a skeleton point diagram sent by the electronic equipment; the skeleton point diagram is obtained by carrying out recognition processing on an indication image, and the indication image is obtained by shooting an indication action of a user through a camera of the electronic equipment when the electronic equipment is in a gesture image conversion mode.

In one example, the bone point map corresponding to the indication image is determined from an initial bone point map of the first frame indication image and an initial bone point map of the last frame indication image, or the bone point map corresponding to the indication image is determined from the initial bone point map, wherein the initial bone point map is obtained by performing recognition processing on each frame indication image.

In one example, the transmission order of the bone point map is determined according to the generation time of the bone point map.

In one example, the pointing motion comprises a gesture motion and/or a limb motion.

In one example, the connection mode between the electronic device and the display device comprises a wired connection and a wireless connection, wherein the wired connection is determined by a charging interface of the electronic device and a universal serial bus interface of the display device; the wireless connection includes bluetooth wireless technology, a local area network protocol, near field communication, or a wide area network server.

In one example, the communication connection between the electronic device and the display device is disconnected according to a selection operation for the gesture image conversion mode.

502. And based on the corresponding relation between a preset skeleton point diagram and the control command, carrying out recognition processing on the skeleton point diagram through an opened skeleton point diagram recognition mode in the display equipment, and determining a target control command corresponding to the skeleton point diagram.

503. And executing the target control instruction.

Fig. 10 is a schematic structural diagram of an interaction control apparatus based on image recognition according to an embodiment of the present application, which is applied to an electronic device, and the electronic device is communicatively connected to a display device; as shown in fig. 10, the apparatus includes:

the shooting unit 61 is configured to, if it is determined that the electronic device is in the gesture image conversion mode, shoot an indication action of a user through a camera of the electronic device, and obtain multiple frames of indication images.

The identification unit 62 is used for carrying out identification processing on the multiple frames of indication images to obtain a bone point diagram corresponding to the indication images; wherein the bone point map includes indicated bone information corresponding to the indicated action in the indicated image.

A sending unit 63, configured to send the bone point diagram to the display device, so that the display device performs recognition processing on the bone point diagram based on a preset correspondence between the bone point diagram and the control instruction, determines a target control instruction corresponding to the bone point diagram, and executes the target control instruction.

The apparatus of this embodiment may execute the technical solution in the method, and the specific implementation process and the technical principle are the same, which are not described herein again.

Fig. 11 is a schematic structural diagram of another interaction control device based on image recognition according to an embodiment of the present application, and based on the embodiment shown in fig. 10, as shown in fig. 11, the recognition unit 62 includes:

the identifying module 621 is configured to perform identification processing on multiple frames of indication images to obtain an initial bone point diagram corresponding to each frame of indication image.

The first determining module 622 is configured to determine the initial bone point map as the bone point map corresponding to the indication image if it is determined that the initial bone point maps corresponding to the indication images of each frame are the same.

A second determining module 623, configured to determine that the initial bone point map of the first frame of the indication image and the initial bone point map of the last frame of the indication image are both bone point maps corresponding to the indication images if it is determined that the initial bone point maps corresponding to each frame of the indication images are different.

In one example, the target control command corresponding to the bone point diagram is obtained by performing recognition processing on the bone point diagram through a bone point diagram recognition mode which is opened in the display device based on a preset corresponding relation between the bone point diagram and the control command.

In an example, the sending unit 63 is specifically configured to:

if a plurality of bone point maps exist, the bone point maps are sequentially transmitted to the display device according to the generation time of the bone point maps.

In one example, the apparatus further comprises:

a disconnection unit 71 configured to disconnect a communication connection between the electronic apparatus and the display apparatus in response to a selection operation for the gesture image conversion mode.

Fig. 12 is a schematic structural diagram of another interaction control apparatus based on image recognition according to an embodiment of the present application, where the interaction control apparatus is applied to a display device, and the display device is communicatively connected to an electronic device; as shown in fig. 12, the apparatus includes:

a receiving unit 81 for receiving a skeleton point diagram transmitted by the electronic device; the skeleton point diagram is obtained by carrying out recognition processing on an indication image, and the indication image is obtained by shooting an indication action of a user through a camera of the electronic equipment when the electronic equipment is in a gesture image conversion mode.

And the determining unit 82 is used for identifying the bone point diagram based on the corresponding relation between the preset bone point diagram and the control instruction, and determining the target control instruction corresponding to the bone point diagram.

And an execution unit 83 for executing the target control instruction.

In an example, on the basis of the embodiment shown in fig. 12, the determining unit 82 is specifically configured to:

and based on the corresponding relation between a preset skeleton point diagram and the control command, carrying out recognition processing on the skeleton point diagram through an opened skeleton point diagram recognition mode in the display equipment, and determining a target control command corresponding to the skeleton point diagram.

Fig. 13 is a schematic structural diagram of an electronic device according to an embodiment of the present application, where the electronic device may be a mobile phone, a plug-in camera module, and as shown in fig. 13, the electronic device includes: a memory 91, a processor 92.

The memory 91 stores therein a computer program that is executable on the processor 92.

The processor 92 is configured to perform the methods provided in the embodiments described above.

The electronic device further comprises a receiver 93 and a transmitter 94. The receiver 93 is used for receiving instructions and data transmitted from an external device, and the transmitter 94 is used for transmitting instructions and data to an external device.

Fig. 14 is a schematic structural diagram of a display device according to an embodiment of the present application, and as shown in fig. 14, the display device includes: memory 101, processor 102.

The memory 101 has stored therein a computer program that is executable on the processor 102.

The processor 102 is configured to perform the methods as provided in the above embodiments.

The display device further comprises a receiver 103 and a transmitter 104. The receiver 103 is used for receiving instructions and data transmitted by an external device, and the transmitter 104 is used for transmitting instructions and data to the external device.

Fig. 15 is a block diagram of an electronic device, which may be a mobile phone, a computer, a digital broadcast terminal, a messaging device, a game console, a tablet device, a medical device, an exercise device, a personal digital assistant, etc., according to an embodiment of the present application.

The apparatus 1100 may include one or more of the following components: processing component 1102, memory 1104, power component 1106, multimedia component 1108, audio component 1110, input/output (I/O) interface(s) 1112, sensor component 1114, and communications component 1116.

The processing component 1102 generally controls the overall operation of the device 1100, such as operations associated with display, telephone calls, data communications, camera operations, and recording operations. The processing components 1102 may include one or more processors 1120 to execute instructions to perform all or a portion of the steps of the methods described above. Further, the processing component 1102 may include one or more modules that facilitate interaction between the processing component 1102 and other components. For example, the processing component 1102 may include a multimedia module to facilitate interaction between the multimedia component 1108 and the processing component 1102.

The memory 1104 is configured to store various types of data to support operations at the apparatus 1100. Examples of such data include instructions for any application or method operating on device 1100, contact data, phonebook data, messages, pictures, videos, and so forth. The memory 1104 may be implemented by any type or combination of volatile or non-volatile memory devices such as Static Random Access Memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic memory, flash memory, magnetic or optical disks.

A power component 1106 provides power to the various components of the device 1100. The power components 1106 may include a power management system, one or more power supplies, and other components associated with generating, managing, and distributing power for the device 1100.

The multimedia component 1108 includes a screen that provides an output interface between the device 1100 and the user. In some embodiments, the screen may include a Liquid Crystal Display (LCD) and a Touch Panel (TP). If the screen includes a touch panel, the screen may be implemented as a touch screen to receive an input signal from a user. The touch panel includes one or more touch sensors to sense touch, slide, and gestures on the touch panel. The touch sensor may not only sense the boundary of a touch or slide action, but also detect the duration and pressure associated with the touch or slide operation. In some embodiments, the multimedia component 1108 includes a front-facing camera and/or a rear-facing camera. The front camera and/or the rear camera may receive external multimedia data when the device 1100 is in an operating mode, such as a shooting mode or a video mode. Each front camera and rear camera may be a fixed optical lens system or have a focal length and optical zoom capability.

The audio component 1110 is configured to output and/or input audio signals. For example, the audio component 1110 includes a Microphone (MIC) configured to receive external audio signals when the apparatus 1100 is in operating modes, such as a call mode, a recording mode, and a voice recognition mode. The received audio signals may further be stored in the memory 1104 or transmitted via the communication component 1116. In some embodiments, audio assembly 1110 further includes a speaker for outputting audio signals.

The I/O interface 1112 provides an interface between the processing component 1102 and peripheral interface modules, which can be keyboards, click wheels, buttons, etc. These buttons may include, but are not limited to: a home button, a volume button, a start button, and a lock button.

The sensor assembly 1114 includes one or more sensors for providing various aspects of state assessment for the apparatus 1100. For example, the sensor assembly 1114 may detect an open/closed state of the apparatus 1100, the relative positioning of components, such as a display and keypad of the apparatus 1100, the sensor assembly 1114 may also detect a change in position of the apparatus 1100 or a component of the apparatus 1100, the presence or absence of user contact with the apparatus 1100, orientation or acceleration/deceleration of the apparatus 1100, and a change in temperature of the apparatus 1100. Sensor assembly 1114 may include a proximity sensor configured to detect the presence of nearby objects in the absence of any physical contact. The sensor assembly 1114 may also include a light sensor, such as a CMOS or CCD image sensor, for use in imaging applications. In some embodiments, the sensor assembly 1114 may also include an acceleration sensor, a gyroscope sensor, a magnetic sensor, a pressure sensor, or a temperature sensor.

The communication component 1116 is configured to facilitate wired or wireless communication between the apparatus 1100 and other devices. The apparatus 1100 may access a wireless network based on a communication standard, such as WiFi,2G or 3G, or a combination thereof. In an exemplary embodiment, the communication component 1116 receives broadcast signals or broadcast related information from an external broadcast management system via a broadcast channel. In an exemplary embodiment, the communication component 1116 also includes a Near Field Communication (NFC) module to facilitate short-range communications. For example, the NFC module may be implemented based on Radio Frequency Identification (RFID) technology, infrared data association (IrDA) technology, ultra Wideband (UWB) technology, bluetooth (BT) technology, and other technologies.

In an exemplary embodiment, the apparatus 1100 may be implemented by one or more Application Specific Integrated Circuits (ASICs), digital Signal Processors (DSPs), digital Signal Processing Devices (DSPDs), programmable Logic Devices (PLDs), field Programmable Gate Arrays (FPGAs), controllers, micro-controllers, microprocessors or other electronic components for performing the above-described methods.

In an exemplary embodiment, a non-transitory computer-readable storage medium comprising instructions, such as the memory 1104 comprising instructions, executable by the processor 1120 of the apparatus 1100 to perform the method described above is also provided. For example, the non-transitory computer readable storage medium may be a ROM, a Random Access Memory (RAM), a CD-ROM, a magnetic tape, a floppy disk, an optical data storage device, and the like.

Embodiments of the present application further provide a non-transitory computer-readable storage medium, where instructions in the storage medium, when executed by a processor of an electronic device, enable the electronic device to perform the method provided by the foregoing embodiments.

An embodiment of the present application further provides a computer program product, where the computer program product includes: a computer program, stored in a readable storage medium, from which at least one processor of the electronic device can read the computer program, and the execution of the computer program by the at least one processor causes the electronic device to perform the solutions provided by any of the above embodiments.

Other embodiments of the disclosure will be apparent to those skilled in the art from consideration of the specification and practice of the disclosure disclosed herein. This application is intended to cover any variations, uses, or adaptations of the disclosure following, in general, the principles of the disclosure and including such departures from the present disclosure as come within known or customary practice in the art to which the disclosure pertains. It is intended that the specification and examples be considered as exemplary only, with a true scope and spirit of the disclosure being indicated by the following claims.

It will be understood that the present disclosure is not limited to the precise arrangements that have been described above and shown in the drawings, and that various modifications and changes may be made without departing from the scope thereof. The scope of the present disclosure is limited only by the appended claims.

Claims

1. An interaction control method based on image recognition is characterized by being applied to electronic equipment, wherein the electronic equipment is in communication connection with display equipment; the method comprises the following steps:

carrying out identification processing on a plurality of frames of the indication images to obtain a bone point diagram corresponding to the indication images; wherein the bone point map comprises indicated bone information corresponding to indicated actions in the indicated image;

2. The method according to claim 1, wherein performing recognition processing on a plurality of frames of the indication images to obtain a bone point diagram corresponding to the indication images comprises:

3. The method according to claim 1, wherein the target control command corresponding to the bone point diagram is obtained by performing recognition processing on the bone point diagram through a bone point diagram recognition mode which is turned on in the display device based on a preset correspondence relationship between the bone point diagram and the control command.

4. The method of claim 1, wherein sending the bone point map to the display device comprises:

5. The method of claim 1, wherein the pointing action comprises a gestural action and/or a limb action.

6. The method according to claim 1, wherein the connection between the electronic device and the display device comprises a wired connection and a wireless connection, wherein the wired connection is a line connection between a charging interface of the electronic device and a universal serial bus interface of the display device; the wireless connection may include a bluetooth communication technology, a local area network protocol, a near field communication, or a wide area network server.

7. The method of any one of claims 1-6, further comprising:

8. The interactive control method based on image recognition is characterized by being applied to display equipment, wherein the display equipment is in communication connection with electronic equipment; the method comprises the following steps:

9. The method according to claim 8, wherein the identifying the bone point diagram based on the correspondence between the preset bone point diagram and the control command, and the determining the target control command corresponding to the bone point diagram comprises:

10. The method according to claim 8, wherein the bone point map corresponding to the indication image is determined from an initial bone point map of a first frame indication image and an initial bone point map of a last frame indication image, or the bone point map corresponding to the indication image is determined from the initial bone point map, wherein the initial bone point map is obtained by performing recognition processing on each frame of the indication image.

11. The method of claim 8, wherein an order of transmission of the bone point map is determined according to a generation time of the bone point map.

12. The method of claim 8, wherein the pointing action comprises a gestural action and/or a limb action.

13. The method according to claim 8, wherein the connection mode between the electronic device and the display device comprises a wired connection and a wireless connection, wherein the wired connection is determined by a charging interface of the electronic device and a universal serial bus interface of the display device; the wireless connection comprises bluetooth wireless technology, a local area network protocol, near field communication, or a wide area network server.

14. The method according to any one of claims 8-13, wherein the communication connection between the electronic device and the display device is disconnected according to a selection operation for the gesture image conversion mode.

15. The interactive control device based on image recognition is characterized by being applied to electronic equipment, wherein the electronic equipment is in communication connection with display equipment; the device comprises:

the sending unit is used for sending the bone point diagram to the display equipment so that the display equipment can identify and process the bone point diagram based on the corresponding relation between a preset bone point diagram and a control instruction, determine a target control instruction corresponding to the bone point diagram and execute the target control instruction.

16. The interactive control device based on image recognition is characterized by being applied to display equipment, wherein the display equipment is in communication connection with electronic equipment; the device comprises:

and the execution unit is used for executing the target control instruction.

17. The apparatus according to claim 16, wherein the determining unit is specifically configured to:

18. The apparatus according to claim 16, wherein the bone point map corresponding to the indication image is determined from an initial bone point map of a first frame indication image and an initial bone point map of a last frame indication image, or wherein the bone point map corresponding to the indication image is determined from the initial bone point map, wherein the initial bone point map is obtained by performing the recognition processing on each frame of the indication image.

19. The apparatus of claim 16, wherein an order of transmission of the bone point map is determined according to a generation time of the bone point map.

20. The apparatus of claim 16, wherein the pointing motion comprises a gesturing motion and/or a limb motion.

21. The apparatus of claim 16, wherein the connection between the electronic device and the display device comprises a wired connection and a wireless connection, wherein the wired connection is determined by a charging interface of the electronic device and a universal serial bus interface of the display device; the wireless connection includes bluetooth wireless technology, a local area network protocol, near field communication, or a wide area network server.

22. The apparatus according to any one of claims 16-21, wherein the communication connection between the electronic device and the display device is disconnected according to a selection operation for the gesture image conversion mode.

23. An electronic device, comprising a memory, a processor, a computer program being stored in the memory and being executable on the processor, the processor implementing the method of any of the preceding claims 1-7 when executing the computer program.

24. A display device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the method of any one of claims 8 to 14 when executing the computer program.

25. A computer-readable storage medium having computer-executable instructions stored therein, which when executed by a processor, are configured to implement the method of any one of claims 1-7 or the method of any one of claims 8-14.

26. A computer program product comprising a computer program which, when executed by a processor, carries out the method of any one of claims 1 to 7 or carries out the method of any one of claims 8 to 14.