WO2022110352A1

WO2022110352A1 - Smart home control method and apparatus, and terminal and storage medium

Info

Publication number: WO2022110352A1
Application number: PCT/CN2020/136692
Authority: WO
Inventors: 朱娟
Original assignee: 捷开通讯(深圳)有限公司
Priority date: 2020-11-30
Filing date: 2020-12-16
Publication date: 2022-06-02
Also published as: CN112286122A

Abstract

A smart home control method and apparatus, and a terminal and a storage medium. The method comprises: when the screen of a terminal is in a screen-off state, if it is detected that the terminal establishes a communication connection with smart home devices, acquiring an air gesture operation of a user (101); determining a corresponding control instruction according to the air gesture operation (102); and controlling the operation state of a corresponding smart home device on the basis of the control instruction (103). By means of the method, operations are performed without the need to turn on a screen, thereby reducing the power consumption and bringing convenience to users.

Description

A smart home control method, device, terminal and storage medium

The present invention requires the priority of the Chinese patent application with the application number 202011372604.0 and the invention title "A smart home control method, device, terminal and storage medium" filed with the China Patent Office on November 30, 2020, the entire contents of which are approved by Reference is incorporated in this application.

technical field

The present invention relates to the field of smart home, in particular to a smart home control method, device, terminal and storage medium.

Background technique

With the continuous development of communication technology, a large number of mobile terminals such as smart phones, tablet computers and notebook computers are popularized and applied. Terminals are developing in the direction of diversification and personalization, and they have increasingly become indispensable terminals in people's life and work. At the same time, the mobile Internet and the Internet of Things are also developing rapidly. There are more and more smart terminals. Users no longer use smart devices such as mobile phones, tablets, laptops, and TVs independently. For example, the mobile phone can be projected to the TV, the user can answer the phone call through the TV, and can also control the tablet or mobile phone on the computer. The interconnection and collaboration of multiple devices brings a lot of convenience to people’s lives. Users can also control devices through command transmission between interconnected devices. For example, to establish interconnection with smart homes at home through mobile phones, users can control the temperature of the air conditioner or switch the air conditioner according to their own needs by controlling the mobile phone, and users can control the mobile phone to turn on the air conditioner according to their own needs. Or turn off the lights in all parts of the house, etc.

During the research and practice of the prior art, the inventors of the embodiments of the present invention found that the operations between these interconnected devices often require users to perform screen interactions. Taking the above scenario as an example, the user needs to turn on the screen of the mobile phone first, and then perform a series of operations on the mobile phone to send commands to the connected air conditioner or smart light for control. Therefore, the operation process is relatively cumbersome, and it usually depends on the graphical operation interface of the device. If it is in the screen-off state, it cannot be operated, which is not convenient enough.

technical problem

Embodiments of the present invention provide a smart home control method, device, terminal, and storage medium, so as to solve the problem that the smart home cannot be controlled in an active screen state.

technical solutions

Embodiments of the present invention provide a smart home control method, device, terminal, and storage medium, which can conveniently control smart home devices from an air while saving power consumption.

An embodiment of the present invention provides a control method for a smart home, which is applied to a terminal, and the method includes:

When the screen of the terminal is in the screen-off state, if it is detected that the terminal establishes a communication connection with the smart home device, acquiring the user's gesture operation in the air;

Determine the corresponding control command according to the air gesture operation;

The operating state of the corresponding smart home device is controlled based on the control instruction.

Correspondingly, an embodiment of the present invention provides a smart home control device, and the device includes:

a detection module, used to detect that the screen of the terminal is in an off-screen state, and used to detect that the terminal establishes a communication connection with the smart home device;

The acquisition module is used to acquire the user's air gesture operation;

a determination module, configured to determine a corresponding control instruction according to the gesture operation in the air;

A processing module for controlling the running state of the corresponding smart home equipment based on the control instruction

Optionally, in some embodiments of the present invention, the determining module is further configured to:

Determine a corresponding target smart home device from the plurality of smart home devices according to the first gesture operation;

The corresponding control instruction is determined according to the second gesture operation.

Optionally, in some embodiments of the present invention, the device further includes:

a judging module for judging whether the face photo matches the sample face photo;

If yes, determine the corresponding target smart home device from the plurality of smart home devices according to the first gesture operation.

Optionally, in some embodiments of the present invention, the obtaining module is further configured to:

is used to obtain the pictures of the first specified number of shots to obtain the first set of pictures;

Optionally, in some embodiments of the present invention, the judgment module is further configured to:

Judging whether there is a picture in the first picture set that matches the first sample gesture picture;

If there is a picture in the first picture set that matches the first sample gesture picture, take the space gesture operation corresponding to the first sample gesture picture as the first gesture operation, and use the photographing element to Obtain the second set of pictures according to the second specified number of pictures obtained within the second preset time period;

for judging whether there is a picture in the second set of pictures that matches the second sample gesture picture;

If there is a picture in the second picture set that matches the second sample gesture picture, the space gesture operation corresponding to the second sample gesture picture is used as the second gesture operation.

an identification module for performing content identification on each picture in the first picture set;

According to the recognition result, determine whether each picture in the first picture set has a hand area;

When it is determined that there is a hand region, it is determined whether there is a picture in the first picture set that matches the first sample gesture picture.

Optionally, in some embodiments of the present invention, the processing module is further configured to:

generating a gesture signal according to the second infrared light;

Determine whether the gesture signal matches the sample gesture signal;

If yes, the gesture operation corresponding to the sample gesture signal is used as a space gesture operation.

Voice recognition processing is performed on the voice information, and a target smart home device corresponding to the voice information is determined from the plurality of smart home devices.

Correspondingly, an embodiment of the present invention further provides a terminal, including a processor and a memory, wherein the memory stores a plurality of instructions, and the processor loads the instructions to execute the steps of the smart home control method as described above.

In addition, an embodiment of the present invention further provides a storage medium, where the storage medium stores a plurality of instructions, and the instructions are adapted to be loaded by a processor to execute any one of the intelligent functions provided in the embodiments of the present invention. Steps in a Home Control Method.

beneficial effect

The beneficial effects of the present application are: when the screen of the terminal is in the screen-off state, if it is detected that the terminal establishes a communication connection with the smart home device, the user's gesture operation in the air can be acquired, and the air gesture control can be performed according to the gesture operation in the air. Smart home equipment, no need to light up the screen to operate, saving power consumption. Secondly, it can overcome the inconvenience that the command transmission between interconnected devices usually needs to be used for operation on a graphical operation interface. Users can directly transmit commands between interconnected devices through air gestures to control the interconnected devices, which brings convenience to users. In addition, users can also define their own preferred gesture actions, with various control methods and great flexibility. It is fast and convenient to control multiple smart home devices, and the operation is simple and easy to learn, which expands the richness of user operations.

Description of drawings

FIG. 1 is an application scenario diagram of a smart home control method provided by an embodiment of the present invention.

FIG. 2 is a first schematic flowchart of a smart home control method provided by an embodiment of the present invention.

FIG. 3 is a second schematic flowchart of a smart home control method provided by an embodiment of the present invention.

FIG. 4 is a schematic diagram of a first gesture operation provided by an embodiment of the present invention.

FIG. 5 is a schematic diagram of a second gesture operation provided by an embodiment of the present invention.

FIG. 6 is a third schematic flowchart of a smart home control method provided by an embodiment of the present invention.

FIG. 7 is a fourth schematic flowchart of a smart home control method provided by an embodiment of the present invention.

FIG. 8 is a fifth schematic flowchart of a smart home control method provided by an embodiment of the present invention. .

FIG. 9 is a sixth schematic flowchart of a smart home control method provided by an embodiment of the present invention.

FIG. 10 is a first schematic flowchart of a smart home control device provided by an embodiment of the present invention.

FIG. 11 is a schematic structural diagram of a terminal provided by an embodiment of the present invention.

Embodiments of the present invention

The technical solutions in the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings in the embodiments of the present invention. Obviously, the described embodiments are only some, but not all, embodiments of the present invention. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative efforts shall fall within the protection scope of the present invention.

In the description of the present invention, it should be understood that the terms "first" and "second" are only used for description purposes, and cannot be interpreted as indicating or implying relative importance or the number of indicated technical features. Thus, features defined as "first", "second" may expressly or implicitly include one or more of said features. In the description of the present invention, " " means two or more, unless otherwise expressly and specifically defined.

The following disclosure provides many different embodiments or examples for implementing different structures of the present invention. In order to simplify the disclosure of the present invention, the components and arrangements of specific examples are described below. Of course, they are only examples and are not intended to limit the invention. Furthermore, the present disclosure may repeat reference numerals and/or reference letters in different instances for the purpose of simplicity and clarity and not in itself indicative of a relationship between the various embodiments and/or arrangements discussed. In addition, the present disclosure provides examples of various specific processes and materials, but one of ordinary skill in the art will recognize the application of other processes and/or the use of other materials. In the above-mentioned embodiments, the description of each embodiment has its own emphasis. For parts that are not described in detail in a certain embodiment, reference may be made to the relevant descriptions of other embodiments.

Embodiments of the present invention provide a smart home control method, device, terminal, and storage medium. The smart home control method can be used in conjunction with a terminal, such as a smart phone, a tablet computer, a notebook computer, or a personal computer. The smart home control method, device, terminal and storage medium will be described in detail below. It should be noted that the description order of the following embodiments is not intended to limit the preferred order of the embodiments.

The present invention will be described in detail below with reference to the accompanying drawings and specific embodiments, please refer to FIG. 1 to FIG. 10 .

Please refer to FIG. 1. FIG. 1 is a schematic diagram of a scenario of a smart home control method provided by an embodiment of the present invention. When it is detected that the terminal has established a communication connection with the smart home device and the screen of the terminal is in an off-screen state, the user's air gesture operation is obtained; the corresponding control instruction is determined according to the air gesture operation, and the control The command is sent to the smart home device; the smart home device receives the control command and responds.

Please refer to FIG. 2 , which is a schematic flowchart of a smart home control method according to Embodiment 1 of the present invention. Wherein, the embodiments of the present invention include:

101. When the screen of the terminal is in an off-screen state, if it is detected that a communication connection is established between the terminal and the smart home device, obtain a user's air gesture operation.

In the embodiment of the present invention, the terminal and the smart home device can establish a communication connection through communication means, and the terminal and the smart home device can use the wireless fidelity WIFI network, the Bluetooth network, the ZigBee network and the Digital Living Network Alliance (Digital Living Network Alliance). Living Network Alliance, DLNA), etc., but not limited to this.

Specifically, when acquiring the user's air gesture operation, the air gesture can be acquired through a sensor device built in the terminal. For example, the user's air gesture operation can be obtained through the built-in camera element or infrared sensor element of the terminal, and the air gesture operation includes but is not limited to swipe up in the air, swipe down in the air, move left in the air, move right in the air , push down from the air, grab from the air, make a fist in the air, compare numbers from the air, gesture V or heart shape and other gestures. In addition, air gestures can also be acquired through the built-in ultrasonic element of the terminal.

102. Determine a corresponding control instruction according to the air gesture operation.

In this embodiment of the present invention, the terminal may determine a corresponding control instruction according to the air gesture operation, and send the determined control instruction to a smart home device, and the number of the smart home devices may be one or more.

It should be noted that this is only an example, and the smart home devices, including but not limited to smart air conditioners, smart refrigerators, smart speakers, smart doors and windows, smart water heaters, smart curtains, smart computers, and smart TVs, can establish communication with the terminal. Connected smart home devices.

103. Control the running state of the corresponding smart home device based on the control instruction.

Specifically, the smart home device receives the control instruction sent by the terminal, the smart home device controls the running state of the smart home device based on the control instruction, and the smart home device receives the control instruction and responds.

For example, when it is determined that the smart home device is a smart air conditioner, the embodiment of the present invention has the following four possible implementation manners:

Optionally, in the first possible implementation manner, when the gesture operation in the air is to make a fist in the air, the corresponding control command is: turn off the current smart home device, that is, turn off the smart air conditioner, and send the control command to The smart home device; the smart home device receives the control instruction and responds to perform the operation of turning off the smart air conditioner.

Optionally, in the second possible implementation manner, when the air gesture operation is to expand from the air, the corresponding control command is: turn on the current smart home device, that is, turn on the smart air conditioner, and send the control command to The smart home device; the smart home device receives the control instruction and responds to execute the operation of turning on the smart air conditioner.

Optionally, in a third possible implementation manner, when the gesture operation in the air is to swipe up in the air, the corresponding control command is: increase the current smart home device, that is, increase the temperature of the smart air conditioner, and set all the The control command is sent to the smart home device; the smart home device receives the control command and responds to perform an operation of increasing the temperature of the smart air conditioner.

Optionally, in a fourth possible implementation manner, when the gesture operation in the air is to swipe down in the air, the corresponding control command is: increase the current smart home device, that is, decrease the temperature of the smart air conditioner, and set all the The control command is sent to the smart home device; the smart home device receives the control command and responds to perform an operation of lowering the temperature of the smart air conditioner.

To sum up, an embodiment of the present invention provides a smart home control method. First, when the screen of the terminal is in an off-screen state, if it is detected that the terminal and the smart home device establish a communication connection, the user's gesture in the air can be obtained. Operation, and control smart home devices from the air according to the gesture operation in the air, without the need to light the screen to operate, saving power consumption. Secondly, it can overcome the inconvenience that the command transmission between interconnected devices usually needs to be used for operation on a graphical operation interface. Users can directly transmit commands between interconnected devices through air gestures to control the interconnected devices, which brings convenience to users. In addition, users can also define their own preferred gesture actions, with various control methods and great flexibility. It is fast and convenient to control multiple smart home devices, and the operation is simple and easy to learn, which expands the richness of user operations.

Please refer to FIG. 3 , which is a schematic flowchart of a smart home control method according to Embodiment 2 of the present invention. Wherein, the embodiments of the present invention include:

201. When the screen of the terminal is in an off-screen state, if it is detected that a communication connection is established between the terminal and the smart home device, acquire a user's air gesture operation.

Specifically, in the embodiment of the present invention, the terminal and the smart home device can establish a communication connection by means of communication, and the terminal and the smart home device can use the wireless fidelity WIFI network, the Bluetooth network, the ZigBee network and the digital life network alliance. (Digital Living Network Alliance, DLNA), etc., but not limited to this.

Specifically, in this embodiment of the present invention, to obtain the user's air gesture operation, the terminal can obtain the user's air gesture operation through the camera element built in the terminal, and the air gesture operation includes a first gesture and a second gesture , the first gesture operation includes but is not limited to digital gestures, the second gesture operation includes but is not limited to swipe up in the air, swipe down in the air, move left in the air, move right in the air, press down in the air, Gestures such as grabbing in the air, making a fist in the air, comparing numbers in the air, gestures such as V-shaped or heart-shaped gestures.

202. Obtaining a first specified number of pictures within a first preset duration by a photographing element, obtaining a first picture set, and judging whether there is a picture in the first picture set that matches the first sample gesture picture; if so, then Go to step 203; otherwise go to step 207.

In this embodiment of the present invention, the number of the first sample gesture pictures may be multiple, and the photographing element is activated, and the photographing element acquires a first set of pictures of a first specified number of shots within a first preset time period, Then, it is judged whether there is a picture in the first picture set that matches the first sample gesture picture, and by comparing each picture in the first picture set with each picture of the first sample gesture picture, the embodiment of the present invention There are two possible situations:

If yes, go to step 203;

If not, step 207 is executed.

203. Use the first sample gesture picture as a first gesture operation, and determine a corresponding target smart home device from the plurality of smart home devices according to the first gesture operation.

In this embodiment of the present invention, the terminal uses the first sample gesture picture as a first gesture operation, and determines a corresponding target smart home device from the plurality of smart home devices according to the first gesture operation.

For example, please refer to FIG. 4. FIG. 4 is a schematic diagram of a first gesture operation. The number of the first sample gesture pictures may be multiple, and the sample gesture pictures include a number one gesture 301, a number two gesture 302 and a number three gesture 303, the smart home device corresponding to the number one gesture 301 is a smart air conditioner, the smart home device corresponding to the number two gesture 302 is a smart TV, the smart home device corresponding to the number three gesture 303 is a smart curtain, and other digital gestures The above methods are in one-to-one correspondence, and are not repeated here. A picture in the first picture set is consistent with the number one gesture 301 in the plurality of first sample gesture pictures, then the first sample gesture picture is used as the first gesture operation, that is, the number one gesture 301 is used as the first gesture operation, and according to the first gesture operation, it is determined that the corresponding smart home device is a smart air conditioner.

204. Obtain a second set of pictures according to the second specified number of pictures obtained by the photographing element within the second preset duration, and determine whether there is a picture in the second set of pictures that matches the second sample gesture picture, and if so , then go to step 205; if not, go to step 207. .

In this embodiment of the present invention, the number of the second sample gesture pictures may be multiple, and the photographing element is activated, and the photographing element obtains the second specified number of pictures within the second preset time period to obtain the first Two sets of pictures, and then, it is judged whether there is a picture in the second set of pictures that matches the second sample gesture picture, and the present invention The embodiment has the following two possible situations:

If yes, step 205 is executed.

If not, step 207 is executed.

205. Determine a corresponding control instruction according to the second gesture operation, and send the control instruction to the target smart home device.

In the embodiment of the present invention, the corresponding control instruction is determined according to the second gesture operation, and the control instruction is sent to the target smart home device.

For example, please refer to FIG. 5. FIG. 5 is a schematic diagram of the second gesture operation. The plurality of second sample gesture pictures include a fist 401 in the air, an expansion 402 in the air, an upward swing 403 in the air, and a downward swing 404 in the air. When the recognized second gesture operation is to make a fist 401 in the air, the corresponding control command is determined according to the operation of the fist 401 in the air, and the control command is: turn off the current smart home equipment, that is, turn off the air conditioner, and set the control command to Send to the smart home equipment; when the recognized second gesture operation is to expand 402 from the air, determine the corresponding control instruction according to the operation of the expanded 402 from the air, and the control instruction is: turn on the current smart home equipment, that is, Turn on the air conditioner, and send the control command to the smart home device; when the recognized second gesture operation is waving 403 in the air, determine the corresponding control command according to the operation of waving 403 in the air, and the The control command is: increase the current smart home device, that is, increase the temperature of the air conditioner, and send the control command to the smart home device; when the recognized second gesture operation is to wave 404 in the air, according to the Said swiping operation 404 in the air is used to determine the corresponding control command. The control command is: lowering the current smart home device, that is, lowering the temperature of the air conditioner, and sending the control command to the smart home device. Other space gestures can be corresponded one-to-one according to the above method, and will not be repeated here.

It should be noted that this is only an example, and the smart home devices, including but not limited to smart air conditioners, smart refrigerators, smart speakers, smart doors and windows, smart water heaters, smart curtains, smart computers, and smart TVs, can establish communication with the terminal. Connected smart home devices. The control instructions corresponding to the air gestures include, but are not limited to, turn on the device, turn off the device, increase a certain function of the device, decrease a certain function of the device, start transmission, stop transmission, and the like.

206. Control the running state of the corresponding smart home device based on the control instruction.

In the embodiment of the present invention, the smart home device receives a control instruction sent by the terminal, and controls the operation state of the corresponding smart home device based on the control instruction.

For example, when it is determined that the target smart home device is a smart air conditioner, the embodiment of the present invention specifically has the following four possible implementation manners:

207. End the smart home control process.

To sum up, an embodiment of the present invention provides a smart home control method. First, when the screen of the terminal is in an off-screen state, if it is detected that the terminal and the smart home device establish a communication connection, the user's information can be obtained through the photographing element. In the gesture operation in the air, the target smart home device is determined according to the first gesture operation, and the smart home device is controlled in the air according to the second gesture operation, without lighting the screen to operate, saving power consumption. Secondly, it can overcome the inconvenience that the command transmission between interconnected devices usually needs to be used for operation on a graphical operation interface. Users can directly transmit commands between interconnected devices through air gestures to control the interconnected devices, which brings convenience to users. In addition, users can also define their own preferred gesture actions, with various control methods and great flexibility. It is fast and convenient to control multiple smart home devices, and the operation is simple and easy to learn, which expands the richness of user operations.

Please refer to FIG. 6. FIG. 6 is a schematic flowchart of a smart home control method according to Embodiment 3 of the present invention. Wherein, the embodiments of the present invention include:

501. When the screen of the terminal is in an off-screen state, if it is detected that a communication connection is established between the terminal and the smart home device, acquire a user's air gesture operation.

Specifically, in this embodiment of the present invention, to obtain the user's air gesture operation, the terminal can obtain the user's air gesture operation through the camera element built in the terminal, and the air gesture operation includes a first gesture and a second gesture , the terminal is connected with a plurality of smart home devices, the first gesture operation includes but is not limited to digital gestures, and the second gesture operation includes but is not limited to swipe up in the air, swipe down in the air, and move left in the air , move right in the air, press down in the air, grab in the air, make a fist in the air, compare numbers in the air, gesture V-shaped or heart-shaped and other gestures, wherein, in the embodiment of the present invention, according to the first gesture The operation determines the corresponding target smart home device from the plurality of smart home devices; and determines the corresponding control instruction according to the second gesture operation.

502. Obtain a first specified number of pictures within a first preset duration by the photographing element, obtain a first picture set, and determine whether there is a picture in the first picture set that matches the first sample gesture picture; , step 503 is performed; otherwise, step 508 is performed.

Specifically, in the embodiment of the present invention, the number of the first sample gesture pictures may be multiple, and the photographing element is activated, and the photographing element acquires the first specified number of first shots within a first preset time period. A set of pictures, and then, it is judged whether there is a picture in the first set of pictures that matches the first sample gesture picture, and by comparing each picture in the first set of pictures with each picture of the first sample gesture picture, this There are the following two possible situations in the embodiment of the invention:

If yes, step 503 is executed.

If not, step 508 is executed.

503. Collect the face photo of the current user, and determine whether the face photo matches the sample face photo, if yes, go to step 504; if not, go to step 508.

Specifically, when the terminal senses a face through a photographing element (the photographing element may be a built-in camera of the terminal, or a camera external to the terminal) in the locked screen state of the terminal, the terminal is triggered to perform face recognition. The terminal collects the face photo of the current user through the photographing element, and obtains the face photo to be recognized; matches the to-be-recognized face photo with the sample face photo pre-recorded in the terminal, and judges the face photo and the sample face photo Whether it matches or not, the specific face recognition process can be implemented with reference to the existing face recognition technology, which will not be repeated here. There are the following two possible situations in this embodiment of the present invention:

If yes, step 504 is executed.

If not, step 508 is executed.

504. Use the first sample gesture picture as a first gesture operation, and determine a corresponding target smart home device from the plurality of smart home devices according to the first gesture operation.

505. Obtain a second set of pictures according to the second specified number of pictures obtained by the photographing element within the second preset duration, and determine whether there is a picture in the second set of pictures that matches the second sample gesture picture, and if so , then go to step 506; if not, go to step 508.

Specifically, in this embodiment of the present invention, the number of the second sample gesture pictures may be multiple, and the photographing element is activated, and the photographing element is used to obtain a second specified number of pictures within a second preset time period. , obtain the second set of pictures, and then, determine whether there is a picture in the second set of pictures that matches the second sample gesture picture, by comparing each picture in the second set of pictures with each picture of the second sample gesture picture , there are the following two possible situations in this embodiment of the present invention:

If yes, step 506 is executed.

If not, step 508 is executed.

506. Determine a corresponding control instruction according to the second gesture operation, and send the control instruction to the target smart home device.

507. Control the running state of the corresponding smart home device based on the control instruction.

508. End the smart home control process.

To sum up, an embodiment of the present invention provides a smart home control method. First, when the screen of the terminal is in an off-screen state, if it is detected that the terminal and the smart home device establish a communication connection, the user's information can be obtained through the photographing element. In the gesture operation in the air, the target smart home device is determined according to the first gesture operation, and the smart home device is controlled in the air according to the second gesture operation, without lighting the screen to operate, saving power consumption. Secondly, before determining the corresponding target smart home device from the plurality of smart home devices according to the first gesture operation, the current user identity is judged, and different users have different permissions to control the smart home device, so as to protect user privacy. The embodiment of the present invention can overcome the inconvenience that the instruction transmission between interconnected devices usually needs to be used for operation on a graphical operation interface, and the user can directly transmit instructions between interconnected devices through air gestures to control the interconnected devices, which brings convenience to the user. In addition, users can also define their own preferred gesture actions, with various control methods and great flexibility. It is fast and convenient to control multiple smart home devices, and the operation is simple and easy to learn, which expands the richness of user operations.

Please refer to FIG. 7 , which is a schematic flowchart of a smart home control method according to Embodiment 4 of the present invention. Wherein, the embodiments of the present invention include:

601. When the screen of the terminal is in an off-screen state, if it is detected that a communication connection is established between the terminal and the smart home device, acquire a user's air gesture operation.

602. Acquire the first picture set and identify whether there is a hand area, if yes, go to step 603; if otherwise, go to 609.

In the embodiment of the present invention, the photographing element is activated, and the photographing element acquires a first set of pictures according to a first preset number of shots within a first preset time, and performs content on each picture in the first set of pictures. Recognition; according to the recognition result, determine whether each picture in the first picture set has a hand area:

If yes, go to step 603;

If not, go to step 609.

603. Determine whether there is a picture in the first picture set that matches the first sample gesture picture.

Specifically, in this embodiment of the present invention, the terminal determines whether there is a picture in the first picture set that matches the first sample gesture picture, by comparing each picture in the first picture set with each of the first sample gesture pictures Picture comparison, there are the following two possible situations in the embodiment of the present invention:

If yes, go to step 604 .

If not, step 609 is executed.

604. Collect the face photo of the current user, and determine whether the face photo matches the sample face photo, if yes, go to step 605; if not, go to step 609.

If yes, step 605 is executed.

If not, step 609 is executed.

605. Use the first sample gesture picture as a first gesture operation, and determine a corresponding target smart home device from the plurality of smart home devices according to the first gesture operation.

606. Obtain a second set of pictures according to the second specified number of pictures obtained by the photographing element within the second preset duration, and determine whether there is a picture in the second set of pictures that matches the second sample gesture picture, and if so , then go to step 607 ; if not, go to step 609 .

If yes, step 607 is executed.

If not, step 609 is executed.

607. Determine a corresponding control instruction according to the second gesture operation, and send the control instruction to the target smart home device.

608. Control the running state of the corresponding smart home device based on the control instruction.

609. End the smart home control process.

To sum up, an embodiment of the present invention provides a smart home control method. First, when the screen of the terminal is in an off-screen state, if it is detected that the terminal and the smart home device establish a communication connection, the user's information can be obtained through the photographing element. In the gesture operation in the air, the target smart home device is determined according to the first gesture operation, and the smart home device is controlled in the air according to the second gesture operation, without lighting the screen to operate, saving power consumption. Secondly, before determining the corresponding target smart home device from the plurality of smart home devices according to the first gesture operation, the current user identity is judged, and whether there is a hand picture in the first picture set is identified to ensure control Different users have different permissions to control smart home devices, and it can also protect user privacy. The embodiment of the present invention can overcome the inconvenience that the instruction transmission between interconnected devices usually needs to be used for operation on a graphical operation interface, and the user can directly transmit instructions between interconnected devices through air gestures to control the interconnected devices, which brings convenience to the user. In addition, users can also define their own preferred gesture actions, with various control methods and great flexibility. It is fast and convenient to control multiple smart home devices, and the operation is simple and easy to learn, which expands the richness of user operations.

Please refer to FIG. 8 , which is a schematic flowchart of a smart home control method according to Embodiment 5 of the present invention. Wherein, the embodiments of the present invention include:

701. When the screen of the terminal is in an off-screen state, if it is detected that a communication connection is established between the terminal and the smart home device, acquire a user's air gesture operation.

Specifically, the terminal and the smart home device can establish a communication connection by means of communication, and the terminal and the smart home device can use the Wi-Fi network, the Bluetooth network, the ZigBee network and the Digital Living Network Alliance (Digital Living Network Alliance). Living Network Alliance, DLNA), etc., but not limited to this.

In the embodiment of the present invention, the first infrared light is emitted to the outside through the signal transmitter of the infrared sensor element, and the first infrared light reflected by the external obstacle is received through the signal receiver of the infrared sensor element. Two infrared light. A gesture signal is generated according to the second infrared light.

702. Determine whether the gesture signal matches the sample gesture signal, if yes, go to step 703; if not, go to step 706.

In the embodiment of the present invention, the sample gesture signal may be multiple, and a sample gesture signal corresponds to an operation gesture, and it is determined whether the gesture signal matches the sample gesture signal:

If yes, go to step 703;

If not, go to step 706;

703. Use the gesture operation corresponding to the sample gesture signal as a space gesture operation.

Specifically, the terminal regards the gesture operation corresponding to the sample gesture signal as an air gesture operation. In this embodiment of the present invention, to obtain the user's air gesture operation, the terminal may obtain the user's air gesture operation through the camera element built in the terminal, and the air gesture operation includes but is not limited to waving up and down in the air. Swing, move left in the air, move right in the air, push down in the air, grab in the air, make a fist in the air, compare numbers in the air, gesture V or heart shape and other gestures.

704. Determine a corresponding control instruction according to the air gesture operation.

705. Control the running state of the smart home device based on the control instruction.

706. End the smart home control process.

To sum up, an embodiment of the present invention provides a smart home control method. First, when the screen of the terminal is in an off-screen state, if it is detected that a communication connection is established between the terminal and the smart home device, the user is acquired through infrared sensing elements. Remote gesture operation, and control smart home devices from the air according to the gesture operation, no need to light the screen to operate, saving power consumption. Secondly, it can overcome the inconvenience that the command transmission between interconnected devices usually needs to be used for operation on a graphical operation interface. Users can directly transmit commands between interconnected devices through air gestures to control the interconnected devices, which brings convenience to users. In addition, users can also define their own preferred gesture actions, with various control methods and great flexibility. It is fast and convenient to control multiple smart home devices, and the operation is simple and easy to learn, which expands the richness of user operations.

Please refer to FIG. 9. FIG. 9 is a schematic flowchart of a smart home control method according to Embodiment 6 of the present invention. Wherein, the embodiments of the present invention include:

801. When the screen of the terminal is in an off-screen state, if it is detected that a communication connection is established between the terminal and a smart home device, use the voice information collected by the voice recognition element.

Specifically, in the embodiment of the present invention, voice recognition processing is performed on the voice information through the voice information collected by the voice recognition element, and a target intelligence corresponding to the voice information is determined from the plurality of smart home devices. For home equipment, for example, the voice information collected by the voice recognition element is "smart air conditioner", and the target smart home equipment is determined from the plurality of smart home equipment as "smart air conditioner".

802. Acquire an air gesture operation of the user.

Optionally, the terminal has a photographing element, a plurality of smart home devices are connected to the terminal, and the running state of the target smart home equipment is controlled based on the corresponding control instruction determined according to the gesture operation in the air, and the photographing element is used in the pre-preset operation state. Suppose that the specified number of pictures are obtained within the duration, and a picture set is obtained; it is judged whether there is a picture in the picture set that matches the sample gesture picture; if there is a picture in the picture set that matches the sample gesture picture, then the sample The air gesture operation corresponding to the gesture picture is used as the air gesture operation. Wherein, before judging whether there is a picture in the picture set that matches the sample gesture picture, content recognition is performed on each picture in the picture set; according to the recognition result, it is judged whether each picture in the picture set has a hand area; when When it is judged that there is a hand area, it is judged whether there is a picture in the picture set that matches the first sample gesture picture.

803. Determine a corresponding control instruction according to the air gesture operation.

804. Control the running state of the smart home device based on the control instruction.

In order to better implement the above method, an embodiment of the present invention may further provide a smart home control apparatus, which may be specifically integrated in a network device, and the network device may be a device such as a mobile terminal.

For example, as shown in FIG. 10, the smart home control device may include a detection module 901, an acquisition module 902, a determination module 903 and a processing module 904, as follows:

The detection module 901 is used to detect that the screen of the terminal is in an off-screen state, and is used to detect that the terminal establishes a communication connection with a smart home device;

an acquisition module 902, configured to acquire the user's gesture operation in the air;

A determination module 903, configured to determine a corresponding control instruction according to the air gesture operation;

The processing module 904 is used to control the running state of the corresponding smart home device based on the control instruction

Optionally, in some embodiments of the present invention, the determining module 903 is further configured to:

Optionally, in some embodiments of the present invention, the obtaining module 902 is further configured to:

Optionally, in some embodiments of the present invention, the processing module 904 is further configured to:

generating a gesture signal according to the second infrared light;

Determine whether the gesture signal matches the sample gesture signal;

An embodiment of the present invention discloses a smart home control device. The smart home control device includes: a detection module detects that when the screen of the terminal is in an off-screen state, the terminal establishes a communication connection with a smart home device, and an acquisition module acquires the user Then, the determination module determines the corresponding control command according to the air gesture operation, and finally, the processing module controls the operation state of the corresponding smart home device based on the control command. In the embodiment of the present invention, when the screen of the terminal is in the screen-off state, if it is detected that the terminal establishes a communication connection with the smart home device, the user's gesture operation in the air can be obtained, and the smart home can be controlled in the air according to the gesture operation in the air The device does not need to light up the screen to operate, saving power consumption. Secondly, it can overcome the inconvenience that the transmission of instructions between interconnected devices usually needs to be used for operation on a graphical operation interface. Users can directly transmit instructions between interconnected devices through air gestures to control the interconnected devices, which brings convenience to users. In addition, users can also define their own preferred gesture actions, with various control methods and great flexibility. It is fast and convenient to control multiple smart home devices, and the operation is simple and easy to learn, which expands the richness of user operations.

The terminals in the embodiments of the present invention are described above from the perspective of modular functional entities, and the terminals in the embodiments of the present invention are respectively described below from the perspective of hardware processing. An embodiment of the present invention also provides a terminal, as shown in FIG. 10 , which shows a schematic structural diagram of a terminal involved in an embodiment of the present invention. The terminal may be a mobile phone, a computer, a digital broadcasting terminal, a message sending and receiving device, a game Consoles, Tablets, Medical Equipment, Fitness Equipment, Personal Digital Assistants, etc. Specifically:

As shown in FIG. 11 , the terminal may include a radio frequency (RF) circuit 111, a memory 112 including one or more computer-readable storage media, an input unit 113, a display unit 114, a sensor 115, an audio circuit 116, Wireless Fidelity Fidelity (Wi-Fi) module 117, a processor 118 including one or more processing cores, a power supply 119 and other components. Those skilled in the art can understand that the terminal structure shown in FIG. 11 does not constitute a limitation on the terminal, and may include more or less components than shown, or combine some components, or arrange different components. in:

The RF circuit 111 can be used for receiving and sending signals during the process of sending and receiving information or talking. In particular, after receiving the downlink information of the base station, it is handed over to one or more processors 118 for processing; in addition, it sends the data related to the uplink to the base station. . Typically, the RF circuit 111 includes, but is not limited to, an antenna, at least one amplifier, a tuner, one or more oscillators, a subscriber identity module (Subscriber Identity Module, SIM) card, transceiver, coupler, low noise amplifier (Low Noise Amplifier, LNA), duplexer, etc. In addition, the RF circuit 111 can also communicate with the network and other devices through wireless communication. The wireless communication can use any communication standard or protocol, including but not limited to the Global System for Mobile Communications (Global System of Mobile communication, GSM), General Packet Radio Service (General Packet Radio Service, GPRS), Code Division Multiple Access (Code Division Multiple Access (CDMA), Wideband Code Division Multiple Access (WCDMA), Long Term Evolution (Long Term Evolution, LTE), e-mail, Short Message Service (Short Messaging Service, SMS), etc.

The memory 112 can be used to store software programs and modules, and the processor 118 executes various functional applications and data processing by running the software programs and modules stored in the memory 112 . The memory 112 may mainly include a stored program area and a stored data area, wherein the stored program area may store an operating system, an application program required for at least one function (such as a sound playback function, an image playback function, etc.), and the like; Data created by the use of the terminal (such as audio data, phone book, etc.), etc. Additionally, memory 112 may include high-speed random access memory, and may also include non-volatile memory, such as at least one magnetic disk storage device, flash memory device, or other volatile solid state storage device. Accordingly, the memory 112 may also include a memory controller to provide access to the memory 112 by the processor 118 and the input unit 113 .

The input unit 113 may be used to receive input numerical or character information, and generate keyboard, mouse, joystick, optical or trackball signal input related to user settings and function control. Specifically, in a specific embodiment, the input unit 113 may include a touch-sensitive surface as well as other input devices. A touch-sensitive surface, also known as a touch display or trackpad, collects the user's touch operations on or near it (such as the user's finger, stylus, or any suitable operation near the surface), and drive the corresponding connection device according to the preset program. Optionally, the touch-sensitive surface may include two parts, a touch detection device and a touch controller. Among them, the touch detection device detects the user's touch orientation, detects the signal brought by the touch operation, and transmits the signal to the touch controller; the touch controller receives the touch information from the touch detection device, converts it into contact coordinates, and then sends it to the touch controller. To the processor 118, and can receive and execute commands sent by the processor 118. Additionally, touch-sensitive surfaces can be implemented using resistive, capacitive, infrared, and surface acoustic wave types. In addition to the touch-sensitive surface, the input unit 113 may also include other input devices. Specifically, other input devices may include, but are not limited to, one or more of physical keyboards, function keys (such as volume control keys, switch keys, etc.), trackballs, mice, joysticks, and the like.

The display unit 114 may be used to display information input by the user or information provided to the user and various graphical user interfaces of the terminal, which may be composed of graphics, text, icons, videos and any combination thereof. The display unit 114 may include a display screen, optionally, a liquid crystal display (Liquid Crystal Display, LCD), an organic light emitting diode (Organic light emitting diode) may be used. Light-Emitting Diode, OLED) and other forms to configure the display. Further, the touch-sensitive surface may cover the display screen, and when the touch-sensitive surface detects a touch operation on or near it, it is transmitted to the processor 118 to determine the type of the touch event, and then the processor 118 displays the touch event according to the type of the touch event. The corresponding visual output is provided on the screen.

The terminal may also include at least one sensor 115, such as a light sensor, a motion sensor, and other sensors. Specifically, the light sensor may include an ambient light sensor and a proximity sensor, wherein the ambient light sensor may adjust the brightness of the display screen according to the brightness of the ambient light, and the proximity sensor may turn off the display screen and/or the backlight when the terminal is moved to the ear . As a kind of motion sensor, the gravitational acceleration sensor can detect the magnitude of acceleration at each position (usually three axes), and can detect the magnitude and position of gravity when stationary, and can be used for applications that recognize the posture of mobile phones (such as horizontal and vertical screen switching, related games, magnetometer attitude calibration), vibration recognition related functions (such as pedometer, tapping), etc.; as for other sensors such as gyroscope, barometer, hygrometer, thermometer, infrared sensor, etc. Repeat.

The audio circuit 116, speakers, and microphones may provide an audio interface between the user and the terminal. The audio circuit 116 can convert the received audio data into an electrical signal, and transmit it to the speaker, which is converted into a sound signal for output; on the other hand, the microphone converts the collected sound signal into an electrical signal, which is received and converted by the audio circuit 116 In the form of audio data, the audio data is output to the processor 118 for processing, and then sent to, for example, another terminal via the RF circuit 111, or the audio data is output to the memory 112 for further processing. The audio circuit 116 may also include an earphone jack to provide communication of peripheral headphones with the terminal.

Wi-Fi is a short-distance wireless transmission technology, and the terminal can help users to send and receive emails, browse web pages, access streaming media, etc. through the Wi-Fi module 117, and it provides users with wireless broadband Internet access. Although FIG. 11 shows the Wi-Fi module 117, it can be understood that it does not belong to the necessary structure of the terminal, and can be completely omitted as required within the scope of not changing the essence of the invention.

The processor 118 is the control center of the terminal, uses various interfaces and lines to connect various parts of the entire mobile phone, and executes by running or executing the software programs and/or modules stored in the memory 112, and calling the data stored in the memory 112. Various functions of the terminal and processing data, so as to monitor the mobile phone as a whole. Optionally, the processor 118 may include one or more processing cores; preferably, the processor 118 may integrate an application processor and a modem processor, wherein the application processor mainly processes the operating system, user interface, and application programs, etc. , the modem processor mainly deals with wireless communication. It can be understood that, the above-mentioned modulation and demodulation processor may not be integrated into the processor 118 .

The terminal also includes a power supply 119 (such as a battery) for supplying power to various components. Preferably, the power supply can be logically connected to the processor 118 through a power management system, so as to manage charging, discharging, and power consumption management functions through the power management system. The power source 119 may also include one or more DC or AC power sources, recharging systems, power failure detection circuits, power converters or inverters, power status indicators, and any other components.

Although not shown, the terminal may further include a camera, a Bluetooth module, and the like, which will not be repeated here. Specifically in this embodiment, the processor 118 in the terminal loads the executable files corresponding to the processes of one or more application programs into the memory 112 according to the following instructions, and the processor 118 executes them and stores them in the memory 112 applications in order to achieve various functions.

The electronic device provided by the embodiment of the present invention can overcome the inconvenience that the transmission of instructions between interconnected devices usually needs to be used for operation on a graphical operation interface. Users can directly transmit instructions between interconnected devices through air gestures to control the interconnected devices. Bring convenience. In addition, users can also define their own preferred gesture actions, with various control methods and great flexibility. It is fast and convenient to control multiple smart home devices, and the operation is simple and easy to learn, which expands the richness of user operations.

On the other hand, an embodiment of the present invention also provides a storage medium on which a computer program is stored, and the computer program is implemented by a processor to execute the methods provided by the above embodiments, for example, including: when the screen of the terminal is When in the screen-off state, if it is detected that the terminal establishes a communication connection with the smart home device, the user's air gesture operation is obtained; the corresponding control instruction is determined according to the air gesture operation; the smart home is controlled based on the control instruction equipment operating status

In the above-mentioned embodiments, the description of each embodiment has its own emphasis. For parts that are not described in detail in a certain embodiment, reference may be made to the relevant descriptions of other embodiments. The above-described embodiments are only some of the embodiments of the present invention, but not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative work, except for such designs mentioned in the embodiments of the present invention that are consistent with the solutions of the embodiments of the present invention, belong to the present invention scope of protection.

A smart home control method, device, terminal, and storage medium provided by the embodiments of the present invention are described above in detail, and specific examples are used in this paper to illustrate the principles and implementations of the present invention. The descriptions of the above embodiments are only It is used to help understand the technical solution of the present invention and its core idea; those of ordinary skill in the art should understand that it can still modify the technical solutions described in the foregoing embodiments, or perform equivalent replacements to some of the technical features; and These modifications or substitutions do not make the essence of the corresponding technical solutions depart from the scope of the technical solutions of the embodiments of the present invention.

Claims

A control method for a smart home, applied to a terminal, comprising:

When the screen of the terminal is in the screen-off state, if it is detected that the terminal establishes a communication connection with the smart home device, acquiring the user's gesture operation in the air;

Determine the corresponding control command according to the air gesture operation;

The operating state of the corresponding smart home device is controlled based on the control instruction.
The method according to claim 1, wherein the terminal is connected with a plurality of smart home devices, the air gesture operation includes a first gesture operation and a second gesture operation; the determining according to the air gesture operation Corresponding control instructions, including:

Determine a corresponding target smart home device from the plurality of smart home devices according to the first gesture operation;

Determine the corresponding control instruction according to the second gesture operation;

The controlling the running state of the smart home device based on the control instruction includes:

The operating state of the target smart home device is controlled based on the corresponding control instruction determined according to the second gesture operation.
The method according to claim 2, wherein before determining the corresponding target smart home device from the plurality of smart home devices according to the first gesture operation, the method further comprises:

Collect the face photo of the current user;

Judging whether the face photo matches the sample face photo;

If yes, determine the corresponding target smart home device from the plurality of smart home devices according to the first gesture operation.
The method according to claim 2 or 3, wherein the terminal has a photographing element; and the acquiring the user's air gesture operation includes:

Obtaining a first set of pictures by acquiring a first specified number of pictures within a first preset duration by the photographing element;

Judging whether there is a picture in the first picture set that matches the first sample gesture picture;

If there is a picture in the first picture set that matches the first sample gesture picture, take the space gesture operation corresponding to the first sample gesture picture as the first gesture operation, and use the photographing element to Obtain the second set of pictures according to the second specified number of pictures obtained within the second preset time period;

Judging whether there is a picture in the second picture set that matches the second sample gesture picture;

If there is a picture in the second picture set that matches the second sample gesture picture, the space gesture operation corresponding to the second sample gesture picture is used as the second gesture operation.
The method according to claim 4, wherein before judging whether there is a picture in the first picture set that matches the first sample gesture picture, the method further comprises:

performing content identification on each picture in the first picture set;

According to the recognition result, determine whether each picture in the first picture set has a hand area;

When it is judged that there is a hand area, it is judged whether there is a picture in the first picture set that matches the first sample gesture picture.
The method according to claim 1, wherein the terminal has an infrared sensor element; the acquiring the user's gesture operation in the air comprises:

The signal transmitter of the infrared sensor element emits first infrared light to the outside world, and the signal receiver of the infrared sensor element receives the second infrared light reflected by the first infrared light through external obstacles;

generating a gesture signal according to the second infrared light;

Determine whether the gesture signal matches the sample gesture signal;

If yes, the gesture operation corresponding to the sample gesture signal is used as a space gesture operation.
The method according to claim 1, wherein the terminal has a voice recognition element, and the terminal is connected with a plurality of smart home devices; the method further comprises:

Voice information collected by the voice recognition element;

Perform voice recognition processing on the voice information, and determine a target smart home device corresponding to the voice information from the plurality of smart home devices;

Controlling the running state of the smart home device based on the control instruction includes:

The target smart home device corresponding to the voice information is controlled based on the control instruction.
The method according to claim 7, wherein the terminal is connected with a plurality of smart home devices; the determining the corresponding control instruction according to the air gesture operation, comprising:

The running state of the target smart home device is controlled based on the corresponding control instruction determined according to the air gesture operation.
The method according to claim 8, wherein the terminal has a photographing element; the acquiring the user's air gesture operation comprises:

Obtaining a specified number of pictures within a preset time period by the photographing element to obtain a picture set;

Determine whether there is a picture in the picture set that matches the sample gesture picture;

If a picture in the picture set matches the sample gesture picture, the space gesture operation corresponding to the sample gesture picture is used as the space gesture operation.
The method according to claim 9, wherein before judging whether there is a picture in the picture set that matches the sample gesture picture, the method further comprises:

performing content identification on each picture in the picture set;

According to the recognition result, determine whether each picture in the picture set has a hand area;

When it is judged that there is a hand region, it is judged whether there is a picture in the picture set that matches the first sample gesture picture.
A smart home control device, comprising:

a detection module, used to detect that the screen of the terminal is in an off-screen state, and used to detect that the terminal establishes a communication connection with the smart home device;

The acquisition module is used to acquire the user's air gesture operation;

a determination module, configured to determine a corresponding control instruction according to the gesture operation in the air;

The processing module is configured to control the running state of the corresponding smart home device based on the control instruction.
The apparatus of claim 11, wherein the determining module is further configured to:

Determine a corresponding target smart home device from the plurality of smart home devices according to the first gesture operation;

The corresponding control instruction is determined according to the second gesture operation.
The apparatus of claim 12, wherein the apparatus further comprises:

a judging module for judging whether the face photo matches the sample face photo;

If yes, determine the corresponding target smart home device from the plurality of smart home devices according to the first gesture operation.
A terminal includes a memory, a processor and a computer program stored on the memory and running on the processor, the processor can implement the steps of a smart home control method when the processor executes the program, wherein the method includes :

When the screen of the terminal is in the screen-off state, if it is detected that the terminal establishes a communication connection with the smart home device, acquiring the user's gesture operation in the air;

Determine the corresponding control command according to the air gesture operation;

The operating state of the corresponding smart home device is controlled based on the control instruction.
The terminal according to claim 14, wherein the terminal is connected with a plurality of smart home devices, the air gesture operation includes a first gesture operation and a second gesture operation; the determining according to the air gesture operation Corresponding control instructions, including:

Determine a corresponding target smart home device from the plurality of smart home devices according to the first gesture operation;

Determine the corresponding control instruction according to the second gesture operation;

The controlling the running state of the smart home device based on the control instruction includes:

The operating state of the target smart home device is controlled based on the corresponding control instruction determined according to the second gesture operation.
The terminal according to claim 15, wherein before the determining the corresponding target smart home device from the plurality of smart home devices according to the first gesture operation, the method further comprises:

Collect the face photo of the current user;

Judging whether the face photo matches the sample face photo;

If yes, determine the corresponding target smart home device from the plurality of smart home devices according to the first gesture operation.
The terminal according to claim 15 or 16, wherein the terminal has a photographing element; the acquiring the user's air gesture operation comprises:

Obtaining a first set of pictures by acquiring a first specified number of pictures within a first preset duration by the photographing element;

Judging whether there is a picture in the first picture set that matches the first sample gesture picture;

If there is a picture in the first picture set that matches the first sample gesture picture, take the space gesture operation corresponding to the first sample gesture picture as the first gesture operation, and use the photographing element to Obtain the second set of pictures according to the second specified number of pictures obtained within the second preset time period;

Judging whether there is a picture in the second picture set that matches the second sample gesture picture;

If there is a picture in the second picture set that matches the second sample gesture picture, the space gesture operation corresponding to the second sample gesture picture is used as the second gesture operation.
The terminal according to claim 17, wherein before judging whether there is a picture in the first picture set that matches the first sample gesture picture, the method further comprises:

performing content identification on each picture in the first picture set;

According to the recognition result, determine whether each picture in the first picture set has a hand area;

When it is judged that there is a hand region, it is judged whether there is a picture in the first picture set that matches the first sample gesture picture.
The terminal according to claim 14, wherein the terminal has an infrared sensor element; the acquiring the user's gesture operation in the air comprises:

The signal transmitter of the infrared sensor element emits first infrared light to the outside, and the signal receiver of the infrared sensor element receives the second infrared light reflected by the first infrared light through external obstacles;

generating a gesture signal according to the second infrared light;

Determine whether the gesture signal matches the sample gesture signal;

If so, the gesture operation corresponding to the sample gesture signal is used as a space gesture operation.
The terminal according to claim 14, wherein the terminal has a voice recognition element, and the terminal is connected with a plurality of smart home devices; the method further comprises:

voice information collected by the voice recognition element;

Perform voice recognition processing on the voice information, and determine a target smart home device corresponding to the voice information from the plurality of smart home devices;

Controlling the running state of the smart home device based on the control instruction includes:

The target smart home device corresponding to the voice information is controlled based on the control instruction.