CN113516840A

CN113516840A - Electric appliance control method, camera device, electronic equipment and storage medium

Info

Publication number: CN113516840A
Application number: CN202111065522.6A
Authority: CN
Inventors: 杨洋; 黄燕青; 汪紫超
Original assignee: Shanghai Chuangmi Technology Co ltd; Beijing Chuangmizhihui Iot Technology Co ltd
Current assignee: Shanghai Imilab Technology Co Ltd
Priority date: 2021-09-13
Filing date: 2021-09-13
Publication date: 2021-10-19
Anticipated expiration: 2041-09-13
Also published as: CN113516840B

Abstract

The disclosure provides an electric appliance control method, an image pickup device, an electronic apparatus, and a storage medium. Wherein, the method comprises the following steps: identifying an electric appliance in the first target image, and acquiring the position of the electric appliance in the first target image; determining the offset angle of the electric appliance relative to the camera device according to the position and a camera coordinate system when the camera device takes the first target image; adjusting the emission direction of the infrared emission device according to the offset angle; searching each infrared code list, and transmitting a first infrared signal corresponding to at least one infrared control code in the infrared code list by using an infrared transmitting device so as to detect whether the infrared code list can control the electric appliance; and under the condition that the infrared code list capable of controlling the electric appliance is detected, recording the corresponding relation between the infrared code list and the electric appliance, and recording the offset angle. This openly can realize controlling electrical apparatus by the camera device who installs infrared emitter, conveniently controls electrical apparatus, improves user experience.

Description

Electric appliance control method, camera device, electronic equipment and storage medium

Technical Field

The present disclosure relates to the field of smart home, and in particular, to an electrical appliance control method, a camera device, an electronic device, and a storage medium.

Background

There are many home appliances such as televisions, projectors, audio equipment, air conditioners, electric fans, etc. that use infrared remote controls. Each device is provided with a special infrared remote controller, and the device is often taken by mistake in the using process, is easy to find and is inconvenient.

Disclosure of Invention

The disclosure provides an electric appliance control method, a camera device, an electronic device and a storage medium, and the control of an electric appliance can be realized by utilizing the camera device provided with an infrared emitting device, so that the control of the electric appliance is facilitated, and the user experience is improved.

In a first aspect, the present disclosure provides an electrical appliance control method applied to a camera device equipped with an infrared emission device, including:

capturing a first target image;

identifying an electric appliance in the first target image, and acquiring the position of the electric appliance in the first target image;

determining the offset angle of the electric appliance relative to the camera device according to the position of the electric appliance in the first target image and the camera coordinate system of the camera device when the camera device takes the first target image;

adjusting the emission direction of the infrared emission device according to the offset angle;

searching at least one infrared code list corresponding to the electric appliance, wherein each infrared code list comprises at least one infrared control code; aiming at each infrared code list, the infrared emission device is adopted to emit a first infrared signal corresponding to at least one infrared control code in the infrared code list so as to detect whether the infrared code list can control the electric appliance;

and under the condition that the infrared code list capable of controlling the electric appliance is detected, recording the corresponding relation between the infrared code list capable of controlling the electric appliance and the electric appliance, and recording the offset angle of the electric appliance relative to the camera device.

In one possible implementation, the list of ir codes that can control the appliance includes: at least part of the infrared control codes can control the infrared code list of the electric appliance.

In a possible implementation manner, searching for at least one infrared encoding list corresponding to the electrical appliance includes:

extracting the image containing the appliance from the first target image;

inputting an image containing the electric appliance into a pre-trained image recognition model, and outputting the brand and/or model of the electric appliance by the image recognition model;

and searching a plurality of pre-stored infrared code lists according to the brand and/or model of the electric appliance, and determining at least one infrared code list corresponding to the brand and/or model of the electric appliance.

In one possible implementation, identifying an appliance in the first target image and obtaining a location of the appliance in the first target image includes:

inputting the first target image into a pre-trained target detection model, and outputting the type of an electric appliance contained in the first target image, the position of the electric appliance in the first target image and a confidence degree by the target detection model;

and recording the type of the electric appliance and the position of the electric appliance in the first target image under the condition that the confidence coefficient is greater than a preset threshold value.

In one possible implementation, the detecting whether the ir-coded list can control the appliance includes: detecting whether at least one infrared control code in the infrared code list can control the electric appliance; wherein the content of the first and second substances,

the mode for detecting whether the infrared control code can control the electric appliance comprises at least one of the following modes:

after a first infrared signal corresponding to the infrared control code is emitted, a state image of the electric appliance is shot, and the state change condition of the electric appliance is determined by using the state image; judging whether the state change condition of the electric appliance is consistent with a control instruction corresponding to the infrared control code, and if so, determining that the infrared control code can control the electric appliance;

after a first infrared signal corresponding to the infrared control code is transmitted, picking up an audio signal, and determining the state change condition of the electric appliance by using the audio signal; and judging whether the state change condition of the electric appliance is consistent with the control instruction corresponding to the infrared control code, and if so, determining that the infrared control code can control the electric appliance.

In one possible implementation manner, the method further includes:

receiving a control instruction for the electric appliance from the terminal equipment; or receiving a voice instruction aiming at the electric appliance, and identifying a control instruction corresponding to the voice instruction by adopting a voice identification technology;

searching the corresponding relation and determining an infrared coding list capable of controlling the electric appliance;

searching an infrared control code corresponding to the control instruction from the infrared code list capable of controlling the electric appliance;

and transmitting a second infrared signal corresponding to the infrared control code corresponding to the control instruction by using the infrared transmitting device.

In one possible implementation manner, the method further includes:

judging whether the electric appliance is controlled by the second infrared signal;

if not, a second target image is taken;

identifying the electric appliance in the second target image and acquiring the position of the electric appliance in the second target image;

re-determining the offset angle of the electric appliance relative to the camera device according to the position of the electric appliance in the second target image and the camera coordinate system of the camera device when the camera device takes the second target image;

adjusting the emission direction of the infrared emission device according to the redetermined offset angle;

and re-transmitting the second infrared signal by using the infrared transmitting device.

In a second aspect, the present disclosure provides a camera device for controlling an electric appliance, the camera device being mounted with an infrared emitting device, the camera device comprising:

the camera module is used for shooting a first target image;

the identification module is used for identifying the electric appliance in the first target image and acquiring the position of the electric appliance in the first target image;

the angle determining module is used for determining the offset angle of the electric appliance relative to the camera device according to the position of the electric appliance in the first target image and a camera coordinate system of the camera device when the camera device takes the first target image;

the emission direction adjusting module can be used for adjusting the emission direction of the infrared emission device according to the offset angle;

the recording module is used for searching at least one infrared code list corresponding to the electric appliance, wherein each infrared code list comprises at least one infrared control code; aiming at each infrared code list, the infrared emission device is adopted to emit a first infrared signal corresponding to at least one infrared control code in the infrared code list so as to detect whether the infrared code list can control the electric appliance; and under the condition that the infrared code list capable of controlling the electric appliance is detected, recording the corresponding relation between the infrared code list capable of controlling the electric appliance and the electric appliance, and recording the offset angle of the electric appliance relative to the camera device.

In one possible implementation, the list of ir codes that can control the appliance includes:

at least part of the infrared control codes can control the infrared code list of the electric appliance.

In one possible implementation, the recording module includes:

a determination submodule for extracting the image containing the appliance from the first target image; inputting an image containing the electric appliance into a pre-trained image recognition model, and outputting the brand and/or model of the electric appliance by the image recognition model; and searching a plurality of pre-stored infrared code lists according to the brand and/or model of the electric appliance, and determining at least one infrared code list corresponding to the brand and/or model of the electric appliance.

In one possible implementation, the identification module includes:

the input sub-module is used for inputting the first target image into a pre-trained target detection model, and outputting the type of an electric appliance contained in the first target image, the position of the electric appliance in the first target image and the confidence coefficient by the target detection model;

and the recording sub-module is used for recording the type of the electric appliance and the position of the electric appliance in the first target image under the condition that the confidence coefficient is greater than a preset threshold value.

In one possible implementation, the recording module includes:

the detection submodule is used for detecting whether at least one infrared control code in the infrared code list can control the electric appliance; wherein the mode for detecting whether the infrared control code can control the electric appliance comprises at least one of the following modes: after a first infrared signal corresponding to the infrared control code is emitted, a state image of the electric appliance is shot, and the state change condition of the electric appliance is determined by using the state image; judging whether the state change condition of the electric appliance is consistent with a control instruction corresponding to the infrared control code, and if so, determining that the infrared control code can control the electric appliance; or after the first infrared signal corresponding to the infrared control code is transmitted, picking up an audio signal, and determining the state change condition of the electric appliance by using the audio signal; and judging whether the state change condition of the electric appliance is consistent with the control instruction corresponding to the infrared control code, and if so, determining that the infrared control code can control the electric appliance.

In one possible implementation manner, the method further includes:

the control module is used for receiving a control instruction aiming at the electric appliance from the terminal equipment; or receiving a voice instruction aiming at the electric appliance, and identifying a control instruction corresponding to the voice instruction by adopting a voice identification technology; searching the corresponding relation and determining an infrared coding list capable of controlling the electric appliance; searching an infrared control code corresponding to the control instruction from the infrared code list capable of controlling the electric appliance; adjusting the emission direction of the infrared emission device according to the offset angle; and transmitting a second infrared signal corresponding to the infrared control code corresponding to the control instruction by using the infrared transmitting device.

In one possible implementation manner, the method further includes:

the rechecking module is used for judging whether the electric appliance is controlled by the second infrared signal; if not, a second target image is taken; identifying the electric appliance in the second target image and acquiring the position of the electric appliance in the second target image; re-determining the offset angle of the electric appliance relative to the camera device according to the position of the electric appliance in the second target image and the camera coordinate system of the camera device when the camera device takes the second target image; adjusting the emission direction of the infrared emission device according to the redetermined offset angle; and re-transmitting the second infrared signal by using the infrared transmitting device.

In a third aspect, the present disclosure provides an electronic device, comprising:

one or more processors;

a memory communicatively coupled to the one or more processors;

one or more computer programs, wherein the one or more computer programs are stored in the memory, which when executed by the electronic device, cause the electronic device to perform the method provided by the first aspect above.

In a fourth aspect, the present disclosure provides a computer-readable storage medium storing computer instructions that, when executed on a computer, cause the computer to perform the method provided by the first aspect.

The technical scheme provided by the disclosure at least comprises the following beneficial effects:

the control of the electric appliance is realized by adopting the camera device provided with the infrared transmitting device. Firstly, identifying an electric appliance from an image shot by a camera device, and determining the offset angle of the electric appliance relative to the camera device so as to determine the emission direction of the infrared emission device when the infrared emission device controls the electric appliance; and then, a plurality of infrared code lists corresponding to the electric appliance are tried, and the infrared code list capable of controlling the electric appliance is found and recorded. Therefore, the electric appliance is controlled by the camera device provided with the infrared transmitting device subsequently, the electric appliance is conveniently controlled, and the user experience is improved.

Other features and aspects of the present disclosure will become apparent from the following detailed description of the image pickup apparatus of the exemplary embodiments with reference to the accompanying drawings.

Drawings

The drawings are included to provide a better understanding of the present solution and are not to be construed as limiting the present disclosure. Wherein:

FIG. 1 is a flow chart of an implementation of an appliance control method according to an embodiment of the present disclosure;

fig. 2 is a flowchart illustrating an implementation of searching at least one infrared code list corresponding to an electrical appliance in an electrical appliance control method according to an embodiment of the present disclosure;

FIG. 3 is a flow chart of an implementation of identifying an appliance in a first target image using a target detection model and obtaining a location of the appliance in the first target image;

FIG. 4 is a diagram illustrating a relationship between an image coordinate system and a pixel coordinate system;

FIG. 5 is a diagram illustrating the relationship between an image coordinate system and a camera coordinate system;

FIG. 6 is a schematic diagram illustrating an implementation of controlling an electrical appliance according to a control command in an embodiment of the present disclosure;

fig. 7 is a first schematic structural diagram of an image capturing apparatus according to an embodiment of the present disclosure;

fig. 8 is a schematic structural diagram of a second image pickup apparatus according to an embodiment of the present disclosure;

fig. 9 is a block diagram of an electronic device according to an embodiment of the invention.

Detailed Description

The present disclosure will be described in further detail below with reference to the accompanying drawings. In the drawings, like reference numbers can indicate functionally identical or similar elements. While the various aspects of the embodiments are presented in drawings, the drawings are not necessarily drawn to scale unless specifically indicated.

Furthermore, in the following detailed description, numerous specific details are set forth in order to provide a better understanding of the present disclosure. It will be understood by those skilled in the art that the present disclosure may be practiced without some of these specific details. In some instances, methods, means, elements, circuits, etc., that are well known to those skilled in the art have not been described in detail so as not to obscure the present disclosure.

Fig. 1 is a flowchart illustrating an implementation of an appliance control method according to an embodiment of the present disclosure. This embodiment can be applied to an image pickup apparatus provided with an infrared emitting apparatus. As shown in fig. 1, the implementation process of this embodiment includes:

step 101: capturing a first target image;

step 102: identifying the electric appliance in the first target image, and acquiring the position of the electric appliance in the first target image;

step 103: determining the offset angle of the electric appliance relative to the camera device according to the position of the electric appliance in the first target image and the camera coordinate system of the camera device when the camera device takes the first target image;

step 104: adjusting the emission direction of the infrared emission device according to the offset angle;

step 105: searching at least one infrared code list corresponding to the electric appliance, wherein each infrared code list comprises at least one infrared control code; aiming at each infrared code list, the infrared transmitting device is adopted to transmit a first infrared signal corresponding to at least one infrared control code in the infrared code list so as to detect whether the infrared code list can control the electric appliance;

step 106: and under the condition that the infrared code list capable of controlling the electric appliance is detected, recording the corresponding relation between the infrared code list capable of controlling the electric appliance and the electric appliance, and recording the offset angle of the electric appliance relative to the camera device.

In the above process, a self-learning process of the image pickup apparatus provided with the infrared emission device (hereinafter referred to as an image pickup apparatus) is realized.

Specifically, in steps 101 to 103, the electrical appliance is identified, and the offset angle of the electrical appliance relative to the camera device is determined, so that it is ensured that the infrared signal emitted by the infrared emitting device can be received by the corresponding electrical appliance in the subsequent control. For example, in step 102, 3 electrical appliances are identified in the first target image, wherein the electrical appliance 1 is a television, the electrical appliance 2 is an air conditioner, the electrical appliance 3 is an audio device, and the "television", "air conditioner" and "audio device" are types of the electrical appliance 1, the electrical appliance 2 and the electrical appliance 3, respectively. For the sake of distinction, in the following description, the infrared signal used to control the electric appliance after the self-learning process is completed is referred to as a second infrared signal to distinguish from the first infrared signal in step 105. The foregoing nomenclature does not imply any sequential limitation of the two infrared signals.

Moreover, since one electrical appliance type may correspond to a plurality of infrared code lists, in order to determine the infrared code list capable of controlling the electrical appliance, in step 105, each infrared code list is detected respectively to determine the infrared code list capable of controlling the electrical appliance, thereby ensuring that the infrared signal emitted by the infrared emission device can control the corresponding electrical appliance during subsequent control.

The infrared code list capable of controlling the electric appliance may include: at least part of the infrared control codes can control the infrared code list of the electric appliance. That is, for a certain electrical appliance, if at least some infrared control codes included in one infrared code list can control the electrical appliance, the infrared code list is an infrared code list capable of controlling the electrical appliance. For some appliances, there may be multiple lists of infrared codes that can control the appliance; for this situation, after determining one of the ir codes, the present disclosure may terminate the detection process of the remaining ir code list corresponding to the electrical appliance in step 105; alternatively, after detecting a plurality of infrared code lists capable of controlling the electric appliance, one of the infrared code lists can be selected and recorded.

Taking a television as an example, the control command for the television may include turning on, turning off, tuning to a next channel, tuning to a previous channel, tuning to a specific channel, entering a specific mode, exiting the specific mode, starting recording, ending recording, turning up the volume, turning down the volume, and the like. Each control instruction corresponds to an infrared control code, and the television can be controlled to execute the corresponding control instruction by transmitting an infrared signal corresponding to the infrared control code to the television. For televisions of different brands and models, infrared control codes corresponding to the same control instruction are likely to be different; therefore, the televisions of different brands and/or models respectively have corresponding infrared code lists, the infrared code lists comprise a plurality of infrared control codes, and each infrared control code can correspond to a specific control instruction. The foregoing is also applicable to other types of electrical appliances and will not be described herein. As can be seen from the foregoing, for an appliance type, the corresponding ir code list may be large in number. In order to improve the efficiency of searching the infrared coding list capable of controlling a single electric appliance, the infrared coding list corresponding to the brand and/or the model of the electric appliance can be determined firstly by further combining the brand and/or the model of the electric appliance, and then the infrared coding list capable of controlling the electric appliance is finally determined by trying one by one. As shown in fig. 2:

fig. 2 is a flowchart illustrating an implementation of searching for at least one infrared code list corresponding to an electrical appliance in an electrical appliance control method according to an embodiment of the present disclosure, where the flowchart includes:

step 201: extracting an image containing an electric appliance from the first target image;

step 202: inputting the image containing the electric appliance into a pre-trained image recognition model, and outputting the brand and/or model of the electric appliance by the image recognition model;

step 203: and searching a plurality of pre-stored infrared code lists according to the brand and/or model of the electric appliance, and determining at least one infrared code list corresponding to the brand and/or model of the electric appliance.

It can be seen that, for one electric appliance type, since the infrared code lists corresponding to different brands and/or models of the electric appliance type are pre-stored, when the infrared code list of a certain electric appliance is determined to be controlled, preliminary screening can be performed according to the brands and/or models of the electric appliance, and then the infrared code list capable of controlling the electric appliance is detected from the infrared code list obtained after the preliminary screening, so that the implementation efficiency of the whole detection process is improved.

In some embodiments, the image recognition model in step 202 may adopt a Convolutional Neural Network (CNN) model. For different types of electrical appliances, corresponding convolutional neural network models can be trained in advance and used for identifying specific brands and/or models of the types of electrical appliances.

In some embodiments, the present disclosure may employ a pre-trained target detection model, identify an appliance in a first target image captured by a camera, and obtain a location of the appliance in the first target image. For example, the target detection model may specifically adopt a YOLO (You Only Look Once) v3 model or a YOLOv5 model. The YOLO model includes a convolutional layer, a pooling layer, and two fully-connected layers. The output layer predicts not only the confidence that an object (object) in the target image belongs to each appliance type, but also the position of the object in the target image, which is represented by a bounding box, using a linear function as an activation function. The YOLO model output data may take the form:

OutPut=（x, y, w, h, p1, p2, p3…）

wherein x represents the abscissa of the center point of the bounding box;

y represents the ordinate of the center point of the bounding box;

w represents the width of the bounding box;

h represents the height of the bounding box;

above, x, y, w, h define the position of the object in the target image.

p1, p2, p3, etc. respectively represent the confidence of predicting that the object belongs to category 1, category 2, category 3, etc. In particular to the target detection model for identifying appliances of the present disclosure, p1, p2, p3, etc. may respectively represent the confidence that objects in the target image belong to different appliance types.

Fig. 3 is a flowchart of an implementation of recognizing an electrical appliance in a first target image by using a target detection model and acquiring a position of the electrical appliance in the first target image, including:

step 301: inputting a first target image into a pre-trained target detection model, and outputting the type of an electric appliance contained in the first target image, the position of the electric appliance in the first target image and a confidence coefficient by using the target detection model;

step 302: and recording the type of the electric appliance and the position of the electric appliance in the first target image under the condition that the confidence coefficient is greater than a preset threshold value.

Using the YOLOv3 or YOLOv5 model, the individual appliances contained in the first target image, the types of appliances to which the individual appliances belong, and the locations of the individual appliances in the first target image can be identified. For example, inputting the first target image into YOLOv3 model or YOLOv5 model, YOLOv3 model or YOLOv5 model outputs a plurality of output results of the above form; one of the output results is: out put1= (100, 100, 80, 50, 0.9, 0.2, 0.1 …), where data 0.9 represents a confidence that the object is predicted to be a television of 0.9. If the preset threshold is 0.8, since the confidence level 0.9 is greater than the preset threshold, it may be determined that the object is an appliance, the type of the appliance to which the object belongs is a television, and it may be determined that the object is in the bounding box defined by (100, 100, 80, 50). The target detection model can output a plurality of output results in a similar form at one time, so as to determine the positions and the types of a plurality of electrical appliances in the first target image.

According to the recognized position of the electric appliance in the first target image and the camera coordinate system of the camera device when the camera device takes the first target image, the offset angle of the electric appliance relative to the camera device can be determined, namely the position of the electric appliance in the real world is determined, so that the camera device can be accurately positioned to the electric appliance when the camera device subsequently controls the electric appliance.

For example, the center point of the bounding box of an appliance is taken as the location of the appliance. The abscissa and ordinate of the center point of the bounding box output by the YOLOv3 model or the YOLOv5 model are generally expressed in pixels. FIG. 4 is a diagram illustrating a relationship between an image coordinate system and a pixel coordinate system. As shown in fig. 4, a rectangular coordinate system (U-V coordinate system) is established with the upper left corner of the image as the origin, also called pixel coordinate system, where the abscissa U and the ordinate V of a pixel are the number of columns and rows in the image array, respectively. In a pixel coordinate system, the coordinates (u, v) of a point represent the column number and row number of the pixel where the point is located, and the position of the pixel in the image is not expressed in physical units. Therefore, an image coordinate system (X-Y coordinate system) expressed in physical units such as millimeters is established with the center point of the image as the origin of the image coordinate system, and the X-axis is parallel to the u-axis and the Y-axis is parallel to the v-axis. The origin of the image coordinate system is also the intersection of the camera optical axis and the image plane.

As can be seen from the relationship between the two coordinate systems in fig. 4, the coordinates of the electrical appliance in the image coordinate system can be determined according to the coordinates of the point representing the position of the electrical appliance (i.e. the center point of the boundary frame corresponding to the electrical appliance) in the pixel coordinate system.

And further determining the position of the electric appliance in the real world by combining with a camera coordinate system of the camera device when the first target image is shot. FIG. 5 is a diagram illustrating a relationship between an image coordinate system and a camera coordinate system. As shown in FIG. 5, the image coordinate system is a plane coordinate system (X-Y coordinate system), the camera coordinate system is a three-dimensional coordinate system (Xc-Yc-Zc coordinate system), the ZC axis of the camera coordinate system is perpendicular to the plane of the plane coordinate system, the point O is the position of the image capturing device, the line segment | OO1| is the focal length of the image capturing device and is recorded as the focal length of the image capturing device

. For a certain point m in the image coordinate system (m:)

) The position of the real object in the camera coordinate system is M: (

). According to the imaging projection relationship, the foregoing parameters have the following relationship:

（1）

（2）

from the principles of solid geometry, the offset angle of the appliance (position point M) with respect to the camera (position point O) can be determined, namely: the offset angles of the point M with respect to the camera coordinate system Xc axis, Yc axis, and Zc axis of the imaging device are:

、

and

. By combining the above equations (1) and (2), the offset angle of M with respect to the Xc axis, Yc axis, and Zc axis of the camera coordinate system of the imaging device can be calculated, that is, the offset angle of the electronic device with respect to the imaging device can be specified.

Alternatively, the offset angle of the appliance (position M) relative to the camera coordinate system of the camera device (position O) can be determined in combination with parameters of the camera device (such as internal parameters) and coordinates of the appliance image (position M) in the image coordinate system. For example, the left-right view angle and the up-down view angle of the image pickup apparatus can be determined from internal references of the image pickup apparatus. Assume that the determined left and right viewing angles are (-90 degrees, +90 degrees). As shown in fig. 5, the abscissa of the m point in the image is x, and the left view angle ratio can be obtained by dividing x by half of the image width; multiplying this ratio by the angle of view +90 degrees yields the offset angle of M with respect to the Xc axis of the camera coordinate system of the image capture device, i.e.:

the offset angle of M with respect to the Xc axis = 90 degrees half x/image width.

Similarly, assuming that the determined up-down viewing angle is (-90 degrees, +90 degrees), then:

m is offset from the Yc axis by an angle = y/half of the image height by 90 degrees.

After the offset angle of the electric appliance relative to the camera coordinate system of the camera device is determined, whether the infrared control codes included in each infrared code list corresponding to the type, brand and/or signal of the electric appliance can control the electric appliance can be detected respectively, namely, the infrared signals are transmitted to the electric appliance according to the infrared control codes, and whether the infrared control codes can control the electric appliance is detected. Specifically, detecting whether the infrared code list can control the electric appliance comprises the following steps: detecting whether at least one infrared control code in the infrared code list can control the electric appliance; wherein the mode for detecting whether the infrared control code can control the electric appliance comprises at least one of the following modes

After a first infrared signal corresponding to an infrared control code is emitted, a state image of the electric appliance is shot, and the state change condition of the electric appliance is determined by using the state image; judging whether the state change condition of the electric appliance is consistent with a control instruction corresponding to the infrared control code, and if so, determining that the infrared control code can control the electric appliance; and/or the presence of a gas in the gas,

after a first infrared signal corresponding to the infrared control code is emitted, picking up an audio signal, and determining the state change condition of the electric appliance by using the audio signal; and judging whether the state change condition of the electric appliance is consistent with the control instruction corresponding to the infrared control code, and if so, determining that the infrared control code can control the electric appliance.

For example, for an infrared control code of a television, if a control instruction corresponding to the infrared control code is to adjust to a subsequent channel, the infrared control code may be transmitted, and then a status image including the television is captured, a channel number in the status image is identified, and the channel number is compared with a channel number of the television before receiving the infrared signal, so as to determine whether the television has performed channel adjustment according to the infrared control code. The present disclosure may identify information such as channel numbers using a pre-trained image recognition model.

For another example, if the control instruction corresponding to the infrared control code is to increase the volume, the audio signal may be picked up after the infrared signal is transmitted, the volume of the audio signal is determined, and then the audio signal is compared with the volume before the television receives the infrared signal, so as to determine whether the television has performed volume adjustment according to the infrared control code.

The foregoing introduces a process of initializing the camera device, in which the camera device identifies the type and position of each electrical appliance, detects an infrared code list capable of controlling each electrical appliance, and records a correspondence relationship between the electrical appliance and the infrared code list capable of controlling the electrical appliance. On the basis, the camera device can adopt the corresponding infrared control codes in the infrared code list to realize the automatic control of the electric appliance according to the instruction of the user.

Fig. 6 is a schematic diagram illustrating an implementation manner of controlling an electrical appliance according to a control instruction in an embodiment of the present disclosure, including:

step 601: receiving a control instruction for the electric appliance from the terminal equipment; or receiving a voice instruction aiming at the electric appliance, and identifying a control instruction corresponding to the voice instruction by adopting a voice identification technology;

step 602: searching the corresponding relation between the electric appliance and the infrared coding list capable of controlling the electric appliance, and determining the infrared coding list capable of controlling the electric appliance;

step 603: searching an infrared control code corresponding to the control instruction from an infrared code list capable of controlling the electric appliance;

step 604: adjusting the emission direction of the infrared emission device according to the offset angle of the electric appliance relative to the camera device;

step 605: and transmitting a second infrared signal corresponding to the infrared control code corresponding to the control instruction by using the infrared transmitting device.

In step 601, the control instruction for the electrical appliance is received from the terminal device, and may include a control instruction for a specific electrical appliance sent by a user through an application program (APP) of the terminal device. The voice command in step 601 may be issued by the user, and the voice command at least includes an appliance name and a command name, such as "turn on the television".

The correspondence relationship in step 602 may include a correspondence relationship between each electrical appliance stored in the imaging device during the initialization process and the infrared code list capable of controlling the corresponding electrical appliance. And aiming at the electric appliance related to the control instruction, determining an infrared code list capable of controlling the electric appliance by searching the corresponding relation.

After transmitting the control signal, i.e., the second infrared signal in step 605, the present disclosure may further determine whether the appliance is controlled, and make necessary adjustments without being controlled. In general, the reason why the electric appliance is not controlled may be that the electric appliance is subjected to a positional change after the initialization of the image pickup device, so that the second infrared signal emitted from the image pickup device at the initially recorded offset angle cannot be received by the electric appliance. In view of this, the present disclosure also proposes further adjustment schemes. Referring to fig. 6, the step 605 may further include:

step 606: judging whether the electric appliance is controlled by the second infrared signal; if not, step 607 is performed:

step 607: capturing a second target image;

step 608: identifying the electrical appliance in the second target image, and acquiring the position of the electrical appliance in the second target image;

step 609: re-determining the offset angle of the electric appliance relative to the camera device according to the position of the electric appliance in the second target image and the camera coordinate system of the camera device when the second target image is shot;

step 610: adjusting the emission direction of the infrared emission device according to the redetermined offset angle;

step 611: and re-transmitting the second infrared signal by using the infrared transmitting device.

The determination manner in step 606 may be the same as the detection manner in the initialization process, that is, the state change of the electrical appliance after receiving the second infrared signal is determined by image recognition, volume recognition, and the like, so as to determine whether the electrical appliance is controlled by the second infrared signal.

Since the position of the electrical appliance may have changed, in step 607, the image capturing device may capture a plurality of second target images at different capturing angles, and perform target detection on the plurality of second target images, so as to determine the position of the electrical appliance in the second target image. The target detection method may be the same as the target detection method in the initialization process, for example, a previously trained YOLOv3 model or YOLOv5 model is used for recognition, and details are not repeated here. It should be noted that the above-mentioned "second target image" is named in order to distinguish from the aforementioned "first target image", and there is no order relationship between the two.

The manner of determining the offset angle in step 609 is the same as that of determining the offset angle in the initialization process, and is not described herein again.

Through the above steps 606 to 611, the camera device re-determines the offset angle of the electrical appliance relative to the camera device, and transmits a second infrared signal to the electrical appliance to control the electrical appliance to adjust the state according to the instruction of the user. Furthermore, the camera device can update the offset angle of the electric appliance, namely record the latest offset angle of the electric appliance; and then when the control instruction is received again, adjusting the emission direction of the infrared emission device according to the updated offset angle.

In summary, the electrical appliance control method provided by the disclosure realizes control of an electrical appliance by adopting the camera device provided with the infrared emission device, and each infrared-controllable household electrical appliance in a room can be fully automatically configured only by installing the camera device in the room. Because install infrared light filling lamp among the camera device usually, this disclosure can regard as the infrared emitter who is used for controlling electrical apparatus with this infrared light filling lamp for the infrared signal that the transmission infrared control code corresponds, consequently need not to change current camera device's hardware architecture.

The utility model also provides a camera device, this camera device installs infrared emission device, can adopt this camera device's infrared light filling lamp as infrared emission device. Fig. 7 is a schematic structural diagram of an image capturing apparatus according to an embodiment of the present disclosure, which includes:

a camera module 710 for capturing a first target image;

the identification module 720 is configured to identify an electrical appliance in the first target image, and acquire a position of the electrical appliance in the first target image;

an angle determining module 730, configured to determine an offset angle of the electrical appliance with respect to the camera device according to a position of the electrical appliance in the first target image and a camera coordinate system of the camera device when the camera device captures the first target image;

a transmission direction adjusting module 740, configured to adjust a transmission direction of the infrared transmitter according to the offset angle;

the recording module 750 is configured to search at least one infrared code list corresponding to the electrical appliance, where each infrared code list includes at least one infrared control code; aiming at each infrared code list, the infrared emission device is adopted to emit a first infrared signal corresponding to at least one infrared control code in the infrared code list so as to detect whether the infrared code list can control the electric appliance; and under the condition that the infrared code list capable of controlling the electric appliance is detected, recording the corresponding relation between the infrared code list capable of controlling the electric appliance and the electric appliance, and recording the offset angle of the electric appliance relative to the camera device.

In a possible embodiment, the list of infrared codes capable of controlling the electrical appliance includes: at least part of the infrared control codes can control the infrared code list of the electric appliance.

Fig. 8 is a schematic structural diagram of an image capturing apparatus according to an embodiment of the present disclosure, and as shown in fig. 8, in a possible implementation, the recording module 750 includes:

a determination submodule 751 for extracting the image containing the appliance from the first target image; inputting an image containing the electric appliance into a pre-trained image recognition model, and outputting the brand and/or model of the electric appliance by the image recognition model; and searching a plurality of pre-stored infrared code lists according to the brand and/or model of the electric appliance, and determining at least one infrared code list corresponding to the brand and/or model of the electric appliance.

As shown in fig. 8, in a possible implementation, the identifying module 720 includes:

the input sub-module 721 is configured to input the first target image into a pre-trained target detection model, and output, by the target detection model, the type of an electrical appliance included in the first target image, and a position and a confidence of the electrical appliance in the first target image;

the recording sub-module 722 is configured to record the type of the electrical apparatus and the position of the electrical apparatus in the first target image if the confidence is greater than a preset threshold.

As shown in fig. 8, in a possible implementation, the recording module 750 includes:

the detection submodule 752 is configured to detect whether at least one infrared control code in the infrared code list can control the electrical appliance; wherein the mode for detecting whether the infrared control code can control the electric appliance comprises at least one of the following modes: after a first infrared signal corresponding to the infrared control code is emitted, a state image of the electric appliance is shot, and the state change condition of the electric appliance is determined by using the state image; judging whether the state change condition of the electric appliance is consistent with a control instruction corresponding to the infrared control code, and if so, determining that the infrared control code can control the electric appliance; or after the first infrared signal corresponding to the infrared control code is transmitted, picking up an audio signal, and determining the state change condition of the electric appliance by using the audio signal; and judging whether the state change condition of the electric appliance is consistent with the control instruction corresponding to the infrared control code, and if so, determining that the infrared control code can control the electric appliance.

As shown in fig. 8, in one possible embodiment, the image capturing apparatus further includes:

a control module 860 for receiving a control instruction for the appliance from the terminal device; or receiving a voice instruction aiming at the electric appliance, and identifying a control instruction corresponding to the voice instruction by adopting a voice identification technology; searching the corresponding relation and determining an infrared coding list capable of controlling the electric appliance; searching an infrared control code corresponding to the control instruction from the infrared code list capable of controlling the electric appliance; adjusting the emission direction of the infrared emission device according to the offset angle; and transmitting a second infrared signal corresponding to the infrared control code corresponding to the control instruction by using the infrared transmitting device.

a rechecking module 870 for judging whether the electrical appliance is controlled by the second infrared signal; if not, a second target image is taken; identifying the electric appliance in the second target image and acquiring the position of the electric appliance in the second target image; re-determining the offset angle of the electric appliance relative to the camera device according to the position of the electric appliance in the second target image and the camera coordinate system of the camera device when the camera device takes the second target image; adjusting the emission direction of the infrared emission device according to the redetermined offset angle; and re-transmitting the second infrared signal by using the infrared transmitting device.

It should be noted that, in the embodiment of the present disclosure, the division of each functional unit is schematic, and is only one logical functional division, and there may be another division manner in actual implementation. Each functional unit in the embodiments of the present disclosure may be integrated into one processing unit, each unit may exist alone physically, or two or more units are integrated into one unit. The integrated unit can be realized in a form of hardware, and can also be realized in a form of a software functional unit.

The integrated unit, if implemented in the form of a software functional unit and sold or used as a stand-alone product, may be stored in a computer readable storage medium. Based on such understanding, the technical solution of the present disclosure may be embodied in the form of a software product, which is stored in a storage medium and includes several instructions for causing a computer device (which may be a personal computer, a server, a network device, or the like) or a processor (processor) to execute all or part of the steps of the method provided by the embodiments of the present disclosure. And the aforementioned storage medium includes: u disk, removable hard disk, read only memory, random access memory, magnetic or optical disk, etc. for storing program codes.

Fig. 9 is a block diagram of an electronic device according to an embodiment of the invention. As shown in fig. 9, the electronic apparatus includes: a memory 910 and a processor 920, the memory 910 having stored therein computer programs operable on the processor 920. The number of the memory 910 and the processor 920 may be one or more. The memory 910 may store one or more computer programs that, when executed by the electronic device, cause the electronic device to perform the methods provided by the above-described method embodiments.

The electronic device further includes:

and a communication interface 930 for communicating with an external device to perform data interactive transmission.

If the memory 910, the processor 920 and the communication interface 930 are implemented independently, the memory 910, the processor 920 and the communication interface 930 may be connected to each other through a bus and perform communication with each other. The bus may be an Industry Standard Architecture (ISA) bus, a Peripheral Component Interconnect (PCI) bus, an Extended ISA (EISA) bus, or the like. The bus may be divided into an address bus, a data bus, a control bus, etc. For ease of illustration, only one thick line is shown in FIG. 9, but this does not indicate only one bus or one type of bus.

Optionally, in an implementation, if the memory 910, the processor 920 and the communication interface 930 are integrated on a chip, the memory 910, the processor 920 and the communication interface 930 may complete communication with each other through an internal interface.

The embodiment of the present disclosure also provides a computer-readable storage medium, which stores computer instructions, and when the computer instructions are run on a computer, the computer is caused to execute the method provided by the above method embodiment.

The embodiment of the present disclosure further provides a computer program product, where the computer program product is used to store a computer program, and when the computer program is executed by a computer, the computer may implement the method provided by the above method embodiment.

The embodiment of the disclosure also provides a chip, which is coupled with the memory, and is used for implementing the method provided by the embodiment of the method.

It should be understood that the processor may be a Central Processing Unit (CPU), other general purpose processor, a Digital Signal Processor (DSP), an Application Specific Integrated Circuit (ASIC), a Field Programmable Gate Array (FPGA) or other Programmable logic device, discrete Gate or transistor logic, discrete hardware components, etc. A general purpose processor may be a microprocessor or any conventional processor or the like. It is noted that the processor may be a processor supporting an Advanced reduced instruction set machine (ARM) architecture.

Further, optionally, the memory may include a read-only memory and a random access memory, and may further include a nonvolatile random access memory. The memory may be either volatile memory or nonvolatile memory, or may include both volatile and nonvolatile memory. The non-volatile Memory may include a Read-Only Memory (ROM), a Programmable ROM (PROM), an Erasable PROM (EPROM), an Electrically Erasable PROM (EEPROM), or a flash Memory. Volatile Memory can include Random Access Memory (RAM), which acts as external cache Memory. By way of example, and not limitation, many forms of RAM are available. For example, Static Random Access Memory (Static RAM, SRAM), Dynamic Random Access Memory (DRAM), Synchronous Dynamic Random Access Memory (SDRAM), Double Data rate Synchronous Dynamic Random Access Memory (DDR SDRAM), Enhanced SDRAM (ESDRAM), SLDRAM (SLDRAM), and Direct RAMBUS RAM (DR RAM).

In the above embodiments, the implementation may be wholly or partly realized by software, hardware, firmware, or any combination thereof. When implemented in software, may be implemented in whole or in part in the form of a computer program product. The computer program product includes one or more computer instructions. When loaded and executed on a computer, cause the processes or functions described in accordance with the embodiments of the disclosure to occur, in whole or in part. The computer may be a general purpose computer, a special purpose computer, a network of computers, or other programmable device. The computer instructions may be stored on a computer readable storage medium or transmitted from one computer readable storage medium to another, for example, from one website, computer, server, or data center to another website, computer, server, or data center via wire (e.g., coaxial cable, fiber optic, Digital Subscriber Line (DSL)) or wireless (e.g., infrared, bluetooth, microwave, etc.). The computer-readable storage medium can be any available medium that can be accessed by a computer or a data storage device, such as a server, a data center, etc., that includes one or more of the available media. The usable medium may be a magnetic medium (e.g., floppy Disk, hard Disk, magnetic tape), an optical medium (e.g., Digital Versatile Disk (DVD)), or a semiconductor medium (e.g., Solid State Disk (SSD)), among others. Notably, the computer-readable storage media referred to in this disclosure may be non-volatile storage media, in other words, non-transitory storage media.

It will be understood by those skilled in the art that all or part of the steps for implementing the above embodiments may be implemented by hardware, or may be implemented by a program instructing relevant hardware, where the program may be stored in a computer-readable storage medium, and the above-mentioned storage medium may be a read-only memory, a magnetic disk or an optical disk, etc.

In the description of the embodiments of the present disclosure, reference to the description of the term "one embodiment," "some embodiments," "an example," "a specific example," or "some examples," etc., means that a particular feature, structure, material, or characteristic described in connection with the embodiment or example is included in at least one embodiment or example of the present disclosure. Furthermore, the particular features, structures, materials, or characteristics described may be combined in any suitable manner in any one or more embodiments or examples. Furthermore, various embodiments or examples and features of different embodiments or examples described in this specification can be combined and combined by one skilled in the art without contradiction.

In the description of the embodiments of the present disclosure, "/" indicates an OR meaning, for example, A/B may indicate A or B; "and/or" herein is merely an association describing an associated object, and means that there may be three relationships, e.g., a and/or B, which may mean: a exists alone, A and B exist simultaneously, and B exists alone.

In the description of the embodiments of the present disclosure, the terms "first" and "second" are used for descriptive purposes only and are not to be construed as indicating or implying relative importance or implying any number of technical features indicated. Thus, a feature defined as "first" or "second" may explicitly or implicitly include one or more of that feature. In the description of the embodiments of the present disclosure, "a plurality" means two or more unless otherwise specified.

The above description is only exemplary of the present disclosure and is not intended to limit the present disclosure, and any modification, equivalent replacement, or improvement made within the spirit and principle of the present disclosure should be included in the scope of the present disclosure.

Claims

1. An electric appliance control method is applied to a camera device provided with an infrared emission device, and is characterized by comprising the following steps:

capturing a first target image;

2. The method of claim 1, wherein the ir-encoded list of controllable appliances comprises:

3. The method according to claim 1 or 2, wherein the searching for the at least one ir-coded list corresponding to the appliance comprises:

extracting the image containing the appliance from the first target image;

4. The method of claim 1 or 2, wherein the identifying the appliance in the first target image and obtaining the location of the appliance in the first target image comprises:

5. The method according to claim 1 or 2, wherein said detecting whether the ir-coded list can control the appliance comprises: detecting whether at least one infrared control code in the infrared code list can control the electric appliance; wherein the content of the first and second substances,

6. The method of claim 1 or 2, further comprising:

7. The method of claim 6, further comprising:

if not, a second target image is taken;

8. A camera device for controlling an electrical appliance, said camera device being equipped with an infrared emitting device, said camera device comprising:

the camera module is used for shooting a first target image;

9. The image capture device of claim 8, wherein the infrared encoded list of controllable appliances comprises:

10. The image pickup apparatus according to claim 8 or 9, wherein the recording module includes:

11. The image pickup apparatus according to claim 8 or 9, wherein the identification module includes:

12. The image pickup apparatus according to claim 8 or 9, wherein the recording module includes:

13. The image pickup apparatus according to claim 8 or 9, further comprising:

14. The image pickup apparatus according to claim 13, further comprising:

15. An electronic device, comprising:

one or more processors;

a memory communicatively coupled to the one or more processors;

one or more computer programs, wherein the one or more computer programs are stored in the memory, which when executed by the electronic device, cause the electronic device to perform the method of any of claims 1-7.

16. A computer-readable storage medium having stored thereon computer instructions which, when run on a computer, cause the computer to perform the method of any one of claims 1 to 7.