KR20180099260A - Techniques for enabling auto-configuration of infrared signaling for device control - Google Patents

Abstract

The present invention provides a method for automatic configuration of infrared signaling for device control. According to embodiments of the present invention, the method comprises: a step of acquiring an image of an environment from an image capturing unit; a step of extracting an object image from the image; a step of using a neural network to classify an electronic device positioned in the environment as a specific candidate among a plurality of candidates based on the object image; a step of acquiring a first image representing the classified electronic device; a step of transmitting an infrared code signal to instruct the specific candidate to allow an infrared transmitter to perform a specific operation to the electronic device in response to acquisition of the first image; a step of acquiring a second image representing the electronic device in response to transmission of the infrared code signal; and a step of determining whether the electronic device has performed the specific operation based on the first image and the second image.

Description

TECHNICAL FIELD [0001] The present invention relates to a technique for automatically setting infrared signaling for device control,

Embodiments of the present invention relate to techniques that enable auto-configuration of infrared signaling for device control.

Many consumer electronics devices that have recently been released can be used in the Internet of Things (IoT). Unfortunately, in order to establish an IoT environment, dedicated devices with wireless network capabilities need to be used, and many existing infrared (IR) -based devices already available in most homes have access to the IoT Can not. In order to overcome these disadvantages, there have been proposed technologies for remotely controlling an existing IR-based home appliance by attaching an IR transmitter to a personal device such as a cellular phone, a smartphone and the like. For example, Korean Utility Model No. 2000-0001257 discloses adding a key for selecting a remote control function to a cellular telephone capable of transmitting infrared rays. However, since these techniques require users to perform troublesome setups, many people find it difficult and inconvenient to use them. Moreover, in IoT, machine-to-machine control is more important than remote control by humans. Therefore, there is a need for techniques that can communicate with IR-based devices in an indoor environment, such as a home or office, without human assistance.

An embodiment of the present invention provides an apparatus, method and computer readable storage medium suitable for automatically setting IR signaling for controlling an IR enabled device.

According to an embodiment of the present invention, it is possible to use an artificial intelligence such as a neural network to recognize an object, transmit an infrared signal to a recognized object, and compare images before and after signal transmission It is possible to control by infrared signaling.

According to one embodiment of the present invention, there is provided a method of enabling automatic setting of infrared signaling for device control, comprising: acquiring an image of an environment from an image capture unit; Deriving an object image from the image; Using a neural network to classify electronic devices located in the environment as specific candidates among a plurality of candidates based on the object image; Obtaining a first image representing the classified electronic device; Causing an infrared transmitter to transmit to the electronic device an infrared code signal for instructing the specific candidate to perform a specific operation in response to the first image being obtained; Obtaining a second image representing the electronic device in response to the infrared code signal being transmitted; And making a determination as to whether the electronic device has performed the specific operation based on the first image and the second image.

And the step of making the determination may include determining that the electronic device has performed the specific operation when the difference between the first image and the second image is equal to or greater than the threshold value.

The specific operation may include an operation of switching on / off of the specific candidate.

The method includes causing the infrared transmitter to transmit, to the electronic device, another infrared code signal for instructing another candidate of the plurality of candidates to perform the specific operation, when it is determined that the specific operation has not been performed step; Obtaining a third image representing the electronic device in response to the other infrared code signal being transmitted; And determining whether the electronic device has performed the specific operation based on at least one of the first image and the second image and the third image.

Causing the infrared transmitter to transmit to the electronic device an additional infrared code signal for instructing the particular candidate to perform additional operation if the specific operation is determined to have been performed; And performing an additional determination as to whether the electronic device has performed the additional operation in response to the additional infrared code signal being transmitted.

The further operation may include an operation that results in a change in the display output of the particular candidate, wherein the step of making the further determination is to determine whether the appearance of a particular area on the surface of the electronic device has changed after the additional infrared code has been transmitted , Making the additional determination based on the degree of the change, or both, and the specific region may correspond to the specific candidate.

The method may further comprise recognizing the appearance from an image feature that represents the particular region, wherein the recognition may be performed separately before and after the additional infrared code is transmitted.

The method may further comprise sensing a sound from the environment, wherein the sensing may be performed separately before and after the additional infrared code is transmitted, and wherein the further action is a change in the audio output of the particular candidate Wherein performing the further determination comprises performing the additional determination based on whether the sound has changed after the additional infrared code has been transmitted, the degree of the change, or both. ≪ RTI ID = 0.0 > can do.

The method may further include, in response to the electronic device being classified as the specific candidate, causing the image capturing unit to be directed to the classified electronic device, wherein the step of acquiring the first image comprises: And acquiring the first image based on a plurality of image frames of the electronic device captured by the image capture unit.

According to another embodiment of the present invention there is provided a computer-readable storage medium having computer-executable instructions for causing the processor to perform the above-described method when executed by at least one processor.

According to another embodiment of the present invention, there is provided an apparatus for enabling automatic setting of infrared signaling for device control, comprising: an image processing unit configured to acquire an image of an environment from an image capturing unit and derive an object image from the image; A neural network based classifier configured to classify an electronic device located in the environment as a specific candidate among a plurality of candidates based on the object image; And a controller, wherein the image processing unit is further configured to obtain a first image representing the classified electronic device, and in response to the first image being obtained, the controller causes the infrared transmitter And to cause the electronic device to transmit an infrared code signal for instructing the specific candidate to perform a specific operation, wherein the image processing unit is configured to, in response to the infrared code signal being transmitted, And the controller is further configured to make a determination as to whether the electronic device has performed the specific operation based on the first image and the second image.

The controller may be configured to make the determination if the controller is configured to determine that the electronic device has performed the specific operation if the difference between the first image and the second image is greater than or equal to a threshold value.

The specific operation may include an operation of switching on / off of the specific candidate.

The controller is further configured to, when it is determined that the specific operation has not been performed, cause the infrared transmitter to transmit to the electronic device another infrared code signal for instructing another candidate of the plurality of candidates to perform the specific operation And the image processing unit may be further configured to obtain a third image representative of the electronic device in response to the other infrared code signal being transmitted, And determine whether the electronic device has performed the specific operation based on at least one of the images and the third image.

The controller may be further configured to cause the infrared transmitter to transmit to the electronic device an additional infrared code signal for instructing the particular candidate to perform additional operation when it is determined that the particular operation has been performed, The controller may also be configured to make an additional determination as to whether the electronic device has performed the additional operation in response to the additional infrared code signal being transmitted.

The additional operation may include an operation that results in a change in the display output of the particular candidate, wherein the controller is configured to make the further determination that the controller is on the surface of the electronic device after the additional infrared code is transmitted And may be configured to make the additional determination based on whether the appearance of the specific region has changed, the degree of the change, or both, and the specific region may correspond to the specific candidate.

The image processing unit may also be configured to recognize the appearance from an image feature that represents the specific area, wherein the recognition may be performed separately before and after the additional infrared code is transmitted.

The apparatus may further comprise a sound sensing unit configured to sense sound from the environment, wherein the sensing may be performed separately before and after the additional infrared code is transmitted, Wherein the controller is configured to make the further determination further comprises the controller determining whether the sound is changed after the additional infrared code is transmitted, the degree of the change, or both And to perform the additional determination based on the determination result.

Wherein the controller is further configured to cause the image capturing unit to be directed to the classified electronic device in response to the electronic device being classified as the specific candidate, and wherein the image processing unit is configured to acquire the first image The image processing unit may be configured to obtain the first image based on a plurality of image frames of the electronic device captured by the image capture unit.

The apparatus may further include the image capturing unit and / or the infrared transmitter.

The technique according to an embodiment of the present invention enables the automatic setting of infrared signaling for controlling infrared-capable devices to perform certain operations. Further, according to the embodiment of the present invention, machine-to-machine control is possible not only for IoT dedicated devices, but also for conventional infrared-capable devices.

Figure 1 is an overview of an exemplary environment in which the disclosed technique may be practiced.
2 is a block diagram of an exemplary device control device.
Figure 3 schematically shows a neural network used by a neural network based classifier according to some embodiments.
4 is a flow diagram of a process that may be used to automatically set up infrared signaling for device control in accordance with some embodiments.
5 is a flow diagram of an additional process that may be used to automatically set up infrared signaling for device control in accordance with some embodiments.

The present invention may have various embodiments, and specific embodiments are illustrated in the drawings and described in detail in the detailed description. It is to be understood, however, that the invention is not to be limited to the specific embodiments, but includes all modifications, equivalents, and alternatives falling within the spirit and scope of the invention. In addition, a detailed description of known technology may be omitted so as not to unnecessarily obscure the various aspects of the present invention.

Hereinafter, embodiments of the present invention will be described in detail with reference to the accompanying drawings.

Figure 1 is an overview of an exemplary environment in which the disclosed technique may be practiced. 1, device control apparatus 110 may include one or more electronic devices (e.g., infrared-capable devices 121, 123, 125, and optionally other networked devices) Devices 141, 143, and 145). In some embodiments, the aforementioned electronic device may include other devices not shown and / or may include more or fewer electronic devices than the electronic device shown in FIG.

In particular, the device control apparatus 110 can control the infrared-ray-capable devices 121, 123, and 125. To this end, the device control unit 110 may enable automatic setting of infrared signaling as described in detail herein. Examples of such infrared ray capable devices 121, 123 and 125 include a television (TV), a compact disc (CD) player, an MP3 player, a stereo system, a DVD (Digital Versatile Disc) , Consumer electronics such as video recorders, air conditioners, fans, and other infrared-capable devices.

Optionally, one or more networks 130 may be deployed in the exemplary environment 100. One or more networks 130 allow device control device 110 to connect to, communicate with, and control networked devices 141, 143, and 145. Examples of networked devices 141, 143, and 145 include devices having various sensors and / or computing and communication capabilities, such as electronic devices, lighting, consumer electronics, mobile devices, and the like. The network 130 may include a wireless network, such as a wireless local area network (WLAN), a wireless personal area network (WPAN), etc., a wired network and / or the Internet.

In some embodiments, device control device 110 may include at least one processor, a computer readable storage medium, and a communication interface. When executed by a processor, processor executable or computer executable instructions that cause a processor to perform the processes discussed herein may be stored in a computer-readable storage medium. For example, the processor may be a central processing unit (CPU), a digital signal processor (DSP), a graphics processing unit (GPU), one or more processor cores, a single-core processor, a dual-core processor, a multiple-core processor, a microprocessor, an integrated circuit (IC), an application- An ASIC) and / or any other suitable processor. Examples of computer-readable storage media include random access memory (e.g., static RAM (SRAM), dynamic random access memory (DRAM), other types of RAM, etc.), read only memory A memory, a ROM, an electrically erasable programmable read only memory (EEPROM), a flash memory, a CD-ROM, a DVD, a magnetic storage device, And the like. The communication interface may include an IR interface, and may further include a wireless network interface and / or a wired network interface. Further, the device control device 110 may include one or more image capturing units, such as image sensors, cameras, etc., for capturing images of the environment (e.g., environment 100). Device controller 110 may also include one or more sensing units that provide information associated with various aspects of the environment (e.g., environment 100), such as an audio sensor, an ambient light sensor, etc. . ≪ / RTI > The aforementioned components of the device control device 110 may be communicatively coupled to one another.

In some embodiments, the device control device 110 may take the form of, or be part of, a mobile robotic device (e.g., a robot cleaner). In this example, the mobile robot moves itself within an indoor environment (e.g., environment 100) such as a home or office, with the induction of an on-board navigation system, The device control apparatus 110 transmits an IR signal to the electronic device and transmits the IR signal to the electronic device in accordance with the transmitted IR signal. You can see if the device is working.

Referring now to FIG. 2, an exemplary implementation of device control device 110 is presented. It should be noted that the configurations described below are by way of example only and not by way of limitation. As shown, the device control apparatus 110 includes an image processing unit 206, a neural network based classifier 208, and a controller 210. In addition, the device control apparatus 110 may include an image capturing unit 202. In addition, the device control apparatus 110 may include a sensing unit 204 (e.g., a sound sensing unit including an audio sensor). Further, the device control apparatus 110 may include an infrared transmitter 212. [ Each of the image capturing unit 202, the sensing unit 204, the image processing unit 206, the neural network-based classifier 208 and the infrared transmitter 212 is coupled with the controller 210, Can be performed under the control of the controller (210).

The image capture unit 202 may capture an image of the exemplary environment 100. The image processing unit 206 may be coupled with the image capturing unit 202 to acquire an image captured from the image capturing unit 202. [ The image processing unit 206 may derive an object image from the acquired image. In some embodiments, the device control device 110 may be implemented in a mobile robotic device that is capable of running in an environment 100, such as a home or office, as described above. The mobile robot can recognize the structure of the environment 100 and construct mapping information using the image capturing unit 202 and / or the sensing unit 204 while moving around the environment 100. In order to recognize an electronic device, such as an infrared-capable device 121, 123, 125, located within the environment 100 in this mapping process, the image processing unit 206 may generate an image of the object Can be derived.

As an example, the image processing unit 206 may acquire an object image through a series of processing operations as follows. This object image may represent an electronic device located within the environment 100. First, an image of the environment 100 may be converted to a monochrome image (e.g., a gray scale image having an appropriate bit depth) (e.g., as needed). Next, the noise can be removed from the monochrome image. For example, a filter such as a Gaussian filter, a median filter, or the like may be used for removing noise. Then, an edge can be detected from the processed image. For example, for such edge detection, a mask or an operator such as a Sobel mask, a Prewitt mask, a Robert mask, a Laplacian operator, or the like may be used. Detecting an edge from an image is associated with binarizing the pixels of the image into one of two classes. Image binarization may be performed by using global fixed thresholding (using one fixed threshold), locally adaptive thresholding (using a fixed threshold), or the like, depending on the manner in which the threshold value used for such classification is applied For example, by applying a fixed threshold to the brightness-corrected image after removing the brightness change of the background of the image, or by using thresholds different from each other for each pixel position, for example, the brightness average of the local region around the pixel, , Hysteresis thresholding (referring to neighboring pixels spatially or temporally to minimize binarization errors), and so on. Finally, based on the detected edge image as described above, the objectization is performed and the object image can be stored. For example, in an edge image, many backgrounds may be removed and an object may be recognized from the remaining portions. In some instances, these objects may be assigned unique numbers. Objects smaller than a normal size of an object that is a target, such as infrared rays capable devices 121, 123, and 125, may be removed as background noise, This is because small objects may exist.

Based on the derived object image, the neural network-based classifier 208 divides an electronic device, for example, an infrared-capable device 121, 123, 125, located in the environment 100 into a plurality of device candidates Which can be classified as a specific one. In some embodiments, the neural network-based classifier 208 may use a Convolutional Neural Network (CNN) for this recognition / classification. CNN is one of the deep learning supervised learning models. For example, the neural network-based classifier 208 may use a neural network, such as CNN, implemented through Google, Inc.'s open-source library, TensorFlow, or other software package .

Figure 3 schematically shows a neural network used by a neural network based classifier according to some embodiments. The neural network based classifier 208 may perform device classification based on the exemplary neural network 320 of FIG. As shown, the exemplary neural network 320 includes an input layer 321 and one or more hidden layers 323-1, 323-2, ..., 323-n Collectively referred to as hidden layer 323), and an output layer 325. Each layer of the neural network 320 may comprise a plurality of nodes.

Each node in the neural network 320 takes an input and calculates the output, in particular the nodes of the hidden layer 323 and the output layer 325 each take a plurality of inputs and calculate one output. For example, the output of a node may be an activation function (e.g., after a bias is applied) to a weighted sum of outputs from a node of a previous layer (e.g., a sigmoid function A ramp function, an identity mapping, a max out function, a soft max function, or the like). It should be noted, however, that the output of the node may be computed in other manners and that the scope of the present disclosure is not limited in any manner.

The neural network 320 may use a training dataset that includes images of a variety of different electronic devices such as TVs, air conditioners, audio players, video players, fans, set-top boxes, And thus the weight, bias, etc., of the neural network 320 can be determined. In one example, such pre-training may be performed by neural network-based classifier 208 itself. In another example, such pre-training may be performed by a separate system. In some embodiments, the output layer 325 during training of the neural network 320 may have a soft max activation function for multi-class classification.

In some embodiments, once the training is complete, the neural network 320 takes as input the object image captured by the image processing unit 206 and determines the probability of each of the plurality of device candidates (i. E., The class probability) A classification value can be output. Accordingly, the neural network based classifier 208 may be configured to classify the electronic devices (e.g., infrared-capable devices 121, 123, 125) in the environment 100 into specific candidates of a plurality of candidates Quot; device candidate ").

The infrared signaling automatic setting of the device control apparatus 110 will be described again with reference to FIG. After classification of the electronic device by the neural network-based classifier 208, under control of the controller 210, the image processing unit 206 generates a first image of the electronic device classified as a specific candidate (i.e., Can be obtained. To this end, the controller 210 may direct the image capture unit 202 to the classified electronic device.

As will be discussed in detail below, the device control apparatus 110 is configured to determine whether an electronic device matches an electronic device candidate with an image representing an electronic device candidate before transmission of an infrared code signal, An image representing a device candidate can be used. For purposes of illustration, for an infrared code signal, the image of the former may be referred to as a "pre-image ", and the latter image may be referred to as a" post image ".

In some embodiments, the first image may be an image captured by the image capture unit 202 at a particular time. In another embodiment, the first image may be an "average image" of the classified electronic device. This average image can be obtained based on a plurality of image frames captured by the image capturing unit 202. [ For example, the image processing unit 206 can obtain an average image by dividing the accumulated pixel value of a plurality of image frames by the number of image frames.

When the first image is obtained as described above, the controller 210 causes the infrared transmitter 212 to transmit the infrared code corresponding to the device candidate from the infrared code (IR code) table stored in the memory And then transmit the infrared code signal (modulated by the selected infrared code) to the electronic device. The infrared code signal may be transmitted to indicate to a particular candidate to perform at least one specific operation. Examples of operations associated with a device candidate include turning the device on / off, adjusting the device's settings (e.g., volume, TV channel, air conditioning temperature, operating mode, etc.) And the like. In some embodiments, an infrared code signal (e.g., an infrared code signal corresponding to an on / off switch of a remote control of a particular candidate) for directing an operation to switch the on / off of a particular candidate is first transmitted to an infrared transmitter Lt; RTI ID = 0.0 > 212 < / RTI > In another embodiment (if at least some of the surfaces are bright when the electronic device is on, it may be sensed by any suitable technique, such as a pre-image such as a first image and / or an ambient light sensor, Other types of infrared code signals may be initially transmitted from the infrared transmitter 212 to the electronic device (if it is determined that the electronic device is on, based on the output of the unit 204, or the brightness inferred therefrom).

Once the infrared code signal is transmitted, under control of the controller 210, the image processing unit 206 may obtain a second image of the electronic device (i.e., another image representing the electronic device). In some instances, the controller 210 may continue to position the image capture unit 202 towards the electronic device until obtaining a second image. In addition, the second image may be an image captured at a specific time by the image capturing unit 202 or an average image calculated based on a plurality of image frames captured by the image capturing unit 202.

The controller 210 then determines whether the electronic device has performed a specified operation on a pre-image (e.g., a first image) and a post-image (e.g., a second image) in order to verify that the electronic device matches a particular candidate Can be determined. For example, if the difference between the pre-image and the post-image (e.g., the sum of the absolute differences of the respective pixel values of the two images) is greater than or equal to the threshold value, then the electronic device may be determined to have performed a particular action, (And thus the current candidate can be excluded).

If it is determined that a specific operation has not been performed, the controller 210 may cause the infrared transmitter 212 to transmit another infrared code signal to the electronic device. This infrared code signal may be transmitted to instruct other candidates of the plurality of candidates (e.g., candidates having classification values corresponding to the next highest class probability) to perform the same or different operations. The image processing unit 206 may then obtain a third image of the electronic device under control of the controller 210 (i.e., another image representing the electronic device). The controller 210 may determine whether the electronic device has performed the indicated operation based on the pre-image (e.g., at least one of the first image and the second image) and the post-image (e.g., the third image) (E.g., a first image that is a first dictionary image, or a second image that can be treated as a previous image at this time, or a suitable combination thereof) and a posterior image (e.g., a third image) . The third image may also be an image captured at a specific time by the image capturing unit 202 or an average image calculated based on a plurality of image frames captured by the image capturing unit 202. [ This infrared code signal transmission and image comparison for identifying a device match can be repeated until it is determined that a particular operation has been performed or all candidates are excluded.

In order to increase the accuracy of the automatic setting, the controller 210 transmits an additional infrared code signal for indicating to the current device candidate that the infrared transmitter 212 should perform the additional operation To be transmitted to the electronic device, and then make further determination as to whether the electronic device has performed additional operations. This additional determination may be made in some instances by comparing an image representing an electronic device before transmission of an additional infrared code signal with an image representing an electronic device after the transmission. In another example, this additional determination may be made through other techniques, such as is discussed further below.

In general, when the on / off code of the device is confirmed, the code of the detailed function of the device is also available according to the standardized infrared protocol. For example, if an infrared code signal is first sent to indicate an on-off switching operation (and, moreover, that the electronic device has been turned off due to this initial signal, / RTI > the infrared code signal to turn on the electronic device, if determined based on the output of the sensing unit 204, such as an ambient light sensor, or the brightness inferred therefrom, has been sent another time) A signal for instructing to make an infrared signal may be selected as an additional infrared code signal. Examples of detailed operations include operations that result in a change in the display output of a device candidate (e.g., a change in a TV channel), an operation that causes a change in the audio output of a device candidate (e.g., adjustment of a TV volume), and the like. Such a further determination as above makes it possible to withdraw the false determination by chance (for example, human intervention irrespective of the transmission of the first infrared code signal, for example, remote control adjustment).

In an example where the additional infrared code signal indicates an operation to change the display output of the device candidate, the controller 210 determines whether a particular region on the surface of the electronic device (e.g., TV) (e.g., the display surface) The change of the appearance of the user can be recognized. Examples of such specific areas include areas containing numbers such as the channel number of a portion close to the edge of the TV display surface, areas of other home appliances having a relatively small display surface (e.g., air conditioners, CD players, DVD players, fans, etc.) An area including the displayed number and / or letter, an area including an indicator lamp capable of exposing the colored light, and the like. Accordingly, the controller 210 can determine whether the electronic device has performed the display output changing operation based on whether the appearance of the specific area has changed after the additional infrared code is transmitted, the degree of such change, or both. In some embodiments, the appearance of a particular region from an image feature (e.g., which may be extracted from an image representing an electronic device) representing a particular region may be recognized by the image processing unit 206, Can be performed separately before and after the additional infrared code is transmitted so that it can be determined whether the appearance has changed and how much has changed. Similarly, in an example where an additional infrared code signal indicates an operation to change the audio output of a device candidate, a sensing unit 204, such as a sound sensing unit, senses sound from the environment 100, And the controller 120 can determine whether the sound has changed after the additional infrared code has been transmitted, the degree of such change, or both, Can be determined. Additionally or alternatively, a neural network (e.g., neural network 320) may be used to recognize changes in appearance and / or changes in sound.

If it is determined that the electronic device (for example, the infrared ray-capable device 121, 123, 125) has performed the indicated operation as described above, it is confirmed that the electronic device matches the device candidate. Infrared signaling to be used can be set automatically. Further, when a communication interface for another wireless protocol such as Bluetooth is added to the device control apparatus 110 in the Internet of Things (IoT) or smart home environment, The device control apparatus 110 can realize intelligent control that accommodates various types and a large number of electronic devices and can realize an economical, comprehensive and realistic meaningful IoT / smart home. Furthermore, various conveniences can be provided through monitoring the usage time of various electronic devices, grasping the usage patterns, performing timing control and remote control for them, and the like.

Referring now to Figure 4, a process 400 that may be used to automatically set up infrared signaling for device control in accordance with some embodiments will be described. Process 400 may include one or more operations 410 through 490 as shown in FIG. In some embodiments, the process 400 may be performed by the device control apparatus 110 shown in Figs. 1 and 2. For example, operations 410, 420, 440 and 470 may be performed by image processing unit 206, operation 430 may be performed by neural network based classifier 208, operations 450, 460, 480, and 490 may be performed by the controller 210.

Process 400 may begin and lead to operation 410 where an image of the environment taken from the image capture unit may be obtained. At operation 420, an object image may be derived from the acquired image. As mentioned above, object images can be derived and stored through black and white processing, noise reduction, edge detection, binarization, and objectification. At operation 430, an electronic device in the environment may be classified as one of a plurality of device candidates using a neural network (e.g., CNN) that takes an object image as input. In some embodiments, in response to this classification, the image capture unit may be directed to an electronic device.

At operation 440, an image representing the electronic device classified as this current device candidate may be obtained (as a dictionary image). As mentioned above, such an image may be an image at a specific time point or an average image calculated from an accumulated frame. In response to the acquisition of the image of the electronic device, an infrared code associated with the operation of the device candidate may be selected (operation 450) and the infrared transmitter may signal the selected infrared code to the electronic device (act 460) ). By way of example, an operation associated with an infrared code may include an operation of making at least one on-off transition of a device candidate. In operation 470, in response to the infrared code being signaled, another image representing the electronic device may be obtained (as a post-image). As mentioned above, such an image may be an image at a specific time point or an average image calculated from an accumulated frame. In decision operation 480, a determination of whether the electronic device has performed an associated action may be made based on the pre- and post-images. As noted above, making this determination may include comparing the difference between the two images to a threshold value.

If the determination is made at operation 480 that the associated operation has not been performed (e.g., if the difference between the pre- and post-images is less than the threshold), the process 400 may continue at operation 490, Other candidates of the plurality of candidates may be changed and other infrared codes associated with its operation (e.g., the same operation) may be selected. The process 400 then returns to operation 460, where the infrared transmitter may signal the newly selected infrared code to the electronic device. In addition, the image obtained in operation 440 may be maintained as a pre-image as it is, or another image obtained in operation 470 may be newly set as a pre-image. However, if at operation 490 all of the plurality of candidates are already excluded, the process 400 may terminate as a failure and a new instance of the process 400 may be initiated as needed.

If it is determined at operation 480 that an associated operation has been performed (e.g., if the difference between the pre- and post-images is greater than or equal to the threshold value), process 400 may terminate. In some embodiments, at the end of the process 400, the electronic device may be identified as being matched with the device candidate, and thus automatic setting of infrared signaling for control of the electronic device may be enabled. In another embodiment, additional process 500 discussed below may follow the end of process 400 to more closely and / or additionally confirm the device match.

With reference to FIG. 5, an additional process 500 that may be used for automatic configuration in accordance with some embodiments is described. Process 500 may include one or more operations 510 through 540 as shown in FIG. In some embodiments, the process 500 may be performed by the device control device 110, particularly the controller 210 shown in Figs. 1 and 2.

Process 500 may begin and continue to operation 510 where the infrared code associated with the additional operation of the current device candidate may be selected. In operation 520, the infrared transmitter may signal the selected infrared code to the electronic device. At decision operation 530, an additional determination may be made as to whether the electronic device performed the associated additional action.

By way of example, the additional operation associated with the infrared code may include an operation (e.g., tuning of the TV channel) resulting in a change in the display output of the device candidate. In this example, the appearance of a particular area on the surface of the electronic device (which corresponds to a device candidate) may be recognized periodically or under an event-driven manner (e.g., under the control of the controller 210) , The recognition of the appearance can be performed separately before and after the signaling of the selected infrared code. Accordingly, the determination operation 530 may be based on whether the appearance of a particular area has changed and / or the degree of such a change.

In another example, the additional operation associated with the infrared code may include an operation (e.g., adjustment of the TV volume) resulting in a change in the audio output of the device candidate. In this example, sounds from the environment (e.g., environment 100) may be sensed periodically (e.g., under the control of the controller 210) or event driven, so that sounds before and after signaling of the selected infrared code Can be performed. Accordingly, the determination operation 530 may be based on whether the sound of the environment has changed and / or the degree of such change.

If it is determined at operation 530 that no associated additional operation has been performed (e.g., if the appearance or sound change is not significant), the process 500 may continue at operation 540, Other infrared codes associated with other additional operations may be selected. The process 500 then returns to operation 520, where the infrared transmitter may signal the newly selected infrared code to the electronic device. However, if all of the plurality of candidates are already excluded in operation 540, the process 500 may terminate as a failure and a new instance of the process 400 may be started again as needed. The process 500 may terminate if it is determined at operation 530 that an associated additional operation has been performed (e.g., if the appearance or sound change is significant). Further, the electronic device can be identified as being matched with the device candidate, thereby enabling automatic setting of infrared signaling for control of the electronic device.

While the exemplary embodiments of this invention have been described in detail, those skilled in the art will appreciate that various modifications are possible in the exemplary embodiments without materially departing from the scope of the present invention. Therefore, the scope of the present invention should not be limited to the above-described embodiments, but should be determined by equivalents to the appended claims, as well as the appended claims.

110: Device control device
202: image capturing unit
204:
206: image processing unit
208: Neural network based classifier
210:
212: Infrared transmitter

Claims (19)

A method for enabling automatic setting of infrared signaling for device control,
Acquiring an image of the environment from the image capture unit;
Deriving an object image from the image;
Using a neural network to classify electronic devices located in the environment as specific candidates among a plurality of candidates based on the object image;
Obtaining a first image representing the classified electronic device;
Causing an infrared transmitter to transmit to the electronic device an infrared code signal for instructing the specific candidate to perform a specific operation in response to the first image being obtained;
Obtaining a second image representing the electronic device in response to the infrared code signal being transmitted; And
And making a determination as to whether the electronic device has performed the specific operation based on the first image and the second image
Way. The method according to claim 1,
Wherein the step of making the determination includes determining that the electronic device has performed the specific operation when the difference between the first image and the second image is equal to or greater than a threshold value.
Way. The method according to claim 1,
Wherein the specific operation includes switching on and off of the specific candidate.
Way. The method according to claim 1,
Causing the infrared transmitter to transmit to the electronic device another infrared code signal for instructing another candidate of the plurality of candidates to perform the specific operation when it is determined that the specific operation has not been performed;
Obtaining a third image representing the electronic device in response to the other infrared code signal being transmitted; And
Further comprising determining whether the electronic device has performed the specific operation based on at least one of the first image and the second image and the third image
Way. The method according to claim 1,
Causing the infrared transmitter to transmit to the electronic device an additional infrared code signal for instructing the specific candidate to perform additional operation when it is determined that the specific operation has been performed; And
Further comprising, in response to said additional infrared code signal being transmitted, making further determination as to whether said electronic device has performed said additional operation
Way. 6. The method of claim 5,
Wherein the additional operation includes an operation that results in a change in the display output of the particular candidate, wherein the step of performing the further determination comprises determining whether the appearance of a particular area on the surface of the electronic device has changed after the additional infrared code is transmitted, And performing the further determination based on both the degree of change, or both, wherein the specific region corresponds to the specific candidate,
Way. The method according to claim 6,
Further comprising recognizing the appearance from an image feature that represents the particular region, wherein the recognition is performed separately before and after the additional infrared code is transmitted,
Way. 6. The method of claim 5,
Further comprising sensing a sound from the environment, wherein the sensing is performed separately before and after the additional infrared code is transmitted,
Wherein the further operation comprises an operation that results in a change in the audio output of the particular candidate, wherein the step of performing the further determination comprises determining whether the sound is changed after the additional infrared code is transmitted, the degree of the change, And performing the additional determination based on the first determination.
Way. The method according to claim 1,
Further comprising the step of causing the image capturing unit to be directed to the classified electronic device in response to the electronic device being classified as the specific candidate, wherein the step of acquiring the first image is captured by the image capturing unit And acquiring the first image based on a plurality of image frames of the electronic device.
Way. 9. A computer-readable storage medium having computer-executable instructions for causing the processor to perform the method of any one of claims 1 to 9 when executed by at least one processor. An apparatus for enabling automatic setting of infrared signaling for device control,
An image processing unit configured to acquire an image of the environment from the image capturing unit and derive an object image from the image;
A neural network based classifier configured to classify an electronic device located in the environment as a specific candidate among a plurality of candidates based on the object image; And
A controller,
Wherein the image processing unit is further configured to obtain a first image representing the classified electronic device, and in response to the first image being obtained, the controller causes the infrared transmitter to inform the specific candidate Wherein the image processing unit is further configured to obtain a second image representing the electronic device in response to the infrared code signal being transmitted, wherein the image processing unit is configured to obtain an infrared code signal to instruct the electronic device, Configured to make a determination as to whether the electronic device has performed the specific operation based on the first image and the second image,
Device. 12. The method of claim 11,
Wherein the controller is configured to make the determination that the controller is configured to determine that the electronic device has performed the specific operation when the difference between the first image and the second image is greater than or equal to a threshold value,
Device. 12. The method of claim 11,
Wherein the specific operation includes switching on and off of the specific candidate.
Device. 12. The method of claim 11,
The controller is further configured to, when it is determined that the specific operation has not been performed, cause the infrared transmitter to transmit to the electronic device another infrared code signal for instructing another candidate of the plurality of candidates to perform the specific operation Respectively,
Wherein the image processing unit is further configured to obtain a third image representative of the electronic device in response to the other infrared code signal being transmitted and wherein the controller is configured to receive at least one of the first image and the second image, 3 image to determine whether the electronic device has performed the particular operation,
Device. 12. The method of claim 11,
The controller is further configured to, when it is determined that the specific operation has been performed, to cause the infrared transmitter to transmit to the electronic device an additional infrared code signal for instructing the particular candidate to perform additional operation,
The controller is further configured to, in response to the further infrared code signal being transmitted, to make further determination as to whether the electronic device has performed the further operation,
Device. 16. The method of claim 15,
Wherein said further operation comprises an operation that results in a change in the display output of said particular candidate, wherein said controller is configured to make said further determination that said controller is operable, after said additional infrared code is transmitted, Wherein the specific region is configured to perform the additional determination based on whether the appearance of the specific region is changed, the degree of change, or both,
Device. 17. The method of claim 16,
Wherein the image processing unit is further configured to recognize the appearance from an image feature that represents the specific area, wherein the recognition is performed separately before and after the additional infrared code is transmitted,
Device. 16. The method of claim 15,
Further comprising a sound detection unit configured to detect sound from the environment, wherein the sensing is performed separately before and after the additional infrared code is transmitted,
Wherein the further action comprises an action that results in a change in the audio output of the particular candidate, wherein the controller is configured to make the further determination further comprises determining whether the sound has changed after the additional infrared code is transmitted, Or both, based on the degree of < RTI ID = 0.0 >
Device. 12. The method of claim 11,
Wherein the controller is further configured to cause the image capturing unit to be directed to the classified electronic device in response to the electronic device being classified as the specific candidate, and wherein the image processing unit is configured to acquire the first image Wherein the image processing unit is configured to obtain the first image based on a plurality of image frames of the electronic device captured by the image capture unit,
Device.

Images