JP6890451B2 - Remote control system, remote control method and program - Google Patents

Description

The present invention relates to a remote control control system, a remote control control method, and a program for controlling a plurality of home appliances with a learning type remote controller that learns control signals of a remote control device (hereinafter, simply referred to as a remote controller) used for controlling home appliances and the like.

For the control of home appliances such as televisions and air conditioners (air conditioners), it is common to control by a remote controller that transmits control information to the home appliances to be controlled by infrared rays or radio waves.
When the user points the remote controller at the home appliance to be controlled and presses a predetermined button on the control panel of the remote controller, a control signal for performing control corresponding to the button is output. For example, if the remote controller is an infrared remote controller, an infrared pulse pattern corresponding to the type of control is emitted from the infrared LED mounted on the remote controller.

In recent years, as the number of remote controls has increased in response to the increase in home appliances, it has become complicated to control using the remote controls of each home appliance. For this reason, a learning type remote controller is used in which the pulse pattern of a control signal in each remote controller of a plurality of home appliances is learned and a single remote controller controls a plurality of home appliances.
By using this learning remote controller, the learning remote controller can be controlled from a remote location using an application such as chat, and the control of each of a plurality of home appliances in the room where the learning remote controller is arranged can be controlled by the Internet, etc. It can be done using the communication line of.

However, in the case of control using infrared control signals of a general learning type remote controller, only the control signal is radiated to the home appliances to be controlled, and whether or not the home appliances operate in response to the control signals is determined. Confirmed by the user who operates the learning remote controller. For example, if an unexpected shield temporarily exists between the home appliance and the learning remote controller, the control signal of the learning remote controller does not reach the home appliance, and the home appliance cannot be operated normally. Further, at this time, since the user is in a remote place, it is not possible to directly image the state of the home electric appliance.
Therefore, as described above, even if the learning type remote controller is controlled from a remote location and the home electric appliance is operated, it is not known whether or not the home electric appliance has transitioned to a desired state and has been operated normally.

Therefore, it is difficult to incorporate and use the above-mentioned learning type remote controller as a control device in an intelligent home network system in which a user controls home appliances from a remote location. In order for a user in a remote location to observe the state of home appliances, a communication method (for example, ECHONET Lite (registered trademark)) capable of acquiring the operating state of the home appliances is implemented in each of the home appliances. It is necessary to secure the control result by detecting the operating state at the time of control by the learning type remote controller.
However, since the function of the above communication method is incorporated in each home electric appliance, the manufacturing cost of the home electric appliance increases, and the home electric appliance which is not widely used in general households and does not have this function is introduced into the above communication method. Cannot be used under the network.

In order to solve the above problem, the image when the home appliance is controlled by operating the learning remote controller and the sensor information for detecting the operation of the home appliance are notified to the mobile terminal of the user who controls the learning remote controller. (See, for example, Patent Document 1). The user confirms whether or not the home electric appliance is in an operating state corresponding to the desired operation by the notification from the above system.

Japanese Unexamined Patent Publication No. 2017-22745

However, Patent Document 1 has the complexity that the user must confirm the content of the notification from the system on the mobile terminal and visually confirm whether or not the operation is normal.
In addition, it is necessary to check the content of the notification indicating the operating status of the home appliances from the system at the specified time, such as when the time is specified by a timer, etc., and determine whether or not it is operating normally as controlled. However, it is difficult to respond one by one at the specified time.

The present invention has been made in view of such circumstances, and when a user controls a home electric appliance from a remote location by a learning remote controller, the home electric appliance operates as desired without being visually confirmed by the user himself / herself. It is an object of the present invention to provide a remote control control system, a remote control control method, and a program capable of easily confirming whether or not the state is in the above state.

The present invention has been made to solve the above-mentioned problems, and the remote control control system of the present invention inputs detection data from a sensor that detects an operating state of a home appliance that receives a control signal radiated by the remote control. The user determines the state detection function unit, the control result determination unit that determines whether or not the home appliance has transitioned to the operating state corresponding to the instruction of the control signal based on the detection data, and the determination result of the control result determination unit. The notification function unit that notifies to , at least the control command indicated by the control signal, the detection data from each of the plurality of sensors that detect the operating state, and the actual control result when these are used. Control that performs machine learning as teacher data and estimates the determination of the control result from the control command indicated by the input control signal and the detection data from each of the plurality of sensors that detect the operating state. A determination model learning unit that generates a result determination model is provided, and the control result determination unit includes at least the control command indicated by the control signal and the operation at each time before and after the instruction by the control signal of the remote control. using a control result determination model and the detected data from a plurality of each of said sensor state detecting with an input, it estimates the control result of the home appliance, and this performs the judgment by the result of the estimation It is a feature.

The remote control control system of the present invention further includes a remote control control unit that controls the remote control to radiate a control signal corresponding to an operation specified by the user, and the control result determination unit controls the control result of the home electric appliance. Is not in a state corresponding to the control signal, the remote control control unit re-radiates the control signal to the remote control.

The remote control control system of the present invention is equipped with the remote controller and includes a radiation direction adjusting mechanism for controlling the radiation direction of the control signal, and the remote controller control unit controls the radiation direction adjustment mechanism to control the home appliance. It is characterized in that the radiation direction of the control signal of the remote controller is adjusted to the arrangement position of the device.

The remote control control system of the present invention further includes a dialogue agent that interacts with the user, and the dialogue agent transmits a command for controlling the home electric appliance of the user to the remote control control unit by the dialogue with the user. It is characterized in that the machine learning process is transmitted to the determination model learning unit.

In the remote control control method of the present invention, the state detection function unit inputs the detection data from the sensor that detects the operating state of the home appliance that has received the control signal from the remote control, and the control result determination unit The control result determination process for determining whether or not the home appliance has transitioned to the operating state corresponding to the instruction of the control signal based on the detection data, and the notification function unit informs the user of the determination result of the control result determination unit. The notification process to notify , the control command indicated by the control signal, the detection data from each of the plurality of sensors for detecting the operating state, and the actual detection data when the determination model learning unit uses them. Machine learning is performed using the control result as teacher data, and the control result is determined from the control command indicated by the input control signal and the detection data from each of the plurality of sensors that detect the operating state. The control result determination unit includes at least a control command indicated by the control signal before and after the instruction by the control signal of the remote control, including a determination model learning process for generating a control result determination model for estimating. The control result determination model that inputs the detection data from each of the plurality of sensors that detect the operating state is used to estimate the control result of the home appliance, and the determination is made based on the estimation result. and said that you do.

The program of the present invention is a state detection function means for inputting detection data from a sensor that detects an operating state of a home appliance device that has received a control signal from a remote control, and the home appliance device uses the detection data to control the control signal. control result judging means for judging whether a transition in the operational state corresponding to the instruction, the notification function unit for notifying the determination result of the control result determining unit to a user, a control command indicated by at least the control signal, the Machine learning is performed using the detection data from each of the plurality of sensors for detecting the operating state and the actual control results when these are used as teacher data, and the control command indicated by the input control signal is performed. When the from the detection data from each of the plurality of sensors for detecting the state of operation, to function as a judgment model learning means for generating a control result judgment model for estimating the determination of the control result, the control result The determination means includes at least the control command indicated by the control signal and the detection data from each of the plurality of sensors for detecting the operating state before and after the instruction by the control signal of the remote control. using a control result determination model as input, it estimates the control result of the home appliance is a program that makes the determination as a result of the estimation.

According to the present invention, when a user controls a home electric appliance from a remote location by a learning type remote controller, it is easy for the user to check whether or not the home electric appliance is in a desired operating state without visually confirming it. It is possible to provide a remote control control system, a remote control control method, and a program that can be confirmed.

It is a figure which shows the configuration example of the remote control control system of 1st Embodiment of this invention. It is a figure which shows the configuration example of the device data table written and stored in the device database 132. It is a figure which shows the configuration example of the command table which is written and stored in the command database 131. It is a figure which shows the configuration example of the state data table written and stored in the state database 133. It is a flowchart which shows the operation example of the position confirmation processing in the room of the home electric appliance using the remote control control system of 2nd Embodiment. It is a flowchart which shows the operation example of the control process of the home electric appliance using the remote control control system of this embodiment. It is a figure which shows the configuration example of the remote control system of the 2nd Embodiment of this invention. It is a figure which shows the configuration example of the control history data table stored in the control history database 134. It is a figure which shows the configuration example of the state data table stored in the state database 133A. It is a flowchart which shows the operation example of the acquisition process of the teacher data which trains the control result judgment model in the control result judgment part in the remote control control system of this embodiment. It is a flowchart which shows the operation example of the control process of the home electric appliance using the remote control control system of the 2nd Embodiment.

<First Embodiment>
Hereinafter, the first embodiment of the present invention will be described with reference to the drawings. FIG. 1 is a diagram showing a configuration example of a remote control control system according to the first embodiment of the present invention.
In FIG. 1, the remote control control system includes a robot device 11, a remote control control device 12, and a storage device 13. The robot device 11, the remote control control device 12, and the storage device 13 are connected by a communication network 100 such as a home network. The communication network 100 is connected to the mobile terminal 200 via the Internet or the like. Each of the robot device 11 and the remote control control device 12 has an application program corresponding to each of the functions described below installed and executes each function. Further, in the present embodiment, an example in which the remote control control system is configured separately for each of the robot device 11 and the remote control control device 12 is shown, but the robot device 11 and the remote control control device 12 are separated into one server or the like. It may be integrated in a computer system. Further, the robot device 11, the remote control control device 12, and the storage device 13 may be integrated in a computer system such as one server. Further, each function may be configured by an individual computer system and may be configured to operate in cooperation with each other.

The robot device 11 includes a microphone (microphone) device 111, an image pickup device 112, a dialogue agent 113, and a speaker device 114, respectively.
The microphone device 111 is provided with a microphone sensor having directivity, detects (acquires) sound, and outputs sound information (intensity, frequency, etc.). Here, the robot device 11 writes and stores the sound information detected by the microphone device 111 in the state database 133.

The image pickup device 112 is a directivity image pickup sensor (for example, a CCD (charge coupled device) image sensor), and writes and stores the captured image data in the state database 133. Here, the robot device 11 writes and stores the image data captured by the image pickup device 112 in the state database 133. Further, the image pickup device 112 may use a spherical image pickup sensor having no directivity as the image pickup sensor.

Further, the robot device 11 is equipped with a learning remote controller 400, a microphone device 111, and an imaging device 112, and the learning remote controller 400 is fixed in order to adjust the radiation direction of infrared rays (control signals described later) of the learning remote controller 400. A direction adjusting mechanism for changing the azimuth and elevation of the table is provided. In this direction adjustment mechanism, the infrared radiation direction of the learning remote controller 400, the acquisition direction for acquiring the directivity sound of the microphone device 111, and the image pickup direction of the image pickup device 112 are the same. A learning remote controller 400, a microphone device 111, and an imaging device 112 are attached to a fixed base. Then, by adjusting the azimuth angle and elevation angle of the fixed base by the direction adjusting mechanism, the infrared radiation direction of the learning type remote controller 400, the acquisition direction for acquiring the directivity sound of the microphone device 111, and the image pickup device 112. The imaging direction of the image can be adjusted. Here, when a spherical image sensor is used as the image pickup device 112, the image pickup device 112 is not mounted on the direction adjusting mechanism, but is fixedly attached to the housing of the robot device 11.

The dialogue agent 113 serves as an interface when a user controls a home electric appliance by a remote control control system and receives a notification of an operating state of the home electric appliance. The dialogue agent 113 communicates with the user by performing a voice dialogue in natural language using the microphone device 111 and the speaker device 114, a voice recognition server and a voice synthesis server which are application servers included in the dialogue agent 113, and a mobile terminal 200. The data related to the control of the home appliance 300 is transmitted and received in an interactive manner such as chatting via.
That is, when the dialogue agent 113 has a dialogue with the user, for example, the speaker device 114 notifies the user of the question by voice, and the voice of the answer from the user is input by the microphone device 111. Here, the dialogue agent 113 performs processing of synthesizing the voice of the question to be given to the user and recognizing the content of the voice of the answer from the user by each of the voice synthesis server and the voice recognition server included in the dialogue agent 113.
Further, the robot device 11 reads the sensor detection data from each of the sensors 14_1, ..., And the sensor 14_n arranged in the vicinity of the home appliance 300_1 from the home appliance 300_1, writes them in the state database, and stores them. Hereinafter, when the home appliances 300_1 to the home appliances 300_m are collectively referred to as the home appliances 300, and the sensors 14_1 to the sensors 14_n are collectively referred to as the sensor 14.

Further, the robot device 11 accesses the home appliance of the home appliance 300 manufacturer on the web based on the name (manufacturer name) of the home appliance 300 input via the dialogue agent 113, and acquires the image data of the home appliance 300. Then, a device ID (identification) for identifying the home electric appliance 300 is assigned, and the device database 132 is written and stored.
Further, the robot device 11 causes the learning type remote controller 400 to learn the command of each individual remote controller of the home electric appliance 300, assigns a command name and a command ID for identifying the command of the command name, and writes the command to the device database 132. Remember with. At this time, the robot device 11 corresponds to the above command ID, and writes and stores the information of the infrared pulse pattern emitted corresponding to the command indicated by each command ID in the command database 131. The command database 131 may be provided in any of the robot device 11, the storage device 13, and the learning remote controller 400.
Further, the robot device 11 reads and acquires the command and infrared pulse pattern of the remote controller of the home electric appliance 300 to be learned from the database in which the commands of the remote controllers of the home electric appliances 300 are registered on the web, and is a learning type. The remote controller 400 may be trained.
Further, when the user inputs the command of each remote controller of the home electric appliance 300 to the learning type remote controller 400 by his / her own operation, the information of the infrared pulse pattern radiated in response to the command indicated by the command ID May be trained.

The remote control control device 12 includes each of a control result determination unit 121 and a remote control control unit 122.
The control result determination unit 121 estimates each state of the home electric appliance 300 by using each of the sound information, the image data, and the sensor detection data stored in the state database 133. Further, the control result determination unit 121 determines from the estimated state whether or not the control corresponding to the control signal radiated from the learning type remote controller 400 is performed in the home electric appliance 300. The control result determination unit 121 notifies the user's mobile terminal 200 of the determination result via the dialogue agent 113.
The remote controller control unit 122 causes the learning remote controller 400 to emit a control signal for controlling each of the home appliances 300. At this time, the remote controller control unit 122 controls the direction adjusting mechanism of the robot device 11 to adjust the radiation direction of the learning remote controller 400.

In the device database 132, information on home appliances controlled by commands registered in the learning remote controller 400 is written and stored in a device data table described in association with a device ID that identifies each home appliance. There is.
FIG. 2 is a diagram showing a configuration example of a device data table written and stored in the device database 132. In FIG. 2, the device data table is associated with each device ID for each record, and is provided with columns for a name (maker name), a command name list, a command ID list, an image, and a position. The name (manufacturer name) is the name of the home electric appliance, for example, the product name, and is described together with the manufacturer name. The command name is the name of the command when it is controlled by the remote controller. For example, if the home electric appliance 300 is an air conditioner, heating is ON, heating is OFF, and the like. The command ID is identification information that identifies the command. The image is image data showing the shape of the home electric appliance 300. The position indicates information indicating the mounting position of the home electric appliance 300, that is, control information of the direction adjusting mechanism with the robot device 11 as the origin.

In the command database 131, a command table in which the command ID of the command registered in the learning type remote controller 400 and the infrared pulse pattern corresponding to this command ID are associated with each other is written and stored.
FIG. 3 is a diagram showing a configuration example of a command table written and stored in a command database 131 provided in any of the robot device 11, the storage device 13, and the learning remote controller 400. In FIG. 3, the command table is associated with each command ID for each record, and a signal pattern (pulse pattern) column is provided. As the signal pattern, information (frequency, pulse width, etc.) of the pulse pattern for controlling the home electric appliance 300 emitted by the learning type remote controller 400 is described in response to the command indicated by the command ID.

FIG. 4 is a diagram showing a configuration example of a state data table written and stored in the state database 133. In FIG. 4, the state data table is associated with each command ID for each record, and is provided with columns for time, camera information, microphone input, and sensor output. The time indicates the time when the infrared rays of the pulse pattern corresponding to the command of this command ID are radiated. The camera information is image data captured by the image pickup apparatus 112 after the infrared rays of the pulse pattern corresponding to the command of this command ID are emitted. The microphone input is voice data acquired by the microphone device 111 after the infrared rays of the pulse pattern corresponding to the command of this command ID are radiated.

Further, the sensor output shows the detection result output by the sensor arranged in the vicinity of the home electric appliance 300 corresponding to the command of the command ID. For example, when the home electric appliance 300 is a television (television set), an illuminance sensor is provided near the screen. Further, when the home electric appliance 300 is an air conditioner, a temperature sensor, an air volume sensor (so-called differential pressure sensor used as an air volume meter), and the like are provided in the vicinity of the air outlet. As described above, as the sensor arranged in the vicinity of the home electric appliance 300, a type that can obtain information that can confirm the operation of each home electric appliance 300 is used. As described above, in the present embodiment, each of the sensors is individually attached to each of the home appliances 300.

Next, with reference to FIG. 5, the flow of the position confirmation process in the room of the home electric appliance using the remote control control system of the present embodiment will be described. FIG. 5 is a flowchart showing an operation example of the position confirmation process in the room of the home electric appliance using the remote control control system of the second embodiment.

Step S11: The robot device 11 acquires an image from the home appliance 300 by the name (maker name) of the home appliance 300 that the user inputs by voice or input means (touch panel or button, keyboard) via the dialogue agent 113. To identify.

Step S12: The robot device 11 searches the homepage of the manufacturer based on the name (manufacturer name) of the specified home electric appliance 300. Further, the robot device 11 may access the mother database on the web in which the device information of the home electric appliances 300 of various manufacturers is registered in advance, and search the device information of the specified home electric appliances 300.
Then, the robot device 11 acquires image data indicating the shape of the home electric appliance 300 from the homepage of the maker of the searched home electric appliance 300 or the record of the mother database. At this time, the robot device 11 assigns a device ID to the home electric appliance 300 corresponding to the acquired image data, and writes and stores the image data of the home electric appliance 300 in the device data table of the device database 132.

Step S13: The robot device 11 adjusts the direction adjusting mechanism to control the imaging direction of the imaging device 112. Then, the robot device 11 searches the home appliance 300 that matches the image data of the home appliance described in the device data table of the device database 132 in the indoor image captured by the image pickup device 112. When a spherical image sensor is used as the image pickup device 112, the robot device 11 uses the spherical image of the room to be imaged in a three-dimensional coordinate system based on the spherical image sensor. The position of the image of the home appliance 300 that matches the image data of the home appliance in which direction is searched for.

Step S14: When the robot device 11 detects the home appliance 300 that matches the image data of the home appliance described in the device data table, the robot device 11 determines the control value of the direction adjusting mechanism at that time as the position of the home appliance 300. Further, when the spherical image sensor is used as the image pickup device 112, and the robot device 11 sets the direction searched from the spherical image as the position of the home electric appliance 300.

Step S15: Then, the robot device 11 writes and stores the control value of the direction adjusting mechanism determined to be the position of the home appliance 300 in the device data table of the device database 132.

Step S16: The dialogue agent 113 confirms with the user whether or not there is another home electric appliance 300 that registers the arrangement position in the device data table. At this time, when the robot device 11 receives a reply from the user via the dialogue agent 113 that there is no other home electric appliance 300 to be registered in the device data table, the robot device 11 ends the process. On the other hand, when the robot device 11 receives a reply from the user that there is another home electric appliance 300 to be registered in the device data table via the dialogue agent, the process returns to step S11.

A flow of processing for controlling a home electric appliance using the remote control control system of the present embodiment will be described with reference to FIG. FIG. 6 is a flowchart showing an operation example of the control process of the home electric appliance using the remote control control system of the present embodiment. In the following description, the home appliance 300 to be controlled will be described as an air conditioner.

Step S21: The remote controller control device 12 acquires each of the home appliance 300 to be controlled and the command name via the dialogue agent 113, and identifies the home appliance 300 to be controlled and each command to be controlled.

Step S22: The remote controller control unit 122 controls the direction adjusting mechanism of the robot device 11 so that the infrared radiation direction of the learning type remote controller 400 is directed to the home appliance 300 specified by the remote controller control device 12. As a result, the remote controller control unit 122 directs the infrared radiation direction of the learning remote controller 400 to the arrangement position of the home appliance 300 to be controlled. At this time, each of the imaging direction of the imaging device 112 and the sound collecting direction of the microphone device 111 also faces the same direction as the infrared radiation direction of the learning type remote controller 400.
Then, the remote controller control unit 122 reads the pulse pattern information corresponding to the command ID to be controlled from the command table of the command database 131 and transmits it to the learning type remote controller 400.

Step S23: The learning type remote controller 400 emits infrared rays of a pulse pattern supplied from the remote controller control unit 122. Since the radiation direction of the learning type remote controller 400 is directed to the home electric appliance 300 to be controlled, the infrared rays reach the home electric appliance 300 to be controlled.

Step S24: The remote control control device 12 writes and stores the image data of the home electric appliance 300 captured by the image pickup device 112 as camera information in the state data table of the state database 133 in correspondence with the command ID.
Similarly, the remote control control device 12 writes and stores the voice data generated by the home electric appliance 300 acquired by the microphone device 111 as the microphone input in the state data table of the state database 133 in correspondence with the command ID.
Further, the remote control control device 12 corresponds to the command ID as the sensor output of the air volume data of the air blown out by the air conditioner acquired from the sensor 14 that detects the air volume arranged at the air outlet of the louver of the air conditioner of the home appliance 300. Then, it is written and stored in the state data table of the state database 133.

Step S25: The control result determination unit 121 reads the image data from the state data table of the state database 133, performs image analysis of the read image data, detects, for example, the operating state of the louver, and operates the home appliance 300. Estimate the state.
Further, the control result determination unit 121 reads voice data from the state data table of the state database 133, performs voice analysis of the read voice data, and detects, for example, the operating sound of the louver and the learning type remote controller 400 command reception sound. Then, the operating state of the home appliance 300 is estimated.
Similarly, the control result determination unit 121 reads the air volume data from the state data table of the state database 133, analyzes the air volume of the read air volume data, and detects, for example, the air volume from the air outlet of the air conditioner. Estimate the operating state of 300.

Then, the control result determination unit 121 determines whether or not each of the operating state of the louver, the operating sound of the louver, the air volume from the outlet, or the learning type remote controller 400 command reception sound exceeds a predetermined threshold value set in advance. Make a judgment. At this time, the control result determination unit 121 determines whether or not the operating state of the louver, the operating sound of the louver, the air volume from the outlet, and the learning type remote controller 400 command reception sound exceed the threshold value with respect to the determination device provided inside. The judgment result is supplied. The above-mentioned determination device determines that the home electric appliance 300 is operating normally when all the determination results exceed the threshold value. On the other hand, if any of the determination results does not exceed the threshold value, the determination device determines that the home electric appliance 300 is not operating normally.
Further, in the above-mentioned determination device, when any of the determination results exceeds the threshold value, the home electric appliance 300 operates normally, while when all the determination results are equal to or less than the threshold value, the home electric appliance 300 operates normally. It may be configured to determine that it is not.

Step S26: The control result determination unit 121 determines whether or not the home electric appliance 300 is operating based on the determination result of the determination device.
At this time, if the determination device determines that the home appliance 300 is operating normally, the control result determination unit 121 proceeds to step S27.
On the other hand, when the determination device determines that the home electric appliance 300 is not operating normally, the control result determination unit 121 advances the process to step S28.

Step S27: The remote control control device 12 notifies the mobile terminal 200 carried by the user at a remote location via the dialogue agent 113 that the home electric appliance 300 is operating normally.

Step S28: The remote control control device 12 notifies the mobile terminal 200 carried by the user at a remote location via the dialogue agent 113 that the home appliance 300 is not operating normally (abnormal processing).
Further, when the remote control device 12 is an abnormal process, the process is returned to step S22, a process of radiating infrared rays to the home electric appliance 300 is performed again, and the process is performed to the mobile terminal 200. It may be configured to notify by.

As described above, according to the present embodiment, the operating state of the home electric appliance 300 is confirmed by the determination device based on each of the image data, the voice data, and the sensor output acquired from the home electric appliance 300 operated by the learning type remote controller. , In order to notify the user of the operating status, when the user controls home appliances with a learning remote controller via a dialogue agent using chat etc. from a remote location, the user does not have to visually check it. , It becomes possible to easily confirm whether or not the home electric appliance is in a desired operating state.
Further, according to the present embodiment, when the control result desired by the user cannot be obtained, the remote control control device 12 automatically controls the home electric appliance designated by the user again, so that the user's dialogue agent is used. It is possible to reduce the control load of the home electric appliance 300.

<Second embodiment>
Hereinafter, a second embodiment of the present invention will be described with reference to the drawings. FIG. 7 is a diagram showing a configuration example of a remote control system according to a second embodiment of the present invention.
In FIG. 7, the remote control control system includes a robot device 11, a remote control control device 12A, and a storage device 13A, respectively. The remote control control device 12A includes each of a control result determination unit 121A, a remote control control unit 122, and a determination model learning unit 123. The storage device 13A includes a command database 131, a device database 132, a state database 133A, and a control history database 134, respectively.
Hereinafter, only the configurations and operations in which the second embodiment is different from that of the first embodiment will be described. In FIG. 7, the same reference numerals are given to the same configurations as those of the first embodiment, and the description thereof will be omitted.

The remote control control device 12A includes a control result determination unit 121A, a remote control control unit 122, and a determination model learning unit 123.
In the present embodiment, the determination device in the control result determination unit 121A is a control result determination model generated by machine learning. This control result determination model is an estimation model using an algorithm selected according to the characteristics of data such as an SVM (support vector machine) or a neural network.
This control result determination model may be created by machine learning as one model for each home appliance 300, or may be created by machine learning as one model for a plurality of home appliances 300.

In the machine learning, as shown below, the control result determination model is created by a control operation performed by a user via the dialogue agent 113 using the teacher data.
In machine learning, the determination model learning unit 123 causes the user to control the home electric appliance 300 via the dialogue agent 113, and then whether the home electric appliance 300 operates correctly in response to the control. Have them judge whether or not.
At this time, the robot device 11 acquires image data, voice data, and sensor output from each of the image pickup device 112, the microphone device 111, and the sensor 14 for each command ID for which machine learning has been performed, and controls the state database 133A. It is written to the history database 134 and stored.

FIG. 8 is a diagram showing a configuration example of a control history data table stored in the control history database 134. In FIG. 8, the control history data table is associated with each of the control IDs for each record, and each column of a time, a command ID, a state ID, and a correctness label is provided. The control ID is information for identifying each of the controls performed on the home electric appliance 300. The time indicates the time when the infrared rays of the pulse pattern corresponding to the command of the command ID are radiated. The command ID is identification information that identifies the command. The state ID is information that identifies data indicating the state of the home electric appliance 300 obtained as a result of performing the control. The correctness label is a label indicating whether or not the home electric appliance 300 has normally operated as a result of performing the control indicated by this control ID. If it operates normally, it becomes "positive", and if it does not operate normally, it becomes "positive". No. " After controlling the home electric appliance, the determination model learning unit 123 determines to the user whether or not the home electric appliance 300 is operating normally corresponding to each of the control IDs via the dialogue agent 113. The judgment result of the user who inquired and answered is written and stored in the control history database 134 as a correctness label.

FIG. 9 is a diagram showing a configuration example of a state data table stored in the state database 133A. In FIG. 9, the state data table is associated with each of the state IDs for each record, and is provided with columns for time, camera information, microphone input, and sensor output. The camera information is based on the infrared rays of the pulse pattern corresponding to the command of this command ID before being emitted (that is, before the command is instructed by infrared rays to the home appliance 300) and after being emitted (that is, before the infrared command is given to the home appliance 300). After the command is instructed), there are two types of image data captured by the imaging device 112. The microphone input is two types of audio data acquired by the microphone device 111 before and after the infrared rays of the pulse pattern corresponding to the command of this command ID are emitted. The sensor output shows the detection result output by the sensor arranged in the vicinity of the home electric appliance 300 corresponding to the command of the command ID. Further, the sensor outputs are two types of sensor outputs detected by the sensor 14 before being radiated (before the instruction) and after being radiated (after the instruction).

Returning to FIG. 7, the determination model learning unit 123 reads out each of the command ID, the state ID, and the correctness / rejection label for each control ID from the control history data table of the control history database 134. Further, the determination model learning unit 123 reads each of the time, camera information, microphone input, and sensor output corresponding to the read state ID from the state data table of the state database 133A. Here, the read command ID, two types of image data before and after the infrared rays are emitted, two types of audio data before and after the infrared rays are emitted, and two types of sensor outputs before and after the infrared rays are emitted. Each is input, and the judgment result indicated by the correctness label is used as output and as teacher data for training the control result judgment model.

Then, the determination model learning unit 123 has two types of image data before and after the command ID and infrared rays are emitted, two types of audio data before and after the infrared rays are emitted, and two types before and after the infrared rays are emitted. Each of the sensor outputs is supplied as an input, and the weighting coefficient of the control result determination model is adjusted so that the determination result indicated by the correctness label is output (estimated). A plurality of types of the teacher data are acquired for each home electric appliance 300 in different indoor states (for example, the same control is performed a plurality of times at intervals), and a large number of the teacher data are obtained. Further, the determination model learning unit 123 sends the user the control result determination model for all the command IDs of the commands of the plurality of remote controllers registered in the learning type remote controller 400 via the dialogue agent 113. Acquire teacher data for learning.

Next, with reference to FIG. 10, the flow of the teacher data acquisition process for learning the control result determination model in the control result determination unit in the remote control control system of the present embodiment will be described. FIG. 10 is a flowchart showing an operation example of the teacher data acquisition process for learning the control result determination model in the control result determination unit 121A in the remote control control system of the present embodiment. The following description shows the operation when the user sets the remote control control system into a mode for performing machine learning of the control result determination model.

Step S31: The robot device 11 identifies the home appliance 300 to be acquired from the home appliance 300 by the name (manufacturer name) of the home appliance 300 that the user inputs by voice or input means via the dialogue agent 113.
Further, the remote controller control unit 122 controls the direction adjusting mechanism of the robot device 11 so that the infrared radiation direction of the learning type remote controller 400 is directed to the home appliance 300 specified by the remote controller control device 12A. As a result, the remote controller control unit 122 directs the infrared radiation direction of the learning remote controller 400 to the arrangement position of the home appliance 300 to be controlled. At this time, each of the imaging direction of the imaging device 112 and the sound collecting direction of the microphone device 111 also faces the same direction as the infrared radiation direction of the learning type remote controller 400. At this time, the remote control control unit 122 acquires the command ID from the user via the dialogue agent 113.

Step S32: The robot device 11 takes an image of the home appliance 300 by the image pickup device 112 and takes an image data before the infrared rays of the pulse pattern corresponding to the command of the command ID are emitted. Further, the robot device 11 acquires voice data of the home electric appliance 300 by the microphone device 111, and detects the sensor output by the sensor 14.
Then, the robot device 11 assigns a control ID, and writes and stores each of the control ID, the command ID, and the state ID in the control history data table of the control history database 134. Further, the robot device 11 writes and stores each of the state ID corresponding to the control ID, the time when the data was acquired, the camera information, the microphone input, and the sensor output in the state data table of the state database 133A.

Step S33: The remote controller control unit 122 reads the pulse pattern information corresponding to the command ID to be controlled from the command table of the command database 131, and transmits the information to the learning remote controller 400.

Step S34: The learning type remote controller 400 emits infrared rays of a pulse pattern supplied from the remote controller control unit 122. Since the radiation direction of the learning type remote controller 400 is directed to the home electric appliance 300 to be controlled, the infrared rays reach the home electric appliance 300 to be controlled.

Step S35: After a predetermined time elapses (for example, one minute), after the infrared rays of the pulse pattern corresponding to the command of the command ID are emitted (after the instruction by the command), the robot device 11 is subjected to the home appliance 300 by the image pickup device 112. Image is taken and image data is taken. Further, the robot device 11 acquires voice data of the home electric appliance 300 by the microphone device 111, and detects the sensor output by the sensor 14.
Then, the robot device 11 writes and stores each of the command ID and the state ID in the record of the corresponding control ID in the control history data table of the control history database 134. Further, the robot device 11 writes and stores each of the state ID corresponding to the control ID, the time when the data was acquired, the camera information, the microphone input, and the sensor output in the state data table of the state database 133A.

Step S36: Whether or not the home appliance 300 is normally operating in a state corresponding to the given command (that is, whether or not the robot device 11 has transitioned to an operating state corresponding to the instruction of the control signal indicated by the command). The user is inquired about the determination of the above via the dialogue agent 113. Then, the robot device 11 obtains an answer as to whether or not the home electric appliance 300 is in an operating state, which is obtained via the dialogue agent 113.

Step S36 :, The robot device 11 writes and stores the answer to the availability of the operating state of the home electric appliance 300 obtained via the dialogue agent 113 in the control history data table of the control history database 134 as a correctness label.
In the mode in which the machine learning of the control result determination model is performed, the robot device 11 performs the processing of the above flowchart as the acquisition of teacher data, and commands of all command IDs of the commands of the plurality of remote controllers registered in the learning type remote controller 400. Do it for all of.

A flow of processing for controlling a home electric appliance using the remote control control system of the present embodiment will be described with reference to FIG. FIG. 11 is a flowchart showing an operation example of the control process of the home electric appliance using the remote control control system of the second embodiment. In the following description, the home appliance 300 to be controlled will be described as an air conditioner.

Step S41: The remote control control device 12 identifies the home appliance 300 to be controlled by the name of the home appliance 300 that the user inputs by voice or input means via the dialogue agent 113.
Further, the remote controller control unit 122 controls the direction adjusting mechanism of the robot device 11 so that the infrared radiation direction of the learning type remote controller 400 is directed to the home appliance 300 specified by the remote controller control device 12. Further, the remote control control unit 122 acquires the command ID from the user via the dialogue agent 113.

Step S42: The remote control control device 12 uses the image data of the home appliance 300 captured by the image pickup device 112 as camera information, and the voice data generated by the home appliance 300 acquired by the microphone device 111 as the microphone input, and uses it as a command ID. Correspondingly, each state data table of the state database 133A is written and stored as data before emitting infrared rays (before instructed by a command).
Further, the remote control control device 12 corresponds to the command ID as the sensor output of the air volume data of the air blown out by the air conditioner acquired from the sensor 14 that detects the air volume arranged at the air outlet of the louver of the air conditioner of the home appliance 300. Then, it is written and stored in the state data table of the state database 133A as data before emitting infrared rays.

Step S43: The remote controller control unit 122 reads the pulse pattern information corresponding to the command ID to be controlled from the command table of the command database 131, and transmits the information to the learning remote controller 400.

Step S44: The learning type remote controller 400 emits infrared rays of a pulse pattern supplied from the remote controller control unit 122. Since the radiation direction of the learning type remote controller 400 is directed to the home electric appliance 300 to be controlled, the infrared rays reach the home electric appliance 300 to be controlled.

Step S45: The remote control control device 12 uses the image data of the home appliance 300 imaged by the image pickup device 112 as camera information and the voice data generated by the home appliance 300 acquired by the microphone device 111 as the microphone input to the command ID. Correspondingly, each state data table of the state database 133A is written and stored as data after emitting infrared rays (after being instructed by a command).
Further, the remote control control device 12 corresponds to the command ID as the sensor output of the air volume data of the air blown out by the air conditioner acquired from the sensor 14 that detects the air volume arranged at the air outlet of the louver of the air conditioner of the home appliance 300. Then, the state data table of the state database 133A is written and stored as data after emitting infrared rays.

Step S46: The control result determination unit 121A reads out each of the command ID and the state ID corresponding to the control ID of the home appliance 300 to be controlled from the control history data table of the control history database 134. Further, the control result determination unit 121A reads out each of the camera information, the microphone input, and the sensor output corresponding to the state ID from the state database 133A. Then, the control result determination unit 121A includes a command ID, image data, audio data and sensor output of the home appliance 300 before the infrared rays are emitted, and an image of the home appliances 300 after the infrared rays are emitted. Each of the data, the voice data, and the sensor output is input as input data to the control result determination model. As a result, the control result determination model determines whether or not the home electric appliance 300 is operating normally based on the above input data, that is, whether or not it has transitioned to an operating state corresponding to the instruction of the control signal indicated by the command. Estimates and outputs the judgment result of.

Here, for example, in the image data, the looper of the air conditioner is stopped before the infrared rays are emitted, and the looper of the air conditioner is operated after the infrared rays are emitted. It is in a state of being. In addition, the voice data is in a silent state because the looper of the air conditioner is stopped before the infrared rays are radiated, and the looper of the air conditioner is operating after the infrared rays are radiated. Therefore, there is an operating noise of the motor. Further, the sensor output is in a calm state because there is no air blown by the air conditioner before the infrared rays are radiated, and the air conditioner is blown by the air conditioner after the infrared rays are radiated. There is an air volume of.

Step S46: The control result determination unit 121A determines whether or not the home electric appliance 300 is operating based on the estimation result of the control result determination model which is the determination device.
At this time, if the control result determination model determines that the home appliance 300 is operating normally, the control result determination unit 121A proceeds to step S47.
On the other hand, when the control result determination model determines that the home appliance 300 is not operating normally, the control result determination unit 121A advances the process to step S48.

Step S47: The remote control control device 12 notifies the mobile terminal 200 carried by the user at a remote location via the dialogue agent 113 that the home electric appliance 300 is operating normally. At this time, the remote control control device 12 writes and stores "positive" indicating that the home electric appliance 300 is operating normally in the correctness label of the corresponding control ID in the control history database 134 of the control history database 134. Let me.

Step S48: The remote control control device 12 notifies the mobile terminal 200 carried by the user at a remote location via the dialogue agent 113 that the home appliance 300 is not operating normally (abnormal processing). At this time, the remote control control device 12 writes and stores "No" indicating that the home electric appliance 300 is not operating normally in the correctness label of the corresponding control ID in the control history database 134 of the control history database 134. Let me.
Further, when the remote control device 12 is an abnormal process, the process is returned to step S22, a process of radiating infrared rays to the home electric appliance 300 is performed again, and the process is performed to the mobile terminal 200. It may be configured to notify by.

As described above, according to the present embodiment, each of the image data, the voice data, and the sensor output acquired from the home electric appliance 300 operated by the learning type remote control, and the correctness determination for the operation of the user's home electric appliance 300 at this time. Whether the home electric appliance 300 is operating normally by the image data, the voice data, and the sensor output when the user controls the home electric appliance 300 from a remote place by the control result judgment model in which the result of the above is machine-learned as the teacher data. Since it is determined whether or not the home electric appliance 300 is operating normally based on the estimation result of whether or not it is, even if each of the image data, the audio data, and the sensor output fluctuates depending on the surrounding environment, the home electric appliance has high accuracy. Since the operating state of the 300 can be estimated, the robustness of the correctness determination of the home electric appliance 300 can be improved.

Further, if the user determines that the determination model learning unit 123 does not perform accurate estimation even after the control result determination model learning is completed, the image data, audio data, and the image data and audio data used by the determination model learning unit at that time are determined. The control result determination model may be re-learned by adding each of the sensor outputs to the input data and the correct state indicated by the user for the input data to the teacher data.
As a result, the accuracy of estimating the operating state of the home appliance 300 from each of the image data, the voice data, and the sensor output of the control result determination model is improved, and the correctness of the operating state of the home appliance 300 of the control result determination unit 121A is determined. It is possible to improve the accuracy of performing.

<Third embodiment>
Hereinafter, a third embodiment of the present invention will be described with reference to the drawings. The third embodiment has the same configuration as the remote control control system of the second embodiment of the present invention.
The difference between the third embodiment and the second embodiment is that the learning algorithm of the control result determination model by the determination model learning unit 123 is different. Hereinafter, only the points different from the second embodiment will be described.

As the teacher data used for machine learning of the control result determination model, the determination model learning unit 123 in the present embodiment has the command ID and the home appliance 300 at the time before infrared rays are emitted, as in the second embodiment. Each of the image data, the voice data and the sensor output and the image data, the voice data and the sensor output of the home appliance 300 at the time after the infrared rays are emitted are used.

However, as the output data in the teacher data of the control result determination model, for example, the operating state of the louver, the operating sound of the louver, and the air volume from the outlet, which are described in the first embodiment, are preset threshold values set in advance. The judgment result of whether or not it exceeds is used. That is, as the output data in the teacher data, the image data, the voice data, and the sensor output after radiating infrared rays to the home appliance 300 are determined by the threshold value, and the determination result is used. Further, when the determination model learning unit 123 performs machine learning of the control result determination model, each of the command IDs in the command ID list of the device table of the device database 132 is sequentially read, and the respective teacher data are acquired to obtain the above-mentioned teacher data. Machine learning of the control result judgment model is performed.

According to the above-described configuration, according to the present embodiment, it is not necessary for the user to confirm the correctness of the operating state of the home appliance 300 via the dialogue agent 113, and the remote control control system alone can perform machine learning of the control result determination model. , The load on the user when using the remote control control system can be reduced as compared with the second embodiment.
In this embodiment as well, as in the second embodiment, when the user determines that the determination model learning unit 123 does not perform accurate estimation, the image data used by the determination model learning unit at that time, The control result determination model may be re-learned and the threshold value may be adjusted by adding the input data of each of the voice data and the sensor output and the correct state indicated by the user to the input data to the teacher data. As a result, even if the threshold value arbitrarily set in advance is not appropriate, it is possible to acquire an appropriate control result determination model while correcting the threshold value by learning.

Further, a program for realizing the functions of the robot device 11, the remote control device 12, and 12A of the remote control system in each of FIGS. 1 and 7 is recorded on a computer-readable recording medium, and the recording medium is used. By loading the recorded program into a computer system and executing it, control processing of a home appliance or the like using a remote control may be performed. The term "computer system" as used herein includes hardware such as an OS and peripheral devices.

Further, the "computer system" includes a homepage providing environment (or a display environment) if a WWW system is used.
Further, the "computer-readable recording medium" refers to a portable medium such as a flexible disk, a magneto-optical disk, a ROM, or a CD-ROM, or a storage device such as a hard disk built in a computer system. Further, a "computer-readable recording medium" is a communication line for transmitting a program via a network such as the Internet or a communication line such as a telephone line, and dynamically holds the program for a short period of time. In that case, it also includes the one that holds the program for a certain period of time, such as the volatile memory inside the computer system that is the server or client. Further, the above-mentioned program may be a program for realizing a part of the above-mentioned functions, and may be a program for realizing the above-mentioned functions in combination with a program already recorded in the computer system.

Although the embodiments of the present invention have been described in detail with reference to the drawings, the specific configuration is not limited to this embodiment, and design changes and the like within a range not deviating from the gist of the present invention are also included.

11 ... Robot device 12, 12A ... Remote control control device 13, 13A ... Storage device 14, 14_1, _14_n ... Sensor 100 ... Communication network 111 ... Microphone device 112 ... Imaging device 113 ... Dialogue agent 114 ... Speaker device 121, 121A ... Control result Judgment unit 122 ... Remote control control unit 123 ... Judgment model learning unit 131 ... Command database 132 ... Equipment database 133, 133A ... Status database 134 ... Control history database 200 ... Mobile terminal 300, 300_1, 300_m ... Home appliances 400 ... Learning remote control

Claims (6)

A state detection function unit that inputs detection data from a sensor that detects the operating state of home appliances that have received the control signal emitted by the remote controller.
A control result determination unit that determines whether or not the home electric appliance has transitioned to an operating state corresponding to the instruction of the control signal based on the detection data.
A notification function unit that notifies the user of the determination result of the control result determination unit ,
Machine learning is performed using at least the control command indicated by the control signal, the detection data from each of the plurality of sensors for detecting the operating state, and the actual control result when these are used as teacher data. , A determination model that generates a control result determination model that estimates the determination of the control result from the input control command indicated by the control signal and the detection data from each of the plurality of sensors that detect the operating state. With the learning department
With
The control result determination unit receives at least the control command indicated by the control signal and the above-mentioned sensors from each of the plurality of sensors for detecting the operating state at each time before and after the instruction by the control signal of the remote controller. The control result determination model that inputs the detection data is used to estimate the control result of the home electric appliance, and the determination is made based on the estimation result.
Remote control system comprising a call. Further provided with a remote control control unit that controls the remote control to radiate a control signal corresponding to an operation specified by the user.
When the control result determination unit determines that the control result of the home electric appliance does not correspond to the control signal,
The remote control control system according to claim 1, wherein the remote control control unit re-radiates the control signal to the remote control. The remote controller is mounted, and a radiation direction adjustment mechanism for controlling the radiation direction of the control signal is provided.
The remote control control according to claim 2 , wherein the remote control control unit controls the radiation direction adjusting mechanism and adjusts the radiation direction of the control signal of the remote control to the arrangement position of the home electric appliance to be controlled. system. With more dialogue agents to interact with the user
The dialogue agent is characterized in that, by interacting with the user, a command for controlling the home electric appliance of the user is transmitted to the remote control control unit, and the machine learning process is transmitted to the determination model learning unit. The remote control control system according to claim 2 or 3. The state detection process in which the state detection function unit inputs the detection data from the sensor that detects the operating state of the home electric appliance that received the control signal from the remote controller.
A control result determination process in which the control result determination unit determines whether or not the home electric appliance has transitioned to an operating state corresponding to the instruction of the control signal based on the detection data.
A notification process in which the notification function unit notifies the user of the determination result of the control result determination unit.
The determination model learning unit trains at least the control command indicated by the control signal, the detection data from each of the plurality of sensors that detect the operating state, and the actual control result when these are used. Control result determination that estimates the determination of the control result from the control command indicated by the input control signal and the detection data from each of the plurality of sensors that detect the operating state. Judgment model learning process to generate a model
Including
The control result determination unit receives at least the control command indicated by the control signal and the above-mentioned sensors from each of the plurality of sensors for detecting the operating state at each time before and after the instruction by the control signal of the remote controller. The control result determination model that inputs the detection data is used to estimate the control result of the home electric appliance, and the determination is made based on the estimation result.
Remote control control wherein a call. Computer,
A state detection function means for inputting detection data from a sensor that detects the operating state of home appliances that have received a control signal from the remote controller.
A control result determination means for determining whether or not the home electric appliance has transitioned to an operating state corresponding to an instruction of the control signal based on the detection data.
Notification function unit for notifying the determination result of the control result determining unit to a user,
Machine learning is performed using at least the control command indicated by the control signal, the detection data from each of the plurality of sensors for detecting the operating state, and the actual control result when these are used as teacher data. , A determination model that generates a control result determination model that estimates the determination of the control result from the input control command indicated by the control signal and the detection data from each of the plurality of sensors that detect the operating state. Learning means
And then allowed to function,
The control result determining means receives at least the control command indicated by the control signal and the above-mentioned sensors from each of the plurality of sensors for detecting the operating state at each time before and after the instruction by the control signal of the remote controller. The control result determination model that inputs the detection data is used to estimate the control result of the home electric appliance, and the determination is made based on the estimation result.
Program.

Images