WO2020165933A1

WO2020165933A1 - Apparatus control device, apparatus control system, apparatus control method, apparatus control program

Info

Publication number: WO2020165933A1
Application number: PCT/JP2019/004802
Authority: WO
Inventors: 正司比田井; 克希小林
Original assignee: 三菱電機株式会社
Priority date: 2019-02-12
Filing date: 2019-02-12
Publication date: 2020-08-20
Also published as: JPWO2020165933A1; JP6851565B2

Abstract

Provided is an apparatus control device (100) which is a device for controlling the operation of a target apparatus by means of an operation command based on a user operation or a voice command, wherein: when the operation command is input, time information is acquired to generate user specifying information; operation history information (104a) including a plurality of pieces of user operation information is stored by accumulating user operation information obtained by associating the operation command with first user environment information including the user specifying information and the time information, each time the operation command is input; a plurality of pieces of user operation information including the first user environment information satisfying the similar condition are recorded as a pattern command; and when a voice command configured with a fixed-form phrase including one or more words is uttered, a pattern command, which includes the first user environment information matching second user environment information including the user specifying information and the time information regarding the operation command based on the voice command configured with the fixed-form phrase, is selected, and the operation of a target apparatus is controlled on the basis of the selected pattern command.

Description

Device control device, device control system, device control method, and device control program

The present invention relates to a device control device, a device control system, a device control method, and a device control program.

The voice operation information, which is the information in which the phrase indicated by the voice command uttered by the user and the operation of the target device corresponding to the phrase are stored in advance, and the user utters by referring to this voice operation information. There is known a device control device that recognizes a voice command as an operation command for a target device. In addition, as a method of registering voice operation information in the device control device, a user's operation is stored, operation history information made up of the stored operations is referred to, and a voice command corresponding to an operation with high operation frequency is generated and A method of registering has been proposed (for example, see Patent Document 1).

JP, 2010-78851, A (for example, refer to an abstract)

However, in order to input a desired instruction by a voice command to the above-mentioned conventional device control device, the user needs to memorize the phrase indicating the registered voice command. That is, the user needs to remember the word or phrase indicating the voice command corresponding to the desired operation command. Therefore, there is a problem that the user's load required for inputting a voice command is heavy.

The present invention has been made to solve the above-mentioned conventional problems, and makes it possible to reduce the burden on the user required for inputting voice commands, a device control device, a device control system, a device control method, And to provide a device control program.

A device control device according to one aspect of the present invention is a device that controls the operation of a target device by at least one of an operation command based on a user operation input from a user operation unit and an operation command based on a voice command input from a microphone. And a user information indicating a time information acquisition unit that acquires time information including at least one of time and day of the week when the operation command is input, and a user who indicates a user who inputs the user operation and a user who utters the voice command. The operation command inputs user operation information obtained by associating the user identification information generating unit that generates information and the operation command with the first user environment information including the user identification information and the time information. The pattern command stores a plurality of user operation information including first user environment information satisfying a predetermined similarity condition by accumulating the operation history information including a plurality of user operation information. As a pattern command recording process for recording in the database section as above, when the voice command including a fixed phrase including one or more words is uttered, the operation command based on the voice command including the fixed phrase An analysis unit that performs a voice analysis process that selects from the database unit the pattern command that includes the first user environment information that matches the second user environment information that includes the user identification information and the time information; A device control unit that controls the operation of the target device based on the pattern command selected by the analysis unit.

A device control method according to another aspect of the present invention is a method for controlling an operation of a target device by at least one of an operation command based on a user operation input from a user operation unit and an operation command based on a voice command input from a microphone. In addition, a step of acquiring time information including at least one of a time and a day of the week when the operation command is input, and user identification information indicating a user who inputs the user operation and a user who utters the voice command. User operation information obtained by associating the step of generating and the first user environment information including the user identification information and the time information with the operation command is accumulated every time the operation command is input. Accordingly, a step of storing operation history information including a plurality of user operation information in the database unit, and a plurality of user operation information including the first user environment information satisfying a predetermined similarity condition as a pattern command, the database unit And the user identification information and the time information about the operation command based on the voice command including the fixed phrase when the voice command including the fixed phrase including one or more words is uttered. Selecting the pattern command including the first user environment information that matches the second user environment information including the following from the database unit, and controlling the operation of the target device based on the selected pattern command And a step of performing.

According to the present invention, the burden on the user for inputting a voice command can be reduced.

It is a block diagram which shows roughly the structure of the television apparatus and apparatus control apparatus as an apparatus control system which concerns on embodiment of this invention. It is a figure which shows the example of the hardware constitutions of the apparatus control system and apparatus control apparatus which concern on this Embodiment. 7 is a flowchart showing the operation of the device control apparatus according to the present embodiment. 7 is a flowchart showing an operation when recording operation history information in the device control apparatus according to the present embodiment. 7 is a flowchart showing an operation at the time of recording a pattern command in the device control apparatus according to the present embodiment. 7 is a flowchart showing an operation of notifying a user in the device control apparatus according to the present embodiment. 9 is a flowchart showing an operation when a fixed phrase which is a voice command is input in the device control apparatus according to the present embodiment. 7 is a flowchart showing an operation of notifying a user of a confirmation in the device control apparatus according to the present embodiment.

A device control device, a device control system, a device control method, and a device control program according to the embodiments of the present invention will be described below with reference to the drawings. In the following description, the case where the device control system is a television device and the target device is a part of the configuration of the television device will be described. However, the device control system may be a device other than the television device. For example, the device control system may be an electronic device such as a smartphone, a mobile phone, a tablet terminal, a car navigation system, or the like. The following embodiments are merely examples, and various modifications can be made within the scope of the present invention.

<<1>> Overview of Device Control Device The device control device according to the present embodiment uses at least one of an operation command based on a user operation input from a user operation unit and an operation command based on a voice command input from a microphone. It is a device for controlling the operation of.

The device control apparatus according to the present embodiment acquires time information including at least one of the time when the operation command is input and the day of the week. Further, the device control apparatus according to the present embodiment includes a user recognition unit that generates first user identification information indicating a first user who is a user who inputs an operation command from a remote controller, and a user who speaks a voice command. And a voice recognition unit that generates second user identification information indicating the second user. The user recognition unit and the voice recognition unit have a function as a user identification information generation unit. Therefore, the user identification information generation unit generates user identification information indicating a user who inputs a user operation and a user who utters a voice command.

The device control apparatus according to the present embodiment provides the user operation information obtained by associating the operation command with the first user environment information including the user identification information and the time information each time the operation command is input. By accumulating, operation history information including a plurality of user operation information is stored in the database unit.

Further, the device control apparatus according to the present embodiment has a pattern command recording process of recording a plurality of user operation information including first user environment information satisfying a predetermined similarity condition as a pattern command in the database unit, When a voice command consisting of a fixed phrase including one or more words is uttered, a second user environment information including user specifying information and time information about an operation command based on the voice command consisting of a fixed phrase is matched. The voice analysis process of selecting the pattern command including the user environment information of No. 1 from the database unit is performed. Furthermore, the device control apparatus according to the present embodiment controls the operation of the target device based on the selected pattern command.

Here, the “similarity condition” includes, for example, that the user indicated by the first user identification information included in the plurality of user operation information is common. The “similarity condition” may include, for example, that the time indicated by the time information included in the plurality of user operation information is within a predetermined time range. The “similarity condition” may include, as a condition, the day of the week indicated by the time information included in the plurality of user operation information is common.

Also, in the present application, the “fixed phrase” does not include a term indicating the function of the target device. Examples of AutoText are "usual" and so on. The fixed phrase can include or consist only of pronouns. Examples of demonstrative pronouns forming a fixed phrase are "that", "that", and the like.

The “first user environment information that matches the second user environment information” is, for example, that the user identification information included in the second user environment information is the same as the user identification information included in the first user environment information. In addition, the difference between the time indicated by the time information included in the second user environment information and the time indicated by the time information included in the first user environment information is less than or equal to a predetermined threshold value. The “first user environment information that matches the second user environment information” is, for example, that the user identification information included in the second user environment information is the same as the user identification information included in the first user environment information. The day of the week indicated by the time information included in the second user environment information may coincide with the day of the week indicated by the time information included in the first user environment information. The “first user environment information that matches the second user environment information” is, for example, that the user identification information included in the second user environment information is the same as the user identification information included in the first user environment information. In addition, the day of the week indicated by the time information included in the second user environment information matches the day of the week indicated by the time information included in the first user environment information, and is indicated by the time information included in the second user environment information. The difference between the time and the time indicated by the time information included in the first user environment information may be equal to or less than a predetermined threshold value.

By using the device control apparatus 100 according to the present embodiment, the target device can be controlled by a voice command including an ambiguous fixed phrase that does not include the function of the target device.

<<2>> Configuration of the device control apparatus 100 <TV apparatus 10>
FIG. 1 is a block diagram schematically showing a configuration of a television device 10 and a device control device 100 as a device control system according to the present embodiment. As shown in FIG. 1, the device control apparatus 100 includes a device control unit 101, an estimation operation unit 102, an analysis unit 103, a database unit 104 as a storage device, a user recognition unit 105, and a voice recognition unit 106. And a voice synthesis unit 107. The device control device 100 is a device that can implement the device control method according to the first embodiment.

As shown in FIG. 1, the television device 10 includes a microphone 11 as an audio acquisition device, an A/D (Analog to Digital) converter 12, a speaker 13 as an audio output device, and a D/A (Digital). A to analog converter 14, a remote controller 20 as a user operation unit, an input unit 15, a camera 16 as an imaging device, a display unit 17 as a display device, and a device control device 100. The device control unit 101 of the device control apparatus 100 includes a tuner that receives a broadcast wave, a processing circuit that generates video data and audio data based on the received broadcast wave, and the like.

The microphone 11 receives a voice command that is a voice instruction uttered by a speaker who is a user, and generates an analog voice signal corresponding to the voice command. The A/D converter 12 converts the analog audio signal generated by the microphone 11 into a digital audio signal.

The voice recognition unit 106 generates text data corresponding to a voice command from the digital voice signal output from the A/D converter 12. This text data is converted into an operation command based on a voice command by the device control unit 101. The voice recognition unit 106 also analyzes the voice of the user who has issued the voice command, and generates a user ID (that is, a user identifier) that is user identification information that identifies this user. The voice recognition unit 106 outputs the user ID together with the text data to the device control unit 101. A technique for generating a user ID that identifies a user from voice is a known technique called speaker identification. Note that the feature information indicating the feature of the user's voice and the user ID may be registered in advance in a memory provided in the voice recognition unit 106 for speaker identification. The user ID based on the voice command is identification information that identifies the user who uttered the voice command.

The voice synthesis unit 107 synthesizes a digital voice signal based on the text data received from the device control unit 101. This text data is, for example, data of a content notifying the user that input by a voice command is possible.

The D/A converter 14 converts the digital audio signal generated by the audio synthesis unit 107 into an analog audio signal. The speaker 13 outputs a voice based on the analog voice signal. The voice synthesis unit 107, the D/A converter 14, and the speaker 13 form a notification unit that notifies the user of information.

The remote controller 20 is a user operation unit operated by a user. However, the user operation unit may be an operation unit provided in the television device 10. The operation command input from the remote controller 20 is transmitted to the input unit 15 by an infrared signal. The input unit 15 has an infrared sensor, and outputs a user operation signal corresponding to an operation command to the device control unit 101 based on the infrared signal detected by the infrared sensor. The user operation signal is an example of “operation command based on user operation”.

The camera 16 photographs the user who operates the remote controller 20 and provides the user image information to the user recognition unit 105. The user recognition unit 105 generates a user ID, which is user identification information that identifies the user who is operating the remote controller 20, based on the image information acquired from the camera 16. The user recognition unit 105, for example, identifies the user who performs the user operation in conjunction with the output of the key event (that is, the key code) by the remote controller 20, and sends the user ID indicating the user who performs the user operation to the device control unit 101. Output. As a method of identifying the user based on the image information, a known face authentication technique or the like can be used. In order to identify the user, the feature information indicating the feature of the user's face and the user ID may be registered in the memory of the user recognition unit 105 in advance. The user ID is user identification information.

The display unit 17 displays a video based on the video data received from the device control unit 101. The display unit 17 may be used as a notification unit that notifies the user of information.

<Device control unit 101>
The device control unit 101 controls the overall operation of the television device 10. The device control unit 101 receives a key event of the remote controller 20 from the input unit 15 and a user ID, which is identification information of a user who operates the remote controller 20, from the user recognition unit 105. The device control unit 101 also receives, from the voice recognition unit 106, the text data and the user ID that is the identification information of the user who uttered the voice command.

The device control unit 101 receives the key event of the remote controller 20 output from the input unit 15, determines the operation that is the target of the key event from the relation with the screen displayed on the display unit 17 at that time, and determines the key event. Determines whether or not corresponds to the operation command. The criterion for determining whether the key event is an operation command differs depending on the specifications of the device requested by the control target controlled by the device control unit 101.

For example, in a situation where the input screen for registering the postal code is displayed on the screen of the display unit 17 to be controlled, if the number key is pressed on the remote controller 20, the device control unit 101 performs the user operation. It is treated as the input of the postal code, and the pressing of the numeric keys is not treated as the input of the operation command, that is, the operation instruction of the target device until the input of the postal code is completed. When nothing is displayed on the screen of the display unit 17 and the number key of the remote controller 20 is pressed, the device control unit 101 determines that the user presses the number key as the channel selection. It is determined that the user operation is a predetermined operation command. When it is determined that the user operation is a predetermined operation command, the device control unit 101, the user operation signal indicating the operation command, the user ID received from the user recognition unit 105, the time of the user operation, and The time information including at least one of the days of the week is output to the analysis unit 103.

In addition, the device control unit 101 has a function of controlling the operation of the television device 10 based on a user's utterance instruction input from the microphone 11, that is, a voice command that is a voice instruction including a term indicating a function of the target device. I have it. That is, the device control unit 101 stores in advance, in the database unit 104 or the like, voice operation information that is information in which the phrase indicated by the voice command uttered by the user and the operation of the target device corresponding to the phrase are associated with each other. It has a function of recognizing a voice command uttered by the user as an operation command for the target device by referring to the voice operation information. For example, when the voice command of the user input from the microphone 11 is “increase the volume.”, the device control unit 101 generates an operation command according to the content of the utterance instruction, and the operation command (that is, A function of performing control based on an operation command based on a voice command) may be provided. Such control of the target device by the voice command including the term indicating the function of the target device is a known technique as disclosed in Patent Document 1.

Further, in the device control apparatus 100 according to the present embodiment, the device control unit 101 causes the device control unit 101 to perform a voice command including a fixed phrase including one or more words that do not include a term indicating a function, such as “usual”. Can be used as Examples of the fixed phrases that do not include terms indicating functions are, in addition to "usual", pronouns such as "that" and "that". The device control unit 101 previously registers a plurality of texts in a memory in the device control apparatus 100 as candidates for a fixed phrase that does not include a term indicating a function. When the text data corresponding to the input voice command matches any of the registered fixed phrase candidates, the text data is output to the analysis unit 103. The memory that stores the candidate for the fixed phrase may be the memory 201 shown in FIG. 2 described later.

<Analysis unit 103>
In the device control apparatus 100, the analysis unit 103, based on the operation history of the user operation by the remote controller 20 and the operation history by the voice command, a series of operation groups having a high operation frequency for each user environment information including the day of the week or the time. , As a pattern command. The pattern command is information in which a series of operation commands, which is an operation pattern, and user environment information at the time when the user performs the series of operations are collected. In the extraction of the pattern command, the user operation information that is patterned in the operations that are regularly executed, such as the day of the week or the time of day, the operation for each specific period, or the operation performed after a certain fixed operation, is extracted for each user. It is taken out as a pattern command.

The operation command is, for example, a predetermined operation of user operations input from the remote controller 20 and a predetermined voice command of voice commands. The operation command is input from the remote controller 20, through the device control unit 101, or generated by the device control unit 101 based on a voice command and provided to the analysis unit 103.

Further, in the present application, the user environment information indicates the environment of the user at the time when the operation command is output from the device control unit 101, and includes, for example, time information including a day of the week and time, and a user ID. The time may include date information.

Further, in the present application, the operation history includes an operation command and user environment information at the time when the operation command is executed.

Also, in the present application, the pattern command is composed of a group of operation commands. The operation command group is a time-series array of a plurality of operation commands, and also includes user environment information at the time of executing the plurality of operation commands. Since the pattern command is composed of a plurality of operation commands extracted a plurality of times, there is a certain time width between the execution times of the operation commands in the pattern command. Therefore, as the time included in the user environment information associated with the operation command, for example, the average time of the plurality of times included in the plurality of user environment information associated with the plurality of operation commands in the pattern command is used.

The reading (that is, extraction) of the pattern command from the database unit 104 by the analysis unit 103 is periodically performed under the user environment information at a specific day of the week, a specific time of day, or a specific time of day on a specific day of the week. This is the process to be executed. The reading of the pattern command from the database unit 104 by the analysis unit 103 is performed for each user and for each operation command. Under the certain user environment information, the analysis unit 103, when each of the operation commands in the pattern command appears at a certain frequency within a certain time period, each of the series of operation commands has a certain pattern. It is determined that it is an operation command, that is, a pattern command. The analysis unit 103 sets a threshold value of an execution time interval between a series of operation commands and a threshold value of the number of appearances of a pattern command, and uses these as a reference at the time of pattern extraction.

In the present embodiment, the analysis unit 103 performs the detection of the pattern command at the level of the operation command, but all information such as the input of the operation command based on the key event of the remote controller 20 or the voice command is used as the operation command. You may handle and may extract a pattern command based on this operation command. The analysis unit 103 may also extract the pattern command without identifying the user.

The database unit 104 stores the operation command provided by the analysis unit 103 from the device control unit 101, the user environment information when the operation command is executed, and the pattern command extracted by the analysis unit 103.

The analysis unit 103 sequentially records the operation command and the user environment information provided by the device control unit 101 in the database unit 104 as an operation history. The time information included in the user environment information is provided from the time information acquisition unit 101a, for example. In addition, the analysis unit 103 reads the operation history from the database unit 104 at predetermined time intervals, and for each user, from one or more operation commands having similar operation patterns within a predetermined fixed period. The following operation command group is extracted, and this operation command group is registered in the database unit 104 as a pattern command.

The analysis unit 103 holds a certain number of time series arrays as a time series array by associating the operation command output from the device control unit 101 with the user environment information at the time of output. In addition, the analysis unit 103 constantly monitors, for each user, the operation command output from the device control unit 101, and when the operation command group including a series of operation commands and the operation based on the operation command group are executed. Of the user environment information of the pattern command is compared with the user environment information of the pattern command stored in the database unit 104. The analysis unit 103 executes the notification to the user when the operation command group and the user environment information are similar to the operation command and the user environment information of the pattern command stored in the database unit 104. This notification is for indicating to the user that the user operation is performed as a habitual operation over a certain number of times under the same user environment information and the target device recognizes it as a pattern command. This notification is basically made only once. Further, the analysis unit 103 may perform notification on the screen of the display unit 17 instead of or in addition to the notification by voice.

For notification to the user, use a phrase that does not include a term indicating a function, such as "usual operation," to notify that the target device recognizes the operation as a habitual operation. By this, the user can understand that, when instructing a series of operations to the target device thereafter, the user can instruct by a voice command including a fixed phrase such as “usual”. Therefore, the user does not need to individually store the command name unique to the pattern command. Therefore, the operation command can be given by the voice command including the fixed phrase, and the operation load on the user can be reduced.

After this notification, when the user gives an operation instruction using a voice command including a fixed phrase that does not include a term indicating the function of the target device, such as “usual”, the analysis unit 103 outputs from the device control unit 101. The user environment information at the time of receiving this instruction is compared with the user environment information of the pattern command recorded in the database unit 104. The analysis unit 103 detects a pattern command that holds similar user environment information, and outputs the pattern command to the estimation operation unit 102 when the pattern commands match each other with a certain accuracy.

Further, the analysis unit 103 extracts, from the user environment information in the operation command output from the device control unit 101, a pattern command corresponding to the user environment information from the database unit 104. When a part of the preceding stage of the extracted pattern command is similar to the operation command in the time series array and the user environment information, the analysis unit 103 detects that the operation command can be detected as a pattern command to the user each time. The user may be notified. That is, when the current operation command is input by the current user, the analysis unit 103 causes the pattern command having the user environment information stored in the database unit 104 that matches the current user environment information of the current user. Is read, and it is determined whether or not the current operation command is similar to the operation command included in the read pattern command. When it is determined that the operation command is similar to the operation command, The notification unit may be notified that there is a pattern command that can be input. Here, the fact that the current operation command is similar to the operation command included in the read pattern command means that the current operation command and the operation command included in the pattern command are of the same type. This is the case of an operation (for example, channel selection, volume adjustment, etc.).

In the notification to the user, the device control unit 101 implements operation guidance using a fixed phrase that does not include a term indicating a function, such as "usual operation?" Further, in addition to the notification by voice output, the operation guidance may be notified by displaying notification information on the screen of the display unit 17. As a result, even when the user forgets that the operation can be instructed using a voice command consisting of a fixed phrase that does not include a term indicating a function such as "usual", the target device side It is notified using a fixed phrase that does not include a term indicating a function such as "usual operation". Therefore, the operation can be executed with a simple reply such as "usually", and the operation load on the user can be reduced.

<Estimation operation unit 102>
The estimation operation unit 102 estimates the operation command based on the pattern command output from the analysis unit 103, and sequentially instructs the device control unit 101 to execute the estimated operation command. When operating using the pattern command, the estimation operation unit 102 outputs an instruction at a time corresponding to the time of each operation command in the user environment information.

If the user operates the remote controller 20, that is, an interrupt operation while the estimation operation unit 102 sequentially instructs the execution of the estimated operation command corresponding to the pattern command, the device control apparatus 100 determines that the interrupt operation is performed. The intent of the user who performed is estimated. That is, the estimation operation unit 102 determines whether the pattern command is different from what the user intended. This determination is made for each operation command of the pattern command by registering a remote control operation that rejects the operation command in advance and referring to this registration information. For example, the estimation operation unit 102 does not regard a volume up/down operation, a fast forward/rewind operation, or the like when playing a recorded program as a remote control operation that rejects the operation command corresponding to the selected pattern command. be able to.

As a result, when the user utters a voice command including a fixed phrase that does not include a term indicating a function such as “usual”, and the pattern command estimated by the estimation operation unit 102 is being executed, When the operation is interrupted, the device control apparatus 100 can determine whether the operation is an additional operation in the series of operations of the pattern command or the user wants to suspend the estimated pattern command. Therefore, when the user does not have the intention of rejecting the selected pattern command, the execution of the process based on the pattern command is not stopped contrary to the user's wish, and the operation load on the user can be reduced.

<Hardware configuration>
FIG. 2 is a diagram showing an example of the hardware configuration of the device control apparatus 100 according to the present embodiment. 2, the same components as those shown in FIG. 1 are designated by the same reference numerals as those shown in FIG. As shown in FIG. 2, the device control apparatus 100 includes a memory 201 such as a flash memory, a processor 202 as an information processing unit, and a storage device 203. The memory 201 includes a flash ROM (Read Only Memory) and a RAM (Random Access Memory). The device control apparatus 100 is, for example, a part of an electronic device. The device control apparatus 100 may be a computer. The memory 201 can record an OS (Operating System) executed by the processor 202, application programs such as the device control program according to the present embodiment, firmware, and the like. The storage device 203 is composed of, for example, an HDD (Hard Disk Drive) or a semiconductor storage device.

The whole or part of the device control apparatus 100 shown in FIG. 1 can be realized by using a memory 201 that stores a program as software and a processor 202 that executes the program stored in the memory 201. The database unit 104 in FIG. 1 can be realized by the storage device 203 in FIG.

<<3>> Operation of Device Control Device 100 FIG. 3 is a flowchart showing an operation of the device control device 100 according to the present embodiment. The device control apparatus 100 according to the present embodiment can control the operation of the target device based on the operation command input from the remote controller 20 which is the user operation unit and acquired by the input unit 15. Further, the device control apparatus 100 can control the operation of the target device based on the voice command input from the microphone 11. In the device control apparatus 100 according to the present embodiment, as the voice command, not only a voice command by a specific command statement including a term indicating a function of the target device but also an ambiguous not including a term indicating a function of the target device. A voice command consisting of a phrase that is an expression can be used. Examples of specific command statements including terms indicating the function of the target device are "please increase the volume", "change to channel 4", and the like. Examples of fixed phrases that are ambiguous expressions that do not include terms that describe the function of the target device are phrases such as "usual" and "usual guy", and demonstrative pronouns such as "that" and "that". ..

Main operations of the device control apparatus 100 according to the present embodiment will be described based on FIG. First, in step S1, when an operation command based on a user operation by the remote controller 20 is input or an operation command based on a voice command is input, the analysis unit 103 sends an operation command and an operation command through the device control unit 101. The user operation information obtained by receiving the user ID indicating the input user and the time information including the time and the day of the week when the operation command is input, and associating the operation command, the user ID, and the time information with each other, The operation history information 104a is stored in the database unit 104. The analysis unit 103 performs an operation of accumulating user operation information each time an operation command is input.

In the next step S2, the analysis unit 103 extracts the user operation information from the operation history information 104a recorded in the database unit 104, and sets a predetermined similar condition among the plurality of extracted user operation information. The satisfying user operation information is recorded in the database unit 104 as a pattern command.

The predetermined similar condition is, for example, any of the following (1) to (3).
(1) The user IDs are common, and the time difference included in the time information is within a predetermined range.
(2) The user ID is common and the day of the week included in the time information is common.
(3) The user ID is common, the day of the week included in the time information is common, and the time difference included in the time information is within a predetermined range.

In the next step S3, the device control unit 101 notifies the user that the input operation of the pattern command can be executed by uttering a voice command including an ambiguous fixed phrase that is a voice command. The notification is performed by the synthetic voice output from the speaker 13 based on the synthetic voice signal generated by the voice synthesis unit 107. Further, the notification may be performed by the video displayed on the display unit 17.

In the next step S4, when a voice command including an ambiguous fixed phrase that is a voice command is uttered, the analysis unit 103 reads a pattern command corresponding to the voice command including the uttered fixed phrase from the database unit 104. The estimation operation unit 102 estimates an operation command corresponding to this pattern command, and provides the estimated pattern command to the device control unit 101. The device control unit 101 controls the target device according to the pattern command received from the estimation operation unit 102.

FIG. 4 is a flowchart showing an operation when recording operation history information in the device control apparatus 100 according to the present embodiment. FIG. 4 shows details of the process of step S1 in FIG.

First, in step S11, when the user operates the remote controller 20, the device control unit 101 receives a user operation signal input via the input unit 15.

In the next step S12, the device control section 101 determines whether or not the received user operation signal is a signal based on a predetermined operation command. That is, the device control unit 101 determines whether or not the user operation signal is an operation command to be recorded as the operation history information 104a, that is, a signal based on a predetermined operation command.

When the user operation signal is a signal based on a predetermined operation command (when the determination in step S12 is “Yes”), the process proceeds to step S13, and the user operation signal is a signal based on the predetermined operation command. If not (when the determination in step S12 is “No”), the process returns to step S11.

When the process proceeds to step S13, the analysis unit 103 receives the operation command and the user environment information including the user ID and the time information from the device control unit 101.

In the next step S14, the analysis unit 103 records the user operation information obtained by associating the operation command with the user environment information in the database unit 104 as the operation history information 104a. That is, the operation history information 104a recorded in the database unit 104 may be composed only of the user operation information obtained by associating the operation command based on the user operation with the user environment information. However, the operation history information 104a recorded in the database unit 104 associates the user operation information obtained by associating the operation command based on the user operation with the user environment information, and the operation command based on the voice command and the user environment information. It is desirable to include both the user operation information obtained by the above. It should be noted that the operation history information 104a recorded in the database unit 104 can be composed only of the user operation information obtained by associating the operation command based on the voice command with the user environment information.

FIG. 5 is a flowchart showing an operation at the time of recording a pattern command in the device control apparatus 100 according to this embodiment. FIG. 5 shows details of the process of step S2 in FIG. The pattern command is a command that can be input by an ambiguous fixed phrase that is a voice command that does not include a term indicating the function of the target device.

First, in step S21, the analysis unit 103 refers to the database unit 104 every predetermined time and determines whether or not the operation history information 104a exists in the database unit 104.

When the operation history information 104a is present (when the determination in step S21 is “Yes”), the process proceeds to step S22, and when the operation history information 104a is not present (when the determination in step S21 is “no”), the step The process of S21 is performed again.

When the process proceeds to step S22, the analysis unit 103 reads the user operation information, which is the operation history within a predetermined fixed period, from the operation history information 104a recorded in the database unit 104.

In the next step S23, the analysis unit 103 determines whether the read user operation information satisfies the condition of the pattern command. That is, in the analysis unit 103, the read user operation information is the user operation information regarding the user operation repeatedly executed by the user, the user operation information includes similar user environment information, and the number of times of repeated execution is It is determined whether or not the command is a pattern command including a plurality of pieces of user operation information (that is, a group of user operation information) regarding user operations exceeding a predetermined reference number of times.

When the user operation information group satisfies the condition of the pattern command (when the determination in step S23 is “Yes”), the process proceeds to step S24, and when the user operation information group does not satisfy the condition of the pattern command (in step S23). If the determination is “no”), the process returns to step S21.

When the process proceeds to step S24, the analysis unit 103 records a series of user operation information groups and the user ID and time information, which is the user environment information, as a pattern command in the database unit 104.

FIG. 6 is a flowchart showing the operation of notifying the user in the device control apparatus 100 according to the present embodiment. FIG. 6 shows details of step S3 in FIG.

First, in step S31, when the user operates the remote controller 20, the device control unit 101 receives a user operation signal input via the input unit 15.

In the next step S32, the device control section 101 determines whether the received user operation signal is a signal based on a predetermined operation command.

When the user operation signal is a signal based on a predetermined operation command (when the determination in step S32 is “Yes”), the process proceeds to step S33, and the user operation signal is a signal based on the predetermined operation command. If not (when the determination in step S32 is “No”), the process returns to step S31.

When the process proceeds to step S33, the analysis unit 103 receives the operation command and the user environment information including the user ID and the time information from the device control unit 101.

In the next step S34, the analysis unit 103 reads the pattern command from the database unit 104.

In the next step S35, the analysis unit 103 refers to the read pattern command, and determines whether or not there is a pattern command including the user ID of the input operation command at the current time point, the user ID that matches the time information, and the time information. To judge. That is, the analysis unit 103 determines whether or not there is a pattern command including user environment information that matches the user environment information of the operation command at the present time.

If there is a user ID of the operation command, a user ID that matches the time information, and a pattern command that includes the time information (that is, if the determination in step S35 is “Yes”), the process proceeds to step S36, and the user of the operation command. If there is no pattern command including the ID, the user ID that matches the time information, and the time information (that is, the determination in step S35 is “No”), the process returns to step S31.

When the process proceeds to step S36, the analysis unit 103 determines whether the current operation command is similar to the pattern command.

When the operation command is similar to the pattern command (that is, when the determination in step S36 is "Yes"), the process proceeds to step S37, and when the operation command is not similar to the pattern command (that is, step S36). If the determination is NO, the process returns to step S31.

When the process proceeds to step S37, the analysis unit 103 causes the device control unit 101 to execute notification by synthetic voice. That is, the analysis unit 103 instructs the device control unit 101 to notify the user of “usual operation” with synthetic voice.

FIG. 7 is a flowchart showing the operation of the device control apparatus 100 according to the present embodiment when a fixed phrase that is a voice command is input. FIG. 6 shows details of step S4 in FIG.

First, in step S41, when the user utters a fixed phrase that is a voice command, the device control unit 101 outputs text data corresponding to the voice command via the microphone 11, the A/D converter 12, and the voice recognition unit 106. receive. The fixed phrase is a voice command that does not include a term indicating a function to be controlled, and is, for example, “usual”, “that”, or the like.

In the next step S42, the analysis unit 103 receives the text data of the fixed phrase from the device control unit 101.

In the next step S43, the analysis unit 103 reads the pattern command from the database unit 104.

In the next step S44, the analysis unit 103 refers to the read pattern command, and determines whether or not there is a pattern command including the user ID of the input voice command at the current time point, the user ID that matches the time information, and the time information. To judge. That is, the analysis unit 103 determines whether or not there is a pattern command including user environment information that matches the user environment information of the voice command at the present time.

If there is a user ID of the voice command, a user ID that matches the time information, and a pattern command that includes the time information (that is, if the determination in step S44 is “Yes”), the process proceeds to step S45, and the user of the voice command. When there is no pattern command including the ID, the user ID that matches the time information, and the time information (that is, the determination in step S44 is “No”), the process returns to step S41.

When the process proceeds to step S45, the analysis unit 103 outputs the user ID of the voice command, the user ID that matches the time information, and the pattern command including the time information to the estimation operation unit 102. That is, the analysis unit 103 outputs the pattern command including the user environment information that matches the user environment information of the voice command to the estimation operation unit 102.

In the next step S46, the estimation operation unit 102 outputs an operation command corresponding to the received pattern command to the device control unit 101. In the next step S47, the device control section 101 controls the operation of the target device based on the operation command received from the estimation operation section 102.

FIG. 8 is a flowchart showing the processing of notification of confirmation of a pattern command in the device control apparatus 100 according to the present embodiment.

First, in step S51, when the user operates the remote controller 20, the device control unit 101 receives a user operation signal input via the input unit 15.

In the next step S52, the device control section 101 determines whether or not the received user operation signal is a signal based on a predetermined operation command.

When the user operation signal is a signal based on a predetermined operation command (that is, when the determination in step S52 is “Yes”), the process proceeds to step S53, and the user operation signal is changed to the predetermined operation command. When the signal is not the base signal (that is, the determination in step S52 is “No”), the process returns to step S51.

When the process proceeds to step S53, the analysis unit 103 receives the operation command and the user environment information including the user ID and the time information from the device control unit 101.

In the next step S54, the analysis unit 103 reads the pattern command from the database unit 104.

In the next step S55, the analysis unit 103 refers to the read pattern command, and determines whether or not there is a pattern command including the user ID of the input operation command at the current time point, the user ID that matches the time information, and the time information. To judge. That is, the analysis unit 103 determines whether or not there is a pattern command including user environment information that matches the user environment information of the operation command at the present time.

If there is a user ID of the operation command, a user ID that matches the time information, and a pattern command including the time information (that is, if the determination in step S55 is “Yes”), the process proceeds to step S56, and the user of the operation command. When there is no pattern command including the ID, the user ID that matches the time information, and the time information (that is, the determination in step S55 is “No”), the process returns to step S51.

When the process proceeds to step S56, the analysis unit 103 determines whether the current operation command is similar to a part of the previous stage of the pattern command. That is, when the current operation command is input by the current user, the analysis unit 103 causes the pattern command having the user environment information stored in the database unit 104 that matches the current user environment information of the current user. Is read and it is determined whether or not the current operation command is similar to the operation command included in the read pattern command.

When the operation command is a part of the preceding stage of the pattern command, that is, when the operation command is similar to the operation command included in the pattern command (that is, when the determination in step S56 is “Yes”), the process proceeds to step S57, and the operation is performed. When the command is not similar to the part of the previous stage of the pattern command (that is, when the determination in step S56 is “No”), the process returns to step S51.

When the process proceeds to step S57, the analysis unit 103 causes the device control unit 101 to execute notification by synthetic voice. In other words, the analysis unit 103 instructs the device control unit 101 to notify the user of the confirmation "Is it the usual operation?" with synthetic voice.

<<4>> Effect of Device Control Device 100 As described above, if the device control device 100, the device control system 10, the device control method, or the device control program according to the present embodiment is used, user input from the remote controller 20 is performed. In addition to the operation and the voice input operation by a predetermined voice command, the target device can be controlled by a voice command that is an ambiguous fixed phrase that does not include the function of the target device.

In addition, since fixed phrases such as “usual”, “sakino”, and demonstrative pronouns such as “that” and “that” can be used as the fixed phrases that are voice commands, the user can You don't have to remember the contents. Therefore, the burden on the user can be reduced.

10 TV device, 11 microphone, 12 A/D converter, 13 speaker, 14 D/A converter, 15 input unit, 16 camera, 20 remote controller, 100 device control unit, 101 device control unit, 101a time information generation unit, 102 estimation Operation part, 103 analysis part, database part, 104a operation history information, 105 user recognition part, 106 voice recognition part, 107 voice synthesis part.

Claims

A device control device for controlling the operation of a target device by at least one of an operation command based on a user operation input from a user operation unit and an operation command based on a voice command input from a microphone,
A time information acquisition unit that acquires time information including at least one of time and day of the week when the operation command is input,
A user identification information generation unit that generates user identification information indicating a user who inputs the user operation and a user who speaks the voice command;
By accumulating the user operation information obtained by associating the operation command with the first user environment information including the user identification information and the time information, each time the operation command is input, a plurality of user operation information is accumulated. A database unit that stores operation history information including user operation information,
A pattern command recording process of recording a plurality of user operation information including first user environment information satisfying a predetermined similarity condition as a pattern command in the database unit, and the voice composed of a fixed phrase including one or more words. When the command is uttered, the first user environment information that matches the second user environment information including the user identification information and the time information about the operation command based on the voice command including the fixed phrase A voice analysis process for selecting the pattern command from the database unit including
A device control unit that controls the operation of the target device based on the pattern command selected by the analysis unit;
An apparatus control device comprising:
The device control apparatus according to claim 1, wherein the similarity condition includes that the user indicated by the user identification information of the plurality of user operation information is common.
The device control apparatus according to claim 1 or 2, wherein the similar condition includes that a time indicated by the time information of the plurality of user operation information is within a predetermined time range.
The device control apparatus according to any one of claims 1 to 3, wherein the similar condition includes that a day of the week indicated by the time information of the plurality of user operation information is common.
The device control device according to any one of claims 1 to 4, wherein the fixed phrase does not include a term indicating a function of the target device.
The device control device according to any one of claims 1 to 5, wherein the fixed phrase includes or is composed only of a demonstrative pronoun.
The time at which the user identification information included in the second user environment information matches the user identification information included in the first user environment information and indicated by the time information included in the second user environment information And the difference between the time indicated by the time information included in the first user environment information is less than or equal to a predetermined threshold value, the second user environment information matches the first user environment information. The device control device according to claim 1, wherein the device control device is a device control device.
The day of the week in which the user identification information included in the second user environment information matches the user identification information included in the first user environment information and which is indicated by the time information included in the second user environment information The second user environment information matches the first user environment information when is coincident with the day of the week indicated by the time information included in the first user environment information. 6. The device control device according to any one of 6 above.
The day of the week in which the user identification information included in the second user environment information matches the user identification information included in the first user environment information and which is indicated by the time information included in the second user environment information Corresponds to the day of the week indicated by the time information included in the first user environment information, and the time indicated by the time information included in the second user environment information and the time included in the first user environment information 7. The second user environment information matches the first user environment information when the difference from the time indicated by the time information is less than or equal to a predetermined threshold value. The device control device according to item 1.
When the pattern command recording process is performed, a notification unit that notifies that the operation command that can be input by the voice command including the fixed phrase is present in the pattern command is further included. Item 10. The device control device according to any one of items 1 to 9.
When a current operation command, which is either an operation command based on the user operation by the current user or an operation command based on the voice command by the current user, is input,
The analysis unit is
Reading the pattern command having the first user environment information stored in the database unit that matches the current user environment information of the current user,
The current operation command is determined whether it is similar to the operation command included in the read pattern command,
The device according to claim 10, wherein, when it is determined that they are similar to each other, the notification unit is notified that there is a pattern command that can be input by the voice command including the fixed phrase. Control device.
When the operation based on the pattern command is being executed by the input of the voice command including the fixed phrase, and when the current operation command is input by the current user, the analysis unit causes the fixed phrase to be input. It is determined whether the process based on the voice command is a process added to the operation based on the pattern command or a process of interrupting and executing the operation based on the pattern command. Item 11. The device control device according to item 11.
A device control apparatus according to any one of claims 1 to 12,
A target device controlled by the device control device,
A device control system comprising:
A device control method for controlling an operation of a target device by at least one of an operation command based on a user operation input from a user operation unit and an operation command based on a voice command input from a microphone,
A step of acquiring time information including at least one of time and day of the week when the operation command is input;
Generating user identification information indicating a user who inputs the user operation and a user who utters the voice command;
By accumulating the user operation information obtained by associating the operation command with the first user environment information including the user identification information and the time information, each time the operation command is input, a plurality of user operation information is accumulated. Storing operation history information including user operation information in the database unit;
Recording a plurality of user operation information including first user environment information satisfying a predetermined similarity condition as a pattern command in the database unit;
A second voice information including the user identification information and the time information about the operation command based on the voice command including the fixed phrase when the voice command including the fixed phrase including one or more words is uttered. Selecting the pattern command including the first user environment information that matches the user environment information from the database unit;
Controlling the operation of the target device based on the selected pattern command;
An apparatus control method comprising:
A device control program for controlling the operation of a target device by at least one of an operation command based on a user operation input from a user operation unit and an operation command based on a voice command input from a microphone,
On the computer,
A process of acquiring time information including at least one of time and day of the week when the operation command is input;
A process of generating user identification information indicating a user who inputs the user operation and a user who utters the voice command;
By accumulating the user operation information obtained by associating the operation command with the first user environment information including the user identification information and the time information, each time the operation command is input, a plurality of user operation information is accumulated. A process of storing operation history information including user operation information in the database unit;
A process of recording a plurality of user operation information including first user environment information satisfying a predetermined similar condition in the database unit as a pattern command;
A second voice information including the user identification information and the time information about the operation command based on the voice command including the fixed phrase when the voice command including the fixed phrase including one or more words is uttered. A process of selecting the pattern command including the first user environment information that matches the user environment information from the database unit;
A process of controlling the operation of the target device based on the selected pattern command;
A device control program for executing the following.