JP2020101822A - Information providing method using voice recognition function, and control method of instrument - Google Patents

Abstract

To provide a new information providing method using voice recognition processing.SOLUTION: An information providing method in an information providing system is connected to a display instrument having a display and a voice input device capable of inputting user's voice, and provides information in response to the user's voice via the display instrument. Display screen information, which allows a display screen including a plurality of selectable items to be displayed on a display of the display instrument, is transmitted to the display instrument. On the display screen of the display, item selection information indicating that one item in the plurality of items was selected is received, and in the case where voice instruction including first voice information expressing instruction content is received from the voice input device when one item is selected, the instruction content is recognized from the first voice information, whether or not the voice instruction includes second voice information indicating an instruction word is determined, and in the case of determination that the second voice information is included in the voice instruction, the instruction content regarding one item is executed.SELECTED DRAWING: Figure 1

Description

The present disclosure relates to an information providing method and a device control method using a voice recognition function.

2. Description of the Related Art Conventionally, there is an apparatus that controls a device by receiving a voice with a microphone (hereinafter, also referred to as a “microphone”), recognizing the received voice, and interpreting the content. Such a microphone may be connected to the device or may be built in an input device (for example, a remote controller (hereinafter, may be referred to as a “remote control”)) attached to the device. By controlling the device by voice, it is possible to provide the user with unprecedented convenience such as turning the power on and off and controlling the device collectively.

Control commands for controlling the device include control commands suitable for input by voice recognition and control commands not suitable for input. Therefore, device control using a multimodal input method in which both an input device such as a remote controller and voice are combined is desirable. Patent Document 1 discloses a device control method that combines a remote control and voice recognition.

JP, 2004-260544, A

In the device control method using the voice recognition function described above, further improvement was necessary for practical use.

In order to solve the above problems, a method according to one embodiment of the present invention is connected to a display device having a display and a voice input device capable of inputting a user's voice, and responds to the user's voice via the display device. A method for providing information in an information providing system for providing information by transmitting display screen information for causing a display of a display device to display a display screen including a plurality of selectable items to the display device, , The item selection information indicating that one of the plurality of items is selected, and when the one item is selected, a voice including the first voice information indicating the instruction content from the voice input device. When the instruction is received, the instruction content is recognized from the first voice information, it is determined whether or not the voice instruction includes the second voice information indicating the instruction word, and the voice instruction includes the second voice information. If it is determined that there is one, the instruction content is executed for one item.

A method according to another aspect of the present invention is a method for controlling a display device having a display, which is connected to a voice input device capable of inputting a user's voice, wherein a computer of the display device is provided with a plurality of selectable items. The display screen including the display screen is displayed, and when one of the multiple items is selected on the display screen of the display, and when the selection of one item is detected, When the voice instruction including the first voice information indicating the instruction content is received from the input device, the instruction content is recognized from the first voice information, the instruction content is executed, and it is detected that the one item is selected. If not, or if it is determined that the instruction content cannot be executed, the voice instruction is transmitted to another computer.

According to each of the above-mentioned aspects, since the access between the server and the client that occurs during device control is reduced, the operability is improved.

Each of the above aspects can be implemented using a system, a device, or a computer program. Alternatively, it may be realized by a combination of the system, the device, and the computer program.

With the above aspect, further improvement can be realized.

FIG. 6 is a sequence diagram showing an outline of processing in Exemplary Embodiment 1. FIG. 3 is a diagram showing a configuration of an information presentation method using a voice recognition function according to an exemplary embodiment 1. FIG. 11 is a diagram showing a first sequence showing communication processing between a server and a client in the exemplary embodiment 1. FIG. 6 is a diagram showing processing in a server in the exemplary embodiment 1. FIG. 6 is a diagram showing processing in a client in the exemplary embodiment 1. It is a figure which shows the example which specifies the position on a map. It is a 1st figure which shows the example which designates the person position on a screen. It is a 2nd figure which shows the example which specifies the person position on a screen. It is a 1st figure which shows the example of the search based on the position on a map. It is a 2nd figure which shows the example of the search based on the position on a map. FIG. 11 is a diagram showing a second sequence showing communication processing between the server and the client in the exemplary embodiment 1. FIG. 9 is a first sequence diagram illustrating an outline of processing according to the first exemplary embodiment. FIG. 11 is a second sequence diagram showing an outline of processing according to the first exemplary embodiment. It is a figure which shows the structure of the information presentation method using the audio|voice recognition function by exemplary Embodiment 2. FIG. 16 is a diagram showing a sequence showing communication processing between a server and a client in the exemplary second embodiment. FIG. 9 is a diagram showing processing in a server in the exemplary second embodiment. FIG. 11 is a diagram showing processing by a client in the exemplary second embodiment. It is a figure which shows the example which displays the program content from a recommended program list. It is a figure which shows the structure of the information presentation method using the conventional voice recognition function.

(Findings that form the basis of the present invention)
The knowledge on which the present invention is based is as follows.

The inventors of the present application considered that further improvement is necessary for practical application of a device that controls a device by receiving voice with a microphone, recognizing the received voice, and interpreting the content.

In controlling the device by voice, by assigning a plurality of control commands to one voice command, the device can be controlled with simple words. There is an advantage that even a user who is unfamiliar with the operation of the remote controller having a large number of buttons can control the device with natural voice.

On the other hand, performing all operations by voice impairs the operability of the user. This will be described using a television (TV) as an example.

FIG. 16 is a diagram showing an example of a screen displayed on the television. For example, it is assumed that a program list 901 is displayed on the screen as shown in FIG. 16 by a voice command of “list of recommended programs”. This is because, for example, the TV receives the voice of the user through the remote controller 902, recognizes the word “recommended program list” (that is, the voice), and interprets the content, so that the device (that is, the TV) gives the user the user. It is a function to present the recommended programs combined. Here, in order to specify the program, the user utters a cursor movement command such as "down" or "up" by voice.

When many recommended programs are displayed, the number of programs displayed on one screen increases. The display screen may span multiple pages. In such a case, in order to specify the program, the user needs to utter a number of cursor movement commands such as “down”, “up”, “next page”, or “previous page”. .. When the voice is repeatedly input, there is a high possibility that the voice recognition will be mistaken. This way of saying the same words over and over is not easy to use.

Regarding such a problem, for example, Patent Document 1 discloses a voice recognition method capable of easily operating a television by combining a remote control and voice recognition.

In this conventional method, when the recommended program list is displayed by the voice command as described above, the user first specifies the program with the remote controller. After that, the user inputs a voice composed of a pair of a demonstrative pronoun (sometimes referred to as “instruction word” or “instruction character string”) and a word (that is, instruction content) for controlling the designated program. By doing so, the program specified by the remote controller is controlled. For example, when the user designates a program with the remote controller 902 while the program list 901 is displayed, the screen state changes to the screen state 903 in which it can be seen that the program is selected. After that, when the user utters "display the contents", the contents of the program designated by the remote controller are displayed as in the program contents display screen 904. In this example, “the” corresponds to the instruction word, and “display content” corresponds to the instruction content. In this specification, the voice information indicating the instruction content may be referred to as “first voice information”, and the voice information indicating the instruction word may be referred to as “second voice information”.

FIG. 17 shows a configuration example of a program information presentation device 1000 that realizes the conventional voice recognition method described in Patent Document 1. In FIG. 17, voice is input by the microphone 1001 and voice recognition is performed by the voice recognition unit 1002. The instruction character string detection unit 1003 extracts an instruction character string from the voice recognition result. The voice synthesis unit 1004 generates a synthetic voice for responding to the user with a voice. The control signal generation processing 1005 generates a signal for controlling the device. The input device 1006 is composed of a mouse, a touch panel, a keyboard, a remote controller, and the like. The input device 1006 is used by the user to select one program from a plurality of programs when information on a plurality of programs is displayed. The input device 1006 receives information on the selected position when one program is selected by the user from a plurality of programs displayed on the screen. The output unit 1007 performs an output process of displaying the selected program, a device control based on the signal generated by the control signal generation process, a control result display, a reproduction process of the synthesized voice generated by the voice synthesis process, and the like. I do.

When using voice commands instead of the buttons provided on the remote control, the number and types of words generated are limited to the number of buttons. Therefore, the name written on the button of the remote controller or the voice command corresponding to the button may be registered in advance as a dictionary for recognition. For each word registered in the dictionary, voices of various people having different ages or sexes are collected to construct an acoustic model and a language model for voice recognition. In order to reduce the false recognition, some ingenuity may be made such as manually customizing the recognition dictionary or model.

However, with home appliances now connected to networks outside the home, it has become possible to obtain program information from the web and search the web using a TV screen. In this case, words that are not related to TV may be input, and it is difficult to know in advance what words will be input. That is, it is impossible to prepare an acoustic model and a language model specialized for a predetermined word group. As a result, the voice recognition accuracy becomes low, and it becomes difficult for the user to input the vocabulary desired by the voice.

In order to recognize words other than words written on the remote controller with high accuracy, it is necessary to construct a model for voice recognition using a large-scale data group. By constructing a statistical speech recognition model using a large-scale data group, even unknown words can be recognized with high accuracy in advance. The speech recognition process based on the statistical model is executed on a server computer (hereinafter, may be simply referred to as “server”) connected to a device via a network because the resources required for memory and calculation amount are large. It

In the technique disclosed in Patent Document 1, the device body to be controlled and the voice recognition processing unit are integrated. Therefore, a voice recognition dictionary can be prepared in advance for the contents described in the remote controller controlling the device body. However, the accuracy of voice recognition was low for free utterances such as web searches. Therefore, the user often finds it difficult to use, and has been forced to limit the usage range of voice recognition.

From the above consideration, it is practically preferable that the server performs the voice recognition process of the voice signal received by the device. However, in the case of performing voice recognition processing via a network, it takes a long time from transmitting a voice signal to returning a response. That is, there is a problem that processing delay occurs.

As an example of a system in which this problem occurs, assume a system that performs a voice recognition process, detects an instruction character string from the recognition result, and then returns a voice response or a control signal according to the detection result of the instruction character string. When the voice recognition process is executed by the server, the server performs a series of processes including the voice recognition process, the detection of the instruction character string, the voice response based on the recognition result, and the device control. In this case, each time the instruction character string is detected in the voice recognition result, the server accesses the device that is the client. This is to inquire what the target item indicated by the instruction character string (for example, “the” or the like) is. As a result, the subsequent processing cannot be performed until the communication processing between the server and the client is completed. Therefore, a processing delay may occur. In this system, it is required to reduce the processing delay caused by accessing the client from the server each time the instruction character string is detected. However, no consideration has been given to the technical solution to meet this requirement.

One aspect of a device control method using the voice recognition function for solving such a problem is an input process for receiving an input from a user, and whether or not a part of the screen is designated by the input process. State detection processing for detecting the state of the selected item, selection information detection processing for obtaining internal information regarding the position on the screen of the selected one item, output processing for returning a response to the user, and communication processing for communicating with an external device. And a voice input process for inputting voice, a voice recognition process for recognizing voice, an instruction character string detection process for detecting an instruction character string based on a voice recognition result, and a state of item selection by a user are managed. Including the selection state management process, a server different from the control target device was made to execute the voice input process, the voice recognition process, the instruction character string detection process, and the selection state management process, and the selection state was changed in the selection state detection process. The selection character string detection process acquires the selection information detected by the selection information detection process only when the state of the selection state management process is updated each time it is detected.

By the selection state management process, the server holds information regarding the state of whether or not one item (for example, an item indicating a program) is selected by the input device. Therefore, when voice recognition processing is performed on the server, it is possible to select whether or not to access the client from the server according to the state held on the server. As a result, processing delay can be reduced.

The device control method described above may further include a dialog management process and a response sentence generation process, and the device may be controlled by a process interactive with the user.

The above-described device control method further includes a voice synthesis process and a control signal generation process, and when a response is returned to the user in the output process, a response is returned with a synthesized voice, or a device control is performed with the generated control signal. By doing so, a response may be returned to the user.

The selection state management process may manage only the state in which a part of the screen is selected in the input process.

The selection state management process may also manage internal information corresponding to the selected location, in addition to the state in which a part of the screen is selected in the input process.

The input process may specify either metadata about the television program or the content of the television program.

The metadata regarding the TV program may be any of the program name, channel name, content, attention level, and recommendation level.

The content of the television program may include any one of a person, an animal, a car, a map, letters, and numbers.

Further, according to one aspect of an information providing method for solving the above-mentioned problem, a display device having a display and a voice input device capable of inputting a user's voice are connected, and in response to the user's voice, via the display device. A method for providing information in an information providing system for providing information by transmitting display screen information for causing a display of a display device to display a display screen including a plurality of selectable items to the display device, , The item selection information indicating that one of the plurality of items is selected, and when the one item is selected, a voice including the first voice information indicating the instruction content from the voice input device. When the instruction is received, the instruction content is recognized from the first voice information, it is determined whether or not the voice instruction includes the second voice information indicating the instruction word, and the voice instruction includes the second voice information. If it is determined that there is one, the instruction content for one item is executed.

The instruction content is an instruction to retrieve information related to one item, and the user may be notified of a search result based on the instruction content.

The search result information for displaying the search result on the display may be transmitted to the display device.

The information providing system may be further connected to a voice output device capable of outputting voice, and may transmit search result information that causes the search result to be output as voice from the voice output device, to the voice output device.

The plurality of items may be items indicating metadata about the television program or contents of the television program.

The metadata may indicate at least one of a television program name, a channel name, an outline of the television program, a degree of attention of the television program, and a degree of recommendation of the television program.

The content of the television program may include information indicating at least one of a person, an animal, a car, a map, characters, and numbers.

The display screen represents a map in a specific area, and each of the plurality of items may be an arbitrary coordinate on the map or an object on the map.

The object may indicate a building on the map.

The object may indicate a road on the map.

The object may indicate a place name on the map.

Another aspect of the device control method using the voice recognition function of the present disclosure is an input process for receiving an input from a user, and a selection state for performing a state detection of whether or not a part of the screen is designated by the input process. A detection process, a selection information detection process that acquires internal information regarding the position of the selected one item on the screen, an output process that returns a response to the user, a communication process that communicates with an external device, and a voice that inputs voice. Input processing, first speech recognition processing for recognizing speech, second speech recognition processing learned by a method different from the first speech recognition processing, and detection of an instruction character string based on the speech recognition result Including the instruction character string detection processing that performs the instruction character string detection processing and the instruction character string detection processing that detects the instruction character string based on the voice recognition result, in the selection state detection processing, a part of the screen is selected by the input processing. , And when both the instruction character string and the instruction character string are detected, output processing is performed according to the result of the first voice recognition processing, and a part of the screen is not selected, or the instruction character string When any of the command character strings is not detected, output processing is performed according to the result of the second voice recognition processing.

As a result, when the screen is designated by the input process and the instruction character string and the command character string are detected, a response can be returned to the user without waiting for the voice recognition result from the server. Therefore, it becomes possible to reduce the delay in the response in the voice conversation as compared with the conventional case.

The device control method described above may further include a dialog management process and a response sentence generation process, and the device may be controlled by a process interactive with the user.

The above-described device control method further includes a voice synthesizing process for generating a synthetic voice and a control signal generating process for generating a control signal, and when the response is returned to the user in the output process, the response is returned by the synthesized voice. Alternatively, a response may be returned to the user by controlling the device with the generated control signal.

The selection state detection process may manage only the state in which a part of the screen is selected in the input process.

The selection state detection process may manage internal information corresponding to the selected location in addition to the state of whether a part of the screen is selected in the input process.

The input process may specify either metadata about the television program or the content of the television program.

The metadata regarding the TV program may be any of the program name, channel name, content, attention level, and recommendation level.

The content of the television program may include any one of a person, an animal, a car, a map, letters, and numbers.

Furthermore, another aspect of the control method for solving the above-mentioned problem is a control method of a display device having a display, which is connected to a voice input device capable of inputting a user's voice, in a computer of the display device, Display a display screen containing multiple selectable items on the display, and detect that one of the multiple items is selected on the display screen of the display, and detect that one item has been selected. When the voice instruction including the first voice information indicating the instruction content is received from the voice input device, the instruction content is recognized from the first voice information, the instruction content is executed, and one item is selected. When it is not detected that the instruction is performed or when it is determined that the instruction content cannot be executed, the voice instruction is transmitted to another computer.

The computer of the display device is further caused to determine whether or not the voice instruction includes the second voice information indicating the reference word, it is detected that one item is selected, and the instruction content is detected from the first voice information. If it is recognized and it is determined that the voice instruction includes the second voice information, the instruction content is executed, and if the selection of one item is not detected, the first voice information is used to If the instruction content is not recognized, or if it is not determined that the voice instruction includes the second voice information, the voice instruction may be transmitted to the other computer.

The instruction content is an instruction to search for information related to one item, and the control method may notify the user of a search result based on the instruction content.

The display device may be connected to the server via a network, and information related to one item may be searched by referring to a database in the server.

The control method may display the search result on the display.

The voice input device may be included in the display device.

The display device may be further connected to a sound output device capable of outputting sound, and the control method may cause the sound output device to transmit search result information that causes the sound output device to output the search result as sound.

The audio output device may be included in the display device.

The plurality of items may be items indicating metadata about the television program or contents of the television program.

The metadata may indicate at least one of a TV program name, a channel name, an overview of the TV program, a TV program attention level, and a TV program recommendation level.

The content of the television program may include information indicating at least one of a person, an animal, a car, a map, characters, and numbers.

The display screen represents a map in a specific area, and each of the plurality of items may be an arbitrary coordinate on the map or an object on the map.

The object may indicate a building on the map.

The object may indicate a road on the map.

The object may indicate a place name on the map.

Furthermore, one mode of a computer program for solving the above-mentioned problem is a computer program which is connected to a voice input device capable of inputting a user's voice and causes a display device having a display to execute the computer program. A computer of a display device displays a display screen including a plurality of selectable items on the display, and causes the display screen of the display to detect that one of the plurality of items is selected, When it is detected that one item is selected, when the voice instruction including the first voice information indicating the instruction content is received from the voice input device, the instruction content is recognized from the first voice information. Then, the instruction content is executed, and when it is not detected that the one item is selected, or when it is determined that the instruction content cannot be executed, the voice instruction is transmitted to another computer.

One aspect of the display device of the present disclosure is a display device connected to a voice input device capable of inputting a user's voice, the display device including a display, a control circuit, and a communication circuit, wherein the control circuit is selected. A display screen including a plurality of possible items is displayed on the display, and on the display screen of the display, it is detected that one of the plurality of items is selected, and the one item is selected. When the voice instruction including the first voice information indicating the instruction content is received from the voice input device while detecting the fact, the instruction content is recognized from the first voice information and the instruction content is displayed. If the execution is not detected and the selection of the one item is not detected, or if it is determined that the instruction content cannot be executed, the communication circuit is instructed to transmit the voice instruction to another computer. ..

Each of the embodiments described below shows one specific example of the present invention. Numerical values, shapes, constituent elements, steps, order of steps, and the like shown in the following embodiments are examples and are not intended to limit the present invention. Further, among the constituent elements in the following embodiments, the constituent elements that are not described in the independent claim indicating the highest concept are described as arbitrary constituent elements. In addition, the contents of each of the embodiments can be combined.

Hereinafter, exemplary embodiments of the present invention will be described with reference to the accompanying drawings.

(Embodiment 1)
FIG. 1 is a sequence diagram showing an outline of an information providing method executed by the information providing system according to the present embodiment for a display device. The information providing system in the present embodiment is connected to a display device having a display and a voice input device capable of inputting a user's voice. Here, "connected" means electrically connected so that an electric signal can be transmitted and received. The “connection” is not limited to a wired connection and may be a wireless connection. In a state where another device (for example, a switching hub, a router, a personal computer (PC), etc.) is connected between the two devices and electric signals can be transmitted and received through them, the two devices are also connected. It corresponds to the state of being.

The information providing system may typically be a combination of one or more devices including a server computer. The information providing system transmits display screen information that causes a display of a display device to display a display screen including a plurality of selectable items to the display device. In response to this, the display device displays the display screen on the display (step S100). This display screen includes a plurality of selectable items. Each of the plurality of items may be, for example, an item indicating a television program as shown in FIG. 13, but is not limited to this. Each of the plurality of items may be an item indicating metadata about the television program or content of the television program. The metadata may be, for example, data indicating at least one of a television program name, a channel name, an outline of the television program, a degree of attention of the television program, and a degree of recommendation of the television program. The content of the television program may include information indicating at least one of a person, an animal, a car, a map, letters, and numbers, for example. When the display screen includes a map image, each of the plurality of items may be coordinate information that identifies a position on the map.

The user can select one item from the plurality of items displayed on the display of the display device. For example, when a plurality of items indicating a television program are displayed, one item can be selected from them. If the display device comprises a touch screen as the display, the selection of items can be done by direct contact with the touch screen. When the display device displays a display screen on an external display, selection of items can be performed by operating a mouse, for example. In the former case, the touch screen functions as the input device, and in the latter case, the mouse functions as the input device.

When one of a plurality of items is selected on the display screen of the display, the display device sends information indicating that (referred to as "item selection information") to the server included in the information providing system. Send. Upon receiving the item selection information, the server determines which item has been selected and records (or updates) the selected/non-selected state of each item (step S110). This process is called a selection state management process. The transmission of item selection information and the selection state management process are executed each time the user changes the selection of an item. In other words, the selection state management process resulting from the user's selection (or change) of items may be executed any number of times before the voice instruction.

After selecting one item, the user gives a voice instruction to that item. For example, an instruction to reproduce a television program corresponding to the selected item or an instruction to display a summary of the television program can be issued by voice. Such instructions may be given, for example, by saying "play it back", "display its contents", etc. The instruction may include first voice information indicating instruction content such as “play” and “display content” and second voice information indicating instruction words such as “that” and “that”. The first audio information is associated with the control command of the display device. When the display device receives some voice instruction from the user, the display device transmits the voice information to the server.

When the server receives the voice information, whether one item is selected (step S111), whether the voice instruction includes the first voice information (step S112), and the voice instruction includes the second voice information. It is determined whether or not (step S113). If the server determines No in any of these three steps, the server ignores the instruction content and returns to the standby state. Alternatively, information indicating that the instruction content is not executed may be transmitted to the display device.

In step S111, the server refers to the information on the selection state updated in the selection state management process (S110) and determines whether or not one item is selected. If one item is selected, the process proceeds to step S112. In step S112, the server determines whether or not the voice instruction includes the first voice information (that is, the instruction content). When it is determined that the voice instruction includes the first voice information, the server recognizes the instruction content (step S113). In subsequent step S114, the server determines whether or not the voice instruction includes the second voice information (that is, the instruction word). When it is determined that the voice instruction includes the second voice information, the server executes the instruction content (step S115). Execution of the instruction content is performed, for example, by transmitting control information of a device corresponding to the requested instruction to the display device. The order of steps S111, S112, and S114 is not limited to the order shown in FIG. 1 and may be interchanged.

With such a method, the server can grasp the selection state of the item on the display screen of the display device in real time by the selection state management process (S110). After the server accepts the voice instruction, it is not necessary to inquire the display device about the selection state, so that the access between the display device and the server can be reduced.

Next, a more specific example of a system that employs the program information presentation method according to the present embodiment will be described.

FIG. 2 shows the configuration of a system that employs the program information presentation method according to the present embodiment. This program information presentation method presents the user with program information by utilizing a voice recognition function that recognizes the user's voice. The system includes a client 121 and a server 120. The client 121 corresponds to the above-described display device or another device connected to the display device. The client 121 may be a device such as a television, a recorder, a smartphone, or a tablet terminal. In the example of FIG. 2, the client 121 includes the microphone 101 that is a voice input device, an input device 108, an output circuit 112, a communication circuit 113b, and a control circuit 114b that controls these. The control circuit 114b detects a selection state detecting unit 109 that detects selection of an item by the user, and a selection information detecting unit 111 that detects position information on the display screen of the program designated by the input device and information of the designated program. Have and.

The server 120 includes a communication circuit 113a that communicates with the client 121, and a control circuit 114a. The control circuit 114a has seven functional units including a selection state management unit 110, a voice recognition unit 102, an instruction character string detection unit 103, a dialogue management unit 104, a response sentence generation unit 105, a voice synthesis unit 106, and a control signal generation unit 107. Have.

In the present embodiment, the microphone 101, which is a voice input device, senses the voice signal of the user. The voice recognition unit 102 of the server 120 converts the sensed voice signal into a character string. After that, the processing by the server 120 is mainly performed. The designated character string detection unit 103 detects a designated pronoun included in the character string converted by the voice recognition unit 102. The dialogue management unit 104 manages a history of interactive processing between the user and the device, a response strategy indicating what kind of interactive processing is to be performed, and the like. Here, the interactive processing refers to processing relating to message exchange between the user and the device using a physical interface such as a touch panel or voice. The history information and the information used for the response strategy are stored in a recording medium such as a memory (not shown).

The response sentence generation unit 105 generates a character string that responds to the user according to the input character string. The voice synthesis unit 106 converts the character string generated by the response sentence generation unit 105 into voice. The control signal generation unit 107 generates a device control command according to the content of the dialogue.

It has been described that the voice synthesis unit 106 generates a synthetic voice from the sentence generated by the response sentence generation unit 105 and presents the voice to the user, but this is an example. For example, when a display device such as a TV is provided in the client 121, the character string may be displayed on the screen.

The input device 108 can be, for example, a mouse, a touch panel, a keyboard, a remote controller, or the like. The input device 108 enables a user to select one program when information of a plurality of programs is displayed on a display device such as a display device.

When a program is selected by the input device 108, information on the selected position on the screen is acquired. The position information may be two-dimensional coordinate information, for example. In addition to a plurality of selectable items indicating programs, another display area that can be designated may be present on the display screen. For example, there may be other display areas such as buttons for page transitions, buttons to end program selection, or buttons to invoke other functions. The user can also specify such a display area. The selection state detection unit 109 in the client 121 detects whether any program is selected by the input device 108. This detection can be performed by determining whether or not the designated position overlaps with the position of the item indicating any program. The detection result is sent to the selection state management unit 110 of the server 120 via the communication circuits 113b and 113a. The selection state management unit 110 manages information indicating whether or not any program is selected. For example, if any program is selected, 1 is set in the internal memory of the selection state management unit 110, and if no program is selected, 0 is set in the internal memory. The value of this internal memory is updated according to the selected state.

The selection information detection unit 111 detects position information of the program designated by the input device 108, information of the designated program, and the like. The detected information is transmitted to the instruction character string detection unit 103 via the communication circuits 113b and 113a. The output circuit 112 outputs information based on the output results of the response sentence generation unit 105, the voice synthesis unit 106, and the control signal generation unit 107. The output circuit 112 performs output processing such as displaying a response sentence on a display, reproducing synthesized voice on a speaker, controlling a device by a generated control signal, and displaying a control result on a display.

The communication circuits 113a and 113b include a communication module for performing communication between the server 120 and the client 121. Here, the communication module performs communication using an existing communication method such as Wi-Fi (registered trademark) or Bluetooth (registered trademark). The type of communication module is not limited as long as it has such a function. The voice signal synthesized by the voice synthesis unit 113 and the control signal for controlling the device are transmitted to the output circuit 108. The output circuit 108 outputs a voice signal, a signal for controlling the device, and information indicating a control result.

Each component of the control circuit 114a in the server 120 described above may be realized by a computer (for example, a CPU) of the server 120 executing a computer program, or each may be provided as a separate and independent circuit or the like. Good.

Each component of the control circuit 114b in the client 121 (each of the selection state detection unit 109 and the selection information detection unit 111) described above is also realized by the computer (for example, CPU) of the client 121 executing a computer program. Alternatively, each may be provided as an independent circuit or the like.

For example, each processing of the server 120 shown in FIG. 3 described later can be realized as a control method performed by a computer of the server 120 that executes a computer program. Similarly, each process of the client 121 shown in FIG. 3, for example, can be realized as a control method performed by a computer of the client 121 that executes a computer program.

In this embodiment, an example in which the server 120 performs the voice recognition process will be described. Each process of the dialogue management unit 104, the response sentence generation unit 105, the voice synthesis unit 106, and the control signal generation unit 107, which is executed after performing the voice recognition, may be executed by the client 121 instead of the server 120.

FIG. 3 shows a sequence of communication processing between the server 120 and the client 121. This sequence is started by the user designating a part on the display screen with the input device 108 such as a remote controller.

In the input device information acquisition process of step S200, the selection state detection unit 109 acquires information indicating the position on the display screen designated by the input device 108. If the input device 108 is a touch panel, the position can be designated by touching with a finger or the like. If the input device 108 is a remote control, it can be performed by button operation.

In the selection state detection process of step S201, the selection state detection unit 109 detects whether one program is selected. This detection is performed by determining whether or not the position designated by the input device 108 corresponds to the position of the item indicating the program, based on the position information acquired by the input device information acquisition process.

In the selection information storage process of step S202, the client 121 acquires information regarding the item selected by the input device 108 (hereinafter, may be referred to as “selection information”) and stores it in a recording medium such as a memory. I do. For example, in the case of a program, information associated with the selected program (for example, program name, broadcast date/time, summary, performer information, etc.) is acquired. In an example in which a map is displayed on the display, as in the example described later, the information on the selected item may be information on the building at the specified position. An example of the map will be described later.

In the selection state transmission process of step S203, the information indicating the presence/absence of program selection by the input device 108 acquired in the selection state detection process is transmitted from the communication circuit 113b of the client 121 to the communication circuit 113a of the server 120.

In the selection state reception process of step S204, the communication circuit 113a of the server 120 receives the information indicating the selection state transmitted from the client 121.

In the selection state management process of step S205, the selection state management unit 110 manages the program selection state based on the information received in the selection state reception process. Specifically, the selection state management unit 110 sets the state in which a program is selected as 1 and the state in which it is not selected as 0, and stores information of 0 or 1 in a specific memory in the server 120. This makes it possible to manage whether or not a program is selected.

The above steps S200 to S205 are executed every time the user changes the selection of the program. Therefore, steps S200 to S205 shown in FIG. 3 may be executed multiple times.

In the voice request transmission process of step S206, the communication circuit 113a in the server 120 transmits a signal requesting that a voice signal be sent to the communication circuit 113b in the client 121. This process is performed, for example, in response to a request from the user to start a voice instruction. The request to start the voice instruction may be made, for example, by pressing the start button displayed on the screen.

In the voice request receiving process of step S207, the client 121 permits the voice input from the microphone 101 associated with the client 121.

In the A/D conversion process of step S208, the client 121 performs A/D conversion (analog-digital conversion) on the input audio signal. As a result, analog voice is converted into a digital voice signal.

In the audio signal transmission process of step S209, the communication circuit 113b of the client 121 transmits a digital audio signal to the server 120.

In step S210, the communication circuit 113a of the server 120 receives the audio signal transmitted from the client 121.

In step S211, the voice recognition unit 102 performs voice recognition processing. The voice recognition process is a process of analyzing an input voice signal and converting it into text data.

In step S212, the instruction character string detection unit 103 detects the instruction character string. The instruction character string detection process is a process of detecting the instruction character string by analyzing the text data generated by the voice recognition process.

In the selection state determination process of step S213, the selection state management unit 110 refers to the selection state information stored in the memory in the selection state management process of step S205 to determine whether or not one item is selected. Determine whether. That is, the selection state management unit 110 determines whether or not the selection state is set based on only the data on the server 120. When it is determined that the selection state is set, the server 120 requests the selection information from the client 121. Upon receiving this request, the client 121 transmits the selection information stored in the memory in the selection information storage processing of step S202 to the server in the selection information transmission processing of step S214.

In the selection information receiving process of step S215, the communication circuit 113a of the server 120 receives the selection information from the communication circuit 113b of the client 121.

In the dialog management process of step S216, the dialog management unit 104 determines the device control method and the voice response method based on the received selection information and the result of the instruction character string detection process, and returns the result to the client 121. The information of is output. The dialogue management process may be a process in which the dialogue management unit 104 determines the response method by referring to a table in which the input voice information and the output information are associated with each other. For example, when the voice "TV, power ON" is input, the dialogue management unit 104 outputs a device control signal for turning on the power of the TV, or an identifier (ID) corresponding to the device control signal. When the user says "display the content of the program", the character "that" is detected from the detection result of the instruction character string. From this, it can be seen that the word “that” refers to the selection information acquired in the selection information receiving process of step S215. As a result, the dialogue management unit 104 can specify the program content from the program information obtained from the selection information and can generate information for replying to the client 121.

In the response result transmission process of step S217, the communication circuit 113a of the server 120 transmits the information generated in the dialogue management process of step S216 to the client 121. The transmitted information may be, for example, a control signal of the device or an ID corresponding to the control signal, or data of synthetic voice or text data for synthesizing voice.

In the response result reception process of step S218, the communication circuit 113b of the client 121 receives the response result from the server 120.

In the response result output process of step S219, the output circuit 112 outputs the control signal of the device, the synthesized voice, the text, etc. received in the response result reception process of step S218 to the user or the device to be controlled through the output means of the device. For example, regarding the control signal of the device, as a response result output process, it is conceivable to control ON/OFF of the power supply of the TV, control of volume, and up/down of the channel. Regarding synthetic speech, it is possible to output response speech from the TV speaker. Regarding the text, the device of the client 121 may synthesize the voice and output the synthesized voice, or the processed text may be displayed on the screen of the TV.

Hereinafter, the program information presentation method using the voice recognition function will be described in more detail by dividing the processing in the server 120 and the processing in the client 121.

FIG. 4 shows details of the processing flow after the server 120 receives the voice instruction.

First, in the voice input process (S300), a voice signal is input from the microphone 101. In the present embodiment, the microphone is provided in the client 121, and the audio signal A/D converted on the client 121 is transferred to the server 120 side.

In the voice recognition process (S301), the voice recognition unit 102 recognizes the input voice. In the voice recognition process, the input voice signal is converted into character string data. By performing speech recognition on the server 120, an acoustic model and a language model constructed from a large-scale data group can be used. The computing power of the server 120 is higher than that of the client 121. Since acoustic models and language models learned from a large-scale data by a statistical learning method can be used, there is an advantage that the recognition rate of various words is high. Also, with the spread of smartphones and FTTH, the environment in which terminals are constantly connected to networks has been established. Therefore, the method of performing voice recognition on the server 120 is practical.

In the instruction character string detection process (S302), the instruction character string detection unit 103 detects the instruction character string from the character string obtained by voice recognition. Here, the instruction string is an instruction such as "this", "that", "that", "this", "that", "that", "this", "that", "that", etc. It is a word or demonstrative. Here, the detection of the instruction character string is performed as follows. First, the instruction character string detection unit 103 divides the input character string into words and parts of speech by morphological analysis. A morpheme is the smallest unit of meaning in a sentence element. Morphological analysis can divide a sentence into multiple morphemes such as words and parts of speech. It is assumed that the designated character string in the sentence can be detected if the designated character string is prepared in advance as a list and the words included in the list match the divided morphemes. In this way, the matching process between words is performed to detect whether or not the designated character string is detected.

The server 120 switches the subsequent processing depending on whether or not the instruction character string is detected (S303). When the instruction character string is detected, the selection state management unit 110 acquires the state as to whether or not the input device 108 on the client side has selected the information regarding the program on the TV screen (S304). Then, the selection state management unit 110 determines whether or not the program is being selected based on the acquired selection state (S305). Specifically, if the input device 108 selects a program on the screen, 1 is designated as the selection state, and if the program is not selected, a selection state other than 1 is designated. The unit 110 acquires information of 1 or other than 1 in the selection state acquisition process (S304). In the selection state determination process (S305), the selection state management unit 110 determines whether or not the selection state, that is, whether or not the selection state is 1. The value of 1 or other than 1 at this time is stored in the selection state management unit 110. Based on this determination result, the process is switched depending on whether or not the program is selected (S306).

If it is determined that the program is selected, the selection state management unit 110 performs the selection information acquisition process (S307) to obtain information related to the program selected on the screen (for example, program name, broadcast date, recording date and time). , Genre, broadcasting station, program content, EPG information, etc.). Here, the information that the server 120 acquires from the client 121 is for performing detailed operations related to the program. For example, detailed information of a program is transmitted from the client 121 to the server 120 so that the server can process it when an operation command such as displaying the program content or displaying the program genre is input.

If it is determined that the instruction character string is detected in the determination of the instruction character string detection (S303) or that the program is not selected in the program selection determination (S306), the dialogue management unit 104 determines that the dialogue management process is performed. (S308) is performed. In the dialogue management processing according to the present embodiment, the dialogue management unit 104 understands the content of the character string recognized by voice recognition, and determines what kind of response should be made in consideration of the input language information, the past context, and the like. , Outputs information indicating the response result. For example, if a response related to device control such as TV program recording setting or TV screen control is performed, the control signal generation process (S309) generates a device control signal according to an instruction from the dialogue management unit 104. The device control of the client 121 is performed. If the user responds by voice, the voice synthesis unit 106 generates a synthesized voice in the voice synthesis process (S310) according to the instruction of the dialogue management unit 104, and outputs a voice signal.

In the signal transmission process (S311), the communication circuit 113a transmits the device control signal and the voice synthesis signal generated by the device signal generation process and the voice synthesis process to the communication circuit 113b of the client 121.

FIG. 5 shows a processing flow regarding a portion related to detection and output of a selected state among the processing executed by the client 121.

The input device information acquisition process (S400) is a process in which the input device 108 acquires information. The input device 108 acquires the position information of the program selected by the user. In the selection state detection process (S401), the selection state detection unit 109 detects whether the input device 108 is selecting a program. When the input device 108 is selecting a program, for example, when the input device 108 is a remote controller, the user designates the program with the cross key and presses the enter button to transition to the state where the program is selected. It means to do. The client 121 may be configured so as to transition to a state in which a program is selected simply by designating the program with the cross key without providing a decision button. When the input device 108 is a touch screen or a display connected to a PC, the user can tap or click on a place where a specific program is displayed so that the program is transited to the selected state. Good. The user can change to the non-selected state by pressing the enter button again while the program is selected. That is, it is possible to know which position is designated by the input device in the input device information acquisition process, and which information at which position is selected in the selection state detection process.

In the selection state saving process (S402), the client 121 performs a process of saving the position information acquired in the input device information acquisition process and the information obtained in the selection state detection process whether or not the selection is being performed. In the selection information detection process (S403), the selection information detection unit 111 detects information about the program or information about the program corresponding to the position information stored in the selection state storage process. In the present specification, “information about a program” refers to metadata about a television program or contents of a television program, for example. The metadata includes, for example, at least one of a TV program name, broadcast date/time, genre, broadcasting station, channel name, TV program content, TV program popularity, TV program recommendation level, performer, and CM company. The recording date and time may be included as metadata. In addition, the content of the television program includes at least one information of a person, an animal, a car, a map, characters, and numbers. However, these are examples and not limited to these. To detect program information, there are a method of searching for information related to a program name from EPGs inside and outside the system, and a method of performing a web search based on the program name and acquiring related information.

In the signal receiving process (S404), the communication circuit 113b of the client 121 receives the device control signal and the synthesized audio signal transmitted from the server 120 by the signal transmitting process of the server.

In the output processing (S405), the output circuit 112 outputs the processing result to the user based on the result of the control signal generation processing (S306) received in the signal reception processing and the result of the voice synthesis processing (S307).

It should be noted that the target specified by the input device 108 is not limited to the icons and lists representing programs and the like. For example, an arbitrary position such as a map may be designated with a mouse. For designation on the map, the x-coordinate and y-coordinate on the screen may be used as the position information, or the coordinates may be expressed in the latitude and longitude information specific to the map. The latitude and longitude values can be associated with an address. Therefore, the latitude and longitude information may be numerically input from the keyboard to specify the address. Alternatively, the address itself may be entered using the keyboard. Since the address is a relatively long character string, it is considered that voice recognition is likely to fail. In such a case, the target to be pointed may be designated by a method that is easy for the user to input.

A button or icon for canceling the position designation may be provided at a position other than the position designated object. In the case of selecting a program, the selection and deselection of the program can be easily performed by repeating the selection of the icon related to the program. However, when designating a specific position on the map, it is difficult to cancel the selection by selecting one point on the map. Therefore, as shown in FIG. 6, a selection cancel button may be provided at the top of the map screen. The selection can be canceled by pressing the selection cancel button, which facilitates the cancellation of the selection. FIG. 6 shows an example in which “XX super” is designated. An "arrow" indicating the cursor is displayed at the specified position. In FIG. 6, deselection is performed by selecting the deselection frame on the upper right of the map.

The display device for displaying such a map may be used for a car navigation system as well as an information device such as a television, a personal computer, a smartphone, and a tablet terminal. The user can obtain (desir) desired information by designating (that is, selecting) an arbitrary point and then giving a voice instruction including a directive indicating the point. For example, it is possible to construct a system that presents requested information in response to voice instructions such as "What is the route to here?" and "Which gas station is the closest to here?"

In the selection state detection unit 109, information indicating that a program has been selected may be stored together with the time when the program was selected. This makes it possible to change the associated instruction word depending on whether the absolute difference t between the time when the program is selected and the current time is smaller or larger than a predetermined threshold value. For example, when the absolute difference t is smaller than a predetermined threshold value, it is referred to as a near name or a middle name, such as "this", "that", "here", "this", "that", "that". If the absolute difference t is larger than a predetermined threshold value, the program is designated by a so-called directive word, and the program may be designated by a directive word called a far name such as "that" or "that". Good. In this way, the specified word may be changed according to the magnitude of the absolute difference t.

In the present embodiment, a specific program is selected using a demonstrative pronoun, so when two or more programs are designated, it may not be known which program the demonstrative pronoun points to. In this case, the first designated program is selected by the near and medium designations such as "this program" and "that program", and the later designated program is called "that program" by the far name. You may select with the directive. Thereby, it is possible to select one from a plurality of candidates by properly using the pronoun.

When designating a program using the demonstrative pronoun, personal identification information using a personal recognition unit (not shown) may be used. For example, when a program is selected by the input device 108, who selected the program may be identified and the personal identification information may be stored in the selection state detection unit 109. (This information is personal identification information A). At this time, the personal identification information and the information indicating which program the person has selected are stored as a pair. Further, when the instruction character string detection unit 103 detects the instruction character string, the individual who uttered the instruction character string may be identified (this identification information is referred to as individual identification information B). If the personal identification information A that matches the personal identification information B is searched from the information held by the selection state detection unit 109, it is determined whether the individual who selected the program matches the individual who uttered the instruction character string. it can. When both match, the program stored as a pair with the personal identification information A is set as the program designated by the designated pronoun and is set as the operation target.

It should be noted that if a user can specify an arbitrary place on the screen by using a touch pad or a joystick mounted on the remote controller, it is possible to specify a desired place on the screen. With this, for example, if you specify a specific person on the screen with the cursor and specify "program that the person is playing", a list of programs in which the person specified by the cursor appears is displayed on the screen. Will also be possible. If only one person is shown on the screen, it is possible to know who this "person" represents by audio alone. However, when two or more persons are shown in FIG. 7A, it is difficult to specify the person only by voice. By using the cursor, as shown in FIG. 7B, it is possible to select one of a plurality of people who appear on TV, which cannot be designated by voice. As a result, it becomes possible to search for information that is specific to the selected person. In order to recognize who the person pointed by the cursor is, existing face detection technology and face recognition technology can be used. A screen example 601 of FIG. 7A is an example in which a person on the screen is designated by a cursor. In the screen example 601, the cursor is placed on the person on the left side. When a person is designated by the cursor, face detection and face recognition processing in the vicinity thereof are performed. Thereafter, as shown in FIG. 7B, the user can visually confirm who has been designated by displaying on the display where and who was recognized, or by presenting it to the user by voice (screen example 602).

In this example, the example of detecting a person has been described, but by using the general object recognition technique, it is possible to recognize an animal, a car, a character, a number, or the like as described above.

When searching for a place using the map displayed on the screen, the map in the specific area is displayed on the display screen, and the arbitrary coordinates on the map designated by the cursor or the object on the map is used as the reference. Search is possible. As shown in FIGS. 8A and 8B, for example, “drugstore north of this location” can display a drugstore north of the location of the cursor. In the display example 701 of FIG. 8A, xx park is designated, and is indicated by an arrow on the drawing. By performing a search by voice, the location of the drug store is presented as in the display example 702 of FIG. 8B. In the display example 702, the search position is displayed as a dotted circle. As a result, the user can intuitively perform a map search using information and a voice of the currently pointed position without knowing the detailed address. Similarly, by asking, "How do you get there?", you can search for directions (route search or car navigation) from the current position to the position specified by the cursor. As a result, normally, after confirming the position on the map, it is necessary to operate a few steps of buttons to search for the way to that position, but you can quickly complete the process by inputting voice and set the setting. It becomes easy.

In the present embodiment, the transmission/reception processing in the selected state and the transmission/reception processing of the selection information are described separately, but the selection information may be transmitted at the time of transmission in the selected state. In such a case, the sequence of data transmission/reception between the server and the client is as shown in FIG. The system configuration and the processing flows of the server and the client are as shown in FIGS. 2, 4 and 5, respectively. In the following, redundant description may be omitted.

FIG. 9 shows a sequence of communication processing between the server 120 and the client 121 when the selection information is also transmitted when transmitting the selected state. This sequence is started by the user designating a part on the display screen with the input device 108 such as a remote controller.

Step S800 is an input device information acquisition process. The selection state detection unit 109 detects where the input device 108 is pointing on the screen of the client 121.

Step S801 is a selection state detection process. The selection state detection unit 109 acquires whether or not the position specified by the input device information acquisition process is specified by the input device 108.

Step S802 is a selection information transmission process. The communication circuit 113b transmits information regarding the selected item to the server 120.

Step S803 is a selection information reception process. The communication circuit 113a of the server 120 receives the selection information from the client 121.

Step S804 is a selection state management process. This is a process by which the selection state management unit 110 manages the selection state via the input device 108 received in the selection state reception process on the server 120 side. In the selection state management process, the selection state management unit 110 sets 0 as a state in which the input device 108 is selecting a specific item and 0 as a state in which it is not selected, and stores information of 0 or 1 on the server 120. Save to a specific memory. In this example, since the selection information has already been transmitted, what kind of information is transmitted is also stored in the memory. For example, in the case of a list of television programs, program names, broadcast dates, contents, etc. are stored, and in the case of maps, place names, latitude and longitude, housing information of selected locations, etc. are stored.

Step S805 is a voice request transmission process. The server 120 sends a signal requesting the client 121 to send an audio signal.

Step S806 is a voice request reception process. Upon accepting the voice request reception process, the client 121 permits input of voice from the microphone 101 associated with the client 121.

In step S807, the client 121 permits voice input and performs A/D conversion (analog-digital conversion). As a result, analog voice is converted into a digital voice signal. In the audio signal transmission process of step S808, the communication circuit 113b of the client 121 transmits the digital audio signal to the server 120.

In step S809, the communication circuit 113a of the server 120 receives the audio signal transmitted from the client 121.

In step S810, the voice recognition unit 102 performs voice recognition processing. Further, in step S811, the instruction character string detection unit 103 detects the instruction character string.

Step S812 is a dialogue management process. The dialogue management unit 104 outputs device control, a voice response method, and the like from the received selection information and the result of the instruction character string detection processing. The method of interaction management processing is the same as the method described above.

Step S813 is a response result transmission process. The response result transmission process is a process of transmitting to the client 121 the control signal output by the dialogue management process, the ID corresponding to the control signal, the synthesized voice, and the text for synthesizing the voice.

Step S814 is a response result reception process. As a result, the communication circuit 113b of the client 121 receives the response result from the server 120.

Step S815 is a response result output process. As the response result output process, the output circuit 112 outputs the control signal of the device, the synthesized voice, the text, etc. received in the response result receiving process to the user terminal or the device to be controlled through the output means of the device.

With the above configuration and processing, it is possible to reduce the processing delay even when the voice recognition processing is performed on the server.

(Embodiment 2)
FIG. 10 is a sequence diagram showing an outline of a control method executed by the information providing system according to the present embodiment for a display device. The information providing system in the present embodiment differs from that in the first embodiment in that the display device also has a voice recognition function. Hereinafter, the description will be focused on the points that are different from the first embodiment, and the description of overlapping items may be omitted.

The display device control method according to the present embodiment causes a computer of the display device to execute the process illustrated in FIG. 10. In this control method, first, a display screen including a plurality of selectable items is displayed on a display mounted or connected to a display device (step S900). Next, it is detected that one of the plurality of items has been selected on the display screen of the display (step S901). Steps S900 and S901 are repeatedly executed each time the item selection is changed.

When the display device receives the voice instruction, the display device determines whether or not one item is selected (step S902). When no item is selected, the display device transmits the received audio information to another computer (hereinafter, referred to as “server”) in the information providing system. When the item is selected, the display device determines whether or not the voice instruction can be executed (step S903). If the voice instruction can be executed, the instruction content is executed (step S904). If the voice instruction is not executable, the display device sends voice information to the server. The server recognizes and executes the voice instruction that the display device cannot execute (steps S911 and S912).

Here, the executable voice instruction means a voice instruction that can be processed within a range of functions preprogrammed in the display device. For example, if the display device can correctly recognize a voice instruction consisting of a combination of a specific instruction word and specific instruction content, but cannot recognize a voice instruction other than that (for example, a web search instruction), the former can be executed. Yes, but the latter is not feasible. In such a case, the latter voice instruction is executed by the server instead, and the response result is returned to the display device.

As described above, the control method according to the present embodiment is performed when the voice instruction including the first voice information indicating the instruction content is received from the voice input device when the selection of one item is detected. , Causing the computer of the display device to recognize the instruction content from the first voice information and execute the instruction content. On the other hand, when it is not detected that one item is selected, or when it is determined that the instruction content cannot be executed, the voice instruction is transmitted to the server. As a result, the display device and the server are accessed only when necessary, so that the processing delay can be reduced.

FIG. 11 is a sequence diagram showing an example of a control method of a display device capable of recognizing a voice instruction including a combination of an instruction word and instruction content. In this control method, steps S905 to S907 are executed instead of step S903 in FIG. Except for this point, the method is the same as that of FIG. In step S905, the display device determines whether or not the voice instruction includes the first voice information indicating the instruction content. If the determination result is No, the display device transmits voice information to the server. If the determination result is Yes, the display device recognizes the instruction content (step S906). In a succeeding step S907, it is determined whether or not the voice instruction includes the second voice information indicating the instruction word. If the determination result is No, the display device transmits voice information to the server. If the determination result is Yes, the display device executes the instruction content (step S904).

As described above, in the control method shown in FIG. 11, it is detected that one item is selected, the instruction content is recognized from the first voice information, and the voice instruction includes the second voice information. If it is determined that the instruction is given, the computer of the display device executes the instruction content. On the other hand, when it is not detected that one item is selected, when the instruction content is not recognized from the first voice information, or when it is not determined that the voice instruction includes the second voice information. , Send voice instructions to the server. As a result, the display device and the server are accessed only when necessary, so that the processing delay can be reduced.

Next, a more specific example of a system that employs the program information presentation method according to the present embodiment will be described.

FIG. 12 shows the configuration of a system that employs the program information presentation method according to the present embodiment. This program information presentation method presents the user with program information by utilizing a voice recognition function that recognizes the user's voice. The system includes a client 121 and a server 120. The client 121 corresponds to the above-described display device or another device connected to the display device. The client 121 may be a device such as a television, a recorder, a smartphone, or a tablet terminal. In the example of FIG. 12, the client 121 includes the microphone 101 that is a voice input device, an input device 108, an output circuit 112, a communication circuit 113b, and a control circuit 114d that controls these. The control circuit 114 d in the present embodiment is different from the selection state detection unit 109 and the selection information detection unit 111 in that it has a voice recognition unit 102 b, an instruction character string detection unit 103, and a command character string detection unit 115. 2 is different from the control circuit 114b shown in FIG.

The server 120 includes a communication circuit 113a that communicates with the client 121, and a control circuit 114c. The control circuit 114c has five functional units: a voice recognition unit 102, a dialogue management unit 104, a response sentence generation unit 105, a voice synthesis unit 106, and a control signal generation unit 107.

In the present embodiment, the microphone 101, which is a voice input device, senses the user's voice signal, and the voice recognition unit 102b converts the sensed voice signal into a character string. The designated character string detection unit 103 determines whether or not the converted character string includes a designated pronoun. The command character string detection unit 115 detects whether the converted character string includes a command character string for controlling a device or the like. The input device 108 enables the user to select one program when a plurality of program information is displayed on the display.

When a program is selected by the input device 108, information on the selected position on the screen is input to the system. The selection state detection unit 109 determines whether a program is selected by the input device 108. The selection information detection unit 111 detects the position information of the program selected by the input device 108, the information regarding the selected program, and the like. The output circuit 112 receives the output results of the response sentence generation unit 105, the voice synthesis unit 106, and the control signal generation unit 107, displays the response sentence on the display, reproduces the synthesized voice on the speaker, and generates the control signal. Performs output processing such as device control and display of control results on the display.

The communication circuits 113a and 113b include a communication module for performing communication between the server 120 and the client 121. As described above, the communication module performs communication using an existing communication method such as Wi-Fi (registered trademark) or Bluetooth (registered trademark). The type of communication module is not limited as long as it has such a function. The voice signal synthesized by the voice synthesis unit 106 and the control signal for controlling the device are transmitted to the output circuit 112. The output circuit 112 outputs an audio signal, a signal for controlling the device, and information indicating the control result.

The voice recognition unit 102a performs voice recognition on the server 120. The dialogue management unit 104 manages a history of interactive processing between the user and the device, a response strategy indicating what kind of interactive processing is to be performed, and the like. The response sentence generation unit 105 generates a character string that responds to the user according to the input character string. The voice synthesis unit 106 converts the character string generated by the response sentence generation unit into voice. The control signal generation unit 107 generates a device control command according to the content of the dialogue.

Each component of the control circuit 114c in the server 120 and the control circuit 114d in the client 121 described above may be realized by a computer (for example, a CPU) of the server 120 executing a computer program, or each component is independent. It may be provided as a circuit or the like.

For example, each process of the server 120 shown in FIG. 13 described later can be realized as a control method of the server 120 performed by a computer of the server 120 that executes a computer program. Similarly, for example, each process of the client 121 illustrated in FIG. 13 can be realized as a control method of the client 121 performed by a computer of the client 121 that executes a computer program.

The present embodiment differs from the related art and the first embodiment in that both the client 121 and the server 120 perform the voice recognition process. The dialogue management unit 104, the response sentence generation unit 105 that performs processing after performing voice recognition, or the speech synthesis unit 106 and the control signal generation unit 107 that generate processing results are provided in the client 121 instead of the server 120. May be.

FIG. 13 shows a sequence of communication processing between the server 120 and the client 121. This sequence is started by the user designating a part on the display screen with the input device 108 such as a remote controller.

Step S500 is an input device information acquisition process. The selection state detection unit 109 acquires information indicating the position on the display screen designated by the input device 108.

Step S501 is a selection state detection process. The selection state detection unit 109 detects whether or not one program has been selected. This detection is performed by determining whether or not the position designated by the input device 108 corresponds to the position of the item indicating the program, based on the position information acquired by the input device information acquisition process.

In step S502, the client 121 inputs a voice and performs A/D conversion (analog-digital conversion). As a result, analog voice is converted into a digital voice signal.

Step S503 is a voice recognition process in which the client 121 recognizes the input voice.

In step S504, the instruction character string detection unit 103 detects the instruction character string. In the instruction character string detection process, the instruction character string is detected by analyzing the text subjected to the voice recognition process.

In step S505, the command character string detection unit 115 detects a command character string. The command character string detection process is a process of detecting the command character string by analyzing the text subjected to the voice recognition process.

In step S506, the selection information detection unit 111 performs selection information detection processing. The input device 108 detects the position information acquired in the information acquisition process.

Step S507 is a voice signal transmission process. The communication circuit 113b of the client 121 sends an audio signal to the server 120.

Step S508 is a voice signal reception process. The communication circuit 113a of the server 120 receives the audio signal.

Step S509 is a voice input process. The voice signal received by the communication circuit 113a is input into the server 120.

Step S510 is a voice recognition process on the server side. The voice recognition unit 102a performs voice recognition processing on the server 120.

Step S511 is a dialogue management process. The dialogue management unit 104 determines the device control method and the voice response method based on the received selection information and the result of the instruction character string detection processing, and outputs information to be returned to the client. The method of the dialogue management process is as described in the first embodiment.

Step S512 is a response result transmission process. The response result transmission process transmits to the client 121 the control signal output by the dialogue management process, the ID corresponding to the control signal, the synthesized voice, and the text for synthesizing the voice.

Step S513 is a response result reception process. As a result, the communication circuit 113b of the client 121 receives the response result from the server 120.

Step S514 is a response result output process. As the response result output process, the output circuit 112 outputs the control signal of the device, the synthesized voice, the text, etc. received in the response result receiving process to the user terminal or the device to be controlled through the output means of the device.

Hereinafter, the program information presentation method using the voice recognition function will be described in more detail by dividing the processing in the server 120 and the processing in the client 121.

FIG. 14 shows a processing flow regarding the server 120 in the configuration shown in FIG.

First, in the voice input process (S600), a voice signal is input from the microphone 101. In the present embodiment, it is assumed that the microphone is included in the client 121. The audio signal A/D converted on the client 121 is transferred to the server 120 side.

In the server side voice recognition process (S601), the voice recognition unit 102a performs a recognition process of the input voice. In the voice recognition processing, the input voice signal is converted into character string data. By performing voice recognition on the server 120, an acoustic model and a language model constructed from a large-scale data group can be used. The computing power of the server 120 is higher than that of the client 121. Since acoustic models and language models learned from statistical data by large-scale data can be used, there is an advantage that the recognition rate of various words is high. Also, with the spread of smartphones and FTTH, the environment in which terminals are always connected to the network has been prepared, so the method of performing voice recognition on the server 120 is practical.

In the dialogue management process (S602), the dialogue management unit 104 understands the content of the voice-recognized character string, and outputs what response is to be made in consideration of the input language information, the past context, and the like. Depending on the output result of the dialogue management process, a determination process (S603) as to whether or not it is a control signal is performed. For example, if a response relating to device control such as TV program recording setting or TV screen control is performed, the control signal generation unit 107 generates a device control signal in the control signal generation process (S604). In the control signal transmission process (S605), the communication circuit 113a of the server 120 transmits the control signal generated by the control signal generation process to the client 121. As a result, device control is performed on the client 121 side.

When it is determined No in step S603 or when step S605 ends, it is determined whether or not to respond to the user by voice (S606). When responding to the user by voice, a response sentence is generated in the response sentence generation process (S607). Subsequently, it is determined whether the output is voice or text (S608). When the output is a voice, the voice synthesizing unit 106 generates a synthetic voice in the voice synthesizing process (S609) and outputs a voice signal. In the voice transmission process (S608), the communication circuit 113a of the server 120 transmits the data converted from the text to the synthetic voice to the client 121.

If the output is text, response sentence transmission processing (S607) is performed. The response generation unit 105 generates text by the response sentence generation processing, and the generated response sentence is transmitted from the server 120 to the client 121.

FIG. 15 shows a processing flow regarding a portion related to detection and output of the selected state among the processing executed by the client 121.

The input device information acquisition process (S700) is a process in which the input device 108 acquires information. The input device 108 acquires the position information of the program selected by the user. In the selection state detection process (S701), the selection state detection unit 109 detects whether the input device 108 is selecting a program on the TV screen. When the input device 108 is selecting a program, for example, when the input device 108 is a remote controller, the user designates the program with the cross key and presses the enter button to transition to the state where the program is selected. It means to do. When the program is selected, the user can press the enter button again to change to the non-selected state. That is, it is possible to know which position is designated by the input device in the input device information acquisition process, and to know which position and which information is selected in the selection state detection process.

In the voice input process (S702), the communication circuit 113a receives the voice transmitted from the client 121. In the voice recognition process (S703), the voice recognition unit 102 recognizes the input voice. The voice recognition in the client 121 has a limit in the words that can be registered, as compared with the server-type voice recognition. In order to reduce erroneous recognition with a limited amount of calculation and memory, it is desirable to register the minimum necessary words in the dictionary. The dictionary may be stored in a memory (not shown) in the circuit that functions as the voice recognition unit 102, or may be stored in a storage device (not shown) provided in the client 121.

The minimum necessary words are, for example, a set of words corresponding to the buttons of the remote controller, and are “power ON”, “power OFF”, “volume up”, “volume down”, and the like. Further, in the present embodiment, since the instruction character string detection process and the command character string detection process described later are performed, the vocabulary used for the detection is registered in the dictionary in advance. For example, in order to recognize the instruction character string, "this", "that", "that", "this", "that", "that", "this", "that", "that" Register the vocabulary and the verb such as. In addition, command vocabularies such as "display contents" and "search" are registered. As a result, the voice recognition unit 102 can recognize words such as “display the contents of the program”. As a result, the instruction character string and the instruction character string can be detected by the subsequent processing.

In the instruction character string detection process (S704), the instruction character string detection unit 103 detects the instruction character string from the character string obtained by voice recognition. The instruction character string is the above-described instruction word or verb. Here, the detection of the instruction character string is performed as follows. First, the designated character string detection unit 103 divides the input character string into words and parts of speech by morphological analysis. A morpheme is the smallest unit of meaning in a sentence element. Morphological analysis can divide a sentence into multiple morphemes such as words and parts of speech. It is assumed that the designated character string in the sentence can be detected if the designated character string is prepared in advance as a list and the words included in the list match the divided morphemes.

In the command character string detection process (S705), the command character string detection unit 115 detects the command character string from the voice recognition result. The command character string detection unit 115 performs morphological analysis as in the instruction character string detection process, and divides the sentence. The command character string is detected by comparing the divided sentences with the word list registered in advance. Here, the command character string registered in the word list is a word corresponding to an operation command such as “contents, display”, “search”, and “record”.

Next, the selection state detection unit 109 uses information obtained by performing the selection state detection process to determine whether or not an area on the screen is selected (S706). The selection state detection unit 109 outputs a flag indicating a program selection state when, for example, a program on the TV screen is selected. In that case, 1 is returned when a program is selected, and a value other than 1 is output when a program is not selected. By using this value, the selection state of the program can be known and the state can be determined. Next, the instruction character string detection unit 103 and the instruction character string detection unit 115 respectively determine whether an instruction character string is detected (S707) and whether an instruction character string is detected (S708). .. To detect the detection of these character strings, the instruction character string is detected by matching with the vocabulary of the list registered in advance as described above.

When the selection state detection unit 109 determines that no item is selected, when the instruction character string detection unit 103 does not detect the instruction character string, or when the instruction character string detection unit 115 determines that the instruction character string is If not detected, signal transmission/reception processing (S709) is performed. In this process, the communication circuit 113a transmits a voice signal to the server 120, and then receives a signal indicating the response result returned from the server 120. The signal indicating the response result includes a voice signal generated by performing voice recognition and dialogue processing in the server 120, or a device control signal. The output circuit 112 performs output processing (S711) and notifies the user of the processing result.

In steps S706 to S708, when it is determined that the selection state detection unit 109 is in the selection state and the instruction character string detection unit 103 detects the instruction character string and the instruction character string, the selection information detection process (S710) is performed. Done. In the selection information detection process (S710), the selection information detection unit 107 acquires the position information and TV program information acquired in the input device information acquisition process. For example, the position on the screen of the program designated on the TV screen by the input device 108 or information related to the program, for example, the above-mentioned metadata about the television program or the content of the television program is acquired. The output circuit 112 performs an output process (S711) based on the information and the command character string acquired here to control the device.

As described above, according to the present embodiment, not only the server 120 but also the client 121 recognizes the voice instruction. The client 121 transmits a voice signal to the server 120 only when the voice instruction cannot be executed, passes the processing to the server 120, and waits for a response result. As a result, the client 121 side can execute a process having a small variation of voice instructions, such as an operation related to a television program, and the server side 120 can execute a process other than that. According to the present embodiment, the access between the client 121 and the server 120 can be minimized, so that the processing delay can be reduced.

Note that the various modifications described in the first embodiment can also be applied to the present embodiment. A new embodiment may be configured by combining the first embodiment and the second embodiment.

In addition, in the above-described embodiment, the microphone 101, which is the voice input device, is described as being provided in the client. However, this configuration is an example. For example, the microphone 101 may exist as a device separate from the client. It is sufficient that the client is connected to such a microphone 101 and can receive a voice input via the microphone 101.

Even if the microphone 101 is provided in the client, the microphone 101 exists as an independent device inside the client 121 and is only internally wired. The microphone 101 can be provided so as to be easily attached and detached. The microphone 101 is not an essential component of the client 121. The client 121 may be connected to the microphone 101 inside or outside thereof.

Further, in the above-described embodiment, it is described that the output circuit 108 outputs the device control signal, the synthesized voice, the text, and the like. It means that the output circuit 108 can be a part of a control signal transmitter (for example, an infrared transmitter of an output terminal or a remote controller), an audio output device (for example, a speaker), or a display. These may be provided integrally or may exist as separate and independent devices.

The present disclosure is useful in the case of performing voice recognition processing on a server regarding an information presentation method using a voice recognition function.

Reference Signs List 101 microphone 102 voice recognition unit 103 instruction character string detection unit 104 dialogue management unit 105 response sentence generation unit 106 voice synthesis unit 107 control signal generation unit 108 input device 109 selection state detection unit 110 selection state management unit 111 selection information detection unit 112 output Circuit 113a, 113b Communication circuit 114a, 114b Control circuit 115 Command character string detection unit 120 Server 121 Client 601 Example of person designation by input device 602 Example of personal authentication result by person designation 701 Example of designating location on map 702 Location on map 901 Display example of program list 902 Remote controller 903 Example of selecting program from program list 904 Example of displaying program content by remote control and voice recognition 1000 Conventional program information presentation device 1001 Microphone 1002 Voice recognition Part 1003 instruction character string detection part 1004 voice synthesis part 1005 control signal generation part 1006 input device 1007 output part

Claims (7)

A method for controlling a device connected to a voice input device capable of inputting a user's voice, comprising:
On the computer of the device,
By the input to the input device by the user, it is detected that one of the items presented by the device is selected,
When it is detected that the one item is selected, when the voice instruction including the first voice information indicating the instruction content is received from the voice input device, the voice instruction is transmitted to another computer. ,
When the voice instruction including the second voice information indicating the instruction content is received from the voice input device when the selection of the one item is not detected, the instruction content is changed from the second voice information. Let me recognize and judge whether or not the voice instruction can be executed,
A control method of causing the instruction content to be executed when it is determined that the instruction content can be executed, and transmitting the voice instruction to another computer when it is determined that the instruction content cannot be executed. In the computer of the device,
It is determined whether or not the first voice information includes third voice information indicating a directive,
When it is detected that the one item is selected, the instruction content is recognized from the first voice information, and it is determined that the first voice information includes the third voice information, Execute the instructions given above,
If it is not determined that the voice instruction includes the third voice information, the voice instruction is transmitted to the other computer,
The control method according to claim 1. The control method according to claim 1, wherein the instruction content is an instruction to search for information related to the one item, and a user is notified of a search result based on the instruction content. The control method according to claim 3, wherein the device is connected to a server via a network, and information related to the one item is searched by referring to a database in the server. 4. The device according to claim 3, wherein the device is further connected to an audio output device capable of outputting audio, and causes the audio output device to transmit search result information that causes the search result to be output as audio from the audio output device. Control method. A computer program for causing a device connected to a voice input device capable of inputting a user's voice to be executed,
The computer program is stored in the computer of the device,
By the input to the input device by the user, it is detected that one of the items presented by the device is selected,
When it is detected that the one item is selected, when the voice instruction including the first voice information indicating the instruction content is received from the voice input device, the voice instruction is transmitted to another computer. ,
When the voice instruction including the second voice information indicating the instruction content is received from the voice input device when the selection of the one item is not detected, the instruction content is changed from the second voice information. Let me recognize and judge whether or not the voice instruction can be executed,
When it is determined that the instruction content is executable, the instruction content is executed, and when it is determined that the instruction content cannot be executed, the voice instruction is transmitted to another computer,
Computer program. A device connected to a voice input device capable of inputting a user's voice,
A control circuit,
Communication circuit,
Equipped with
The control circuit is
Detecting that one of the plurality of items presented by the device has been selected by the input to the input device by the user,
When it is detected that the one item is selected, when the voice instruction including the first voice information indicating the instruction content is received from the voice input device, the voice instruction is transmitted to another computer. To the communication circuit,
When the voice instruction including the second voice information indicating the instruction content is received from the voice input device when the selection of the one item is not detected, the instruction content is changed from the second voice information. Recognize and determine whether the voice instruction can be executed,
When it is determined that the instruction content is executable, the instruction content is executed, and when it is determined that the instruction content cannot be executed, the communication circuit is instructed to transmit the voice instruction to another computer,
machine.

Images