JP7278830B2 - TERMINAL DEVICE, TERMINAL DEVICE CONTROL METHOD, AND PROGRAM - Google Patents

Description

The present invention relates to a terminal device, a terminal device control method, and a program.

Conventionally, research has been conducted on technology for recognizing voices uttered by passengers in vehicles. Patent Document 1 describes an audio signal input unit for inputting an audio signal based on a user's utterance, an operation signal input unit for inputting an operation signal based on a user's manual operation, and an index for noise included in the audio signal. An SN ratio calculation unit that calculates a certain noise index, and a control unit that accepts either an audio signal or an operation signal as an input signal and executes control based on the input signal, wherein the control unit is configured such that the SN ratio calculation unit An invention of an in-vehicle device is disclosed that determines which of an audio signal or an operation signal is to be accepted as an input signal in the next operation input based on a calculated noise index.

JP 2017-102822 A

In the field of speech recognition, the main mechanism is that speech is transmitted from a terminal device to a server device via a network, speech recognition is performed in the server device, and the result is sent back to the terminal device. However, in the conventional technology, sufficient consideration has not been given to a mechanism for requesting speech recognition selectively or in parallel to a plurality of server devices. For this reason, with the conventional technology, there were cases where it was not possible to effectively utilize a plurality of server devices having a speech recognition function.

SUMMARY OF THE INVENTION The present invention has been made in consideration of such circumstances, and includes a terminal device, a terminal device control method, and a program that have a speech recognition function and can effectively utilize a plurality of mutually different server devices. One of the purposes is to provide

A terminal device, a terminal device control method, and a program according to the present invention employ the following configuration.

(1): A terminal device according to an aspect of the present invention is a terminal device mounted in a vehicle, and a plurality of server devices having a voice recognition function for voices in the vehicle interior picked up by microphones. Two or more preprocessing units that perform preprocessing corresponding to each of two or more server devices, and a communication unit that processes audio preprocessed by each of the two or more preprocessing units and a communication control unit for transmitting to the server device.

(2): In the aspect of (1) above, the communication control unit controls at least the first server device among the plurality of server devices for the voice that has not been preprocessed by the preprocessing unit. is to be sent.

(3): In the aspect of (2) above, a first OS that realizes a preprocessing unit that performs preprocessing according to a server device other than the first server device among the two or more preprocessing units; , and a second OS for retrieving audio that has not been preprocessed by the preprocessing unit.

(4): In the above aspects (1) to (3), some or all of the two or more pretreatment units perform treatment sequentially.

(5): In the aspects (1) to (4) above, a plurality of communication control units are provided corresponding to each of the plurality of server devices.

(6): A method of controlling a terminal device according to another aspect of the present invention is a method of controlling a terminal device mounted on a vehicle, wherein each of two or more preprocessing units provided in the terminal device is controlled by a microphone. Preprocessing is performed on the sound collected in the vehicle interior according to each of two or more server devices out of a plurality of server devices having a speech recognition function, and preprocessing is performed by each of the two or more preprocessing units. is transmitted to the corresponding server device using the communication unit.

(7): A program according to another aspect of the present invention is a program executed by a terminal device mounted on a vehicle, wherein the terminal device receives voice in the vehicle cabin picked up by a microphone, Preprocessing corresponding to each of two or more server devices out of a plurality of server devices having a voice recognition function is performed, and the preprocessed voice corresponding to each of the plurality of server devices is transmitted using a communication unit. and is transmitted to the corresponding server device.

According to the aspects (1) to (7) above, it is possible to effectively utilize a plurality of different server devices each having a speech recognition function.

1 is a configuration diagram of a service system 1 including a terminal device 100; FIG. It is a figure which shows the structure of the terminal device 100 which concerns on 1st Embodiment, and the apparatus mounted in the vehicle M. FIG. FIG. 2 is a diagram showing an arrangement example of a display/operation device 20; 3 is a diagram showing an example of arrangement of speaker units 30. FIG. 2 is a diagram illustrating an example of a configuration of a server device 200; FIG. FIG. 4 is a diagram for explaining processing executed by a preprocessing unit; FIG.

Embodiments of a terminal device, a terminal device control method, and a program according to the present invention will be described below with reference to the drawings. A terminal device is a device that implements part or all of a service system. The terminal device is mounted, for example, on a vehicle (hereinafter referred to as vehicle M). The service system transmits voices collected in the vehicle interior of the vehicle M to the server device, returns the result of information processing including voice recognition performed by the server device to the vehicle M, and provides some service ( It is a system that provides information, equipment control, etc.).

In the service system, for example, a speech recognition function that recognizes the voice of the passenger (a function that converts voice into text), a natural language processing function (a function that understands the structure and meaning of text), and other various service functions are used in an integrated manner. is realized by Some or all of these functions may be realized by AI (Artificial Intelligence) technology.

[overall structure]
FIG. 1 is a configuration diagram of a service system 1 including a terminal device 100. As shown in FIG. The service system 1 includes, for example, a terminal device 100 and a plurality of server devices 200-1, 200-2, 200-3, 200-4, . The numbers following the hyphen at the end of the code are assumed to be identifiers for distinguishing services. When not distinguishing which server device it is, it may simply be referred to as the server device 200 . Although three server devices 200 are shown in FIG. 1, the number of server devices 200 may be two or less, or may be three or more. Each server device 200 is operated by a different service provider. Accordingly, services in the present invention are realized by different providers. Any entity (corporation, organization, individual, etc.) can be a service provider.

The terminal device 100 communicates with the server device 200 via the network NW. The network NW includes, for example, some or all of the Internet, cellular network, Wi-Fi network, WAN (Wide Area Network), LAN (Local Area Network), public line, telephone line, wireless base station, and the like.

The terminal device 100 transmits the voice uttered by the passenger of the vehicle M to the server device 200 and provides the passenger with any service based on the information returned from the server device 200 . The service may simply display the contents of speech recognition, may translate into any language, or may be any other service.

[vehicle]
FIG. 2 is a diagram showing the configuration of the terminal device 100 and devices mounted on the vehicle M according to the first embodiment. On the vehicle M, for example, one or more microphones 10, a display/operation device 20, a speaker unit 30, an in-vehicle communication device 60, and a terminal device 100 are mounted. Also, a general-purpose communication device 70 such as a smart phone may be brought into the vehicle and used as a communication device. These devices are connected to each other by multiplex communication lines such as CAN (Controller Area Network) communication lines, serial communication lines, wireless communication networks, and the like. Note that the configuration shown in FIG. 2 is merely an example, and a part of the configuration may be omitted, or another configuration may be added.

The microphone 10 is a sound pickup unit that collects sounds emitted inside the vehicle. The display/operation device 20 is a device (or device group) that displays images and can accept input operations. The display/operation device 20 includes, for example, a display device configured as a touch panel. The display/operation device 20 may further include a HUD (Head Up Display) or a mechanical input device. The speaker unit 30 includes, for example, a plurality of speakers (sound output units) arranged at different positions in the vehicle interior. The display/operation device 20 may be shared by the terminal device 100 and the navigation device 40 .

The vehicle-mounted communication device 60 is, for example, a wireless communication device that can access the network NW using a cellular network or a Wi-Fi network.

FIG. 3 is a diagram showing an arrangement example of the display/operation device 20. As shown in FIG. The display/operation device 20 includes, for example, a first display 22, a second display 24, and an operation switch ASSY26. The display/operation device 20 may further include a HUD 28 .

The vehicle M has, for example, a driver's seat DS provided with a steering wheel SW and a passenger's seat AS provided in the vehicle width direction (Y direction in the figure) with respect to the driver's seat DS. The first display 22 is a horizontally long display device that extends from the middle of the instrument panel between the driver's seat DS and the passenger's seat AS to a position facing the left end of the passenger's seat AS. The second display 24 is installed in the middle of the vehicle width direction between the driver's seat DS and the front passenger's seat AS and below the first display. For example, both the first display 22 and the second display 24 are configured as touch panels, and have LCDs (Liquid Crystal Displays), organic ELs (Electroluminescence), plasma displays, etc. as display units. The operation switch ASSY 26 is a combination of dial switches, button switches, and the like. The display/operation device 20 outputs to the terminal device 100 the details of operations performed by the passenger. The content displayed by the first display 22 or the second display 24 may be determined by the terminal device 100 .

FIG. 4 is a diagram showing an arrangement example of the speaker units 30. As shown in FIG. The speaker unit 30 includes, for example, speakers 30A-30H. The speaker 30A is installed on a window pillar (so-called A pillar) on the driver's seat DS side. The speaker 30B is installed under the door near the driver's seat DS. The speaker 30C is installed on the window pillar on the side of the passenger seat AS. The speaker 30D is installed under the door near the passenger seat AS. The speaker 30E is installed under the door near the right rear seat BS1 side. The speaker 30F is installed under the door near the left rear seat BS2 side. The speaker 30G is installed near the second display 24 . The speaker 30H is installed on the ceiling (roof) of the passenger compartment.

In such an arrangement, for example, if the speakers 30A and 30B exclusively output sound, the sound image is localized near the driver's seat DS. Further, when the sound is output exclusively from the speakers 30C and 30D, the sound image is localized near the front passenger seat AS. Further, when the sound is exclusively output from the speaker 30E, the sound image is localized near the right rear seat BS1. Further, when the sound is exclusively output from the speaker 30F, the sound image is localized near the left rear seat BS2. Further, when the sound is exclusively output from the speaker 30G, the sound image is localized near the front of the vehicle compartment, and when the sound is exclusively output from the speaker 30H, the sound image is localized near the upper part of the vehicle compartment. Become. Not limited to this, the speaker unit 30 can localize a sound image at an arbitrary position in the vehicle compartment by adjusting distribution of sound output from each speaker using a mixer or an amplifier.

[Terminal device]
Returning to FIG. 2, the terminal device 100 includes, for example, a first management unit 110, a second management unit 120, communication control units 150-1 to 150-3, and a pairing application execution unit 152. First management unit 110 includes preprocessing units 112 - 1 and 112 - 2 , display control unit 116 , and audio control unit 118 . The second management unit 120 includes a preprocessing unit 122-3, a display control unit 126, and an audio control unit 128. When not distinguishing which communication control unit it is, it is simply referred to as the communication control unit 150 . The illustration of three communication control units 150 is merely an example corresponding to the number of server devices 200 in FIG. 1, and the number of communication control units 150 may be two or four or more. may be The software arrangement shown in FIG. 2 is simply shown for explanation. can do.

Each component of the terminal device 100 is implemented, for example, by a hardware processor such as a CPU (Central Processing Unit) executing a program (software). Some or all of these components are hardware (circuit part; circuitry) or by cooperation of software and hardware. The program may be stored in advance in a storage device (a storage device with a non-transitory storage medium) such as a HDD (Hard Disk Drive) or flash memory, or may be stored in a removable storage such as a DVD or CD-ROM. It may be stored in a medium (non-transitory storage medium) and installed by loading the storage medium into a drive device.

The management unit 110 functions by executing programs such as an OS (Operating System) and middleware. The same applies to the management unit 120, and the terminal device 100 may have an OS for realizing the management unit 110 and an OS for realizing the management unit 120 separately.

Preprocessing unit 112-1 and communication control unit 150-1 perform processing corresponding to server device 200-1. Preprocessing unit 112-2 and communication control unit 150-2 perform processing corresponding to server device 200-2. Preprocessing unit 122-3 and communication control unit 150-3 perform processing corresponding to server device 200-3. Each preprocessing unit performs or does not perform acoustic processing and the like on the voice so that the corresponding server device 200 is in a state suitable for voice recognition. The communication control unit 150 transmits the voice output from the corresponding preprocessing unit or the result of voice processing to the corresponding server device 200 . Details of these will be described later.

The communication control unit 150 may cooperate with the general-purpose communication device 70 via the pairing application execution unit 152 and communicate with the server device 200 . Communication control unit 150-1 uses in-vehicle communication device 60 to communicate with server device 200-1. Communication control unit 150-2 uses in-vehicle communication device 60 to communicate with server device 200-2. Communication control unit 150-3 cooperates with general-purpose communication device 70 via pairing application execution unit 152 and communicates with server device 200-3. Each of the in-vehicle communication device 60 and the general-purpose communication device 70 is an example of a “communication unit”. The pairing application execution unit 152 performs pairing with the general-purpose communication device 70 by, for example, Bluetooth (registered trademark), and connects the communication control unit 150-3 and the general-purpose communication device 70 together. Note that the communication control unit 150-3 may be connected to the general-purpose communication device 70 by wired communication using USB (Universal Serial Bus) or the like.

Display control units 116 and 126 cause first display 22 or second display 24 to display an image based on the information returned from server device 200 .

Audio control units 118 and 128 cause some or all of the speakers included in speaker unit 30 to output audio based on the information returned from server device 200 .

[Server device]
FIG. 5 is a diagram showing an example of the configuration of the server device 200. As shown in FIG. Here, description of physical communication from the terminal device 100 to the network NW is omitted.

The server device 200 has a communication unit 210 . The communication unit 210 is a network interface such as a NIC (Network Interface Card). Further, the server device 200 includes, for example, a speech recognition unit 220, a natural language processing unit 222, and a reply information generation unit 224. These components are implemented by, for example, a hardware processor such as a CPU executing a program (software). Some or all of these components may be realized by hardware (including circuitry) such as LSI, ASIC, FPGA, GPU, etc., or by cooperation of software and hardware. good too. The program may be stored in advance in a storage device such as an HDD or flash memory (a storage device with a non-transitory storage medium), or may be stored in a removable storage medium such as a DVD or CD-ROM (non-transitory storage medium). physical storage medium), and may be installed by mounting the storage medium in a drive device.

The server device 200 has a storage unit 250 . The storage unit 250 is implemented by the various storage devices described above. The storage unit 250 stores data such as a dictionary DB (database) 252 and programs.

When the communication unit 210 acquires voice or the result of voice processing from the terminal device, the voice recognition unit 220 performs voice recognition and outputs character information converted into text, and the natural language processing unit 222 processes the character information. Semantic interpretation is performed while referring to the dictionary DB 252. The dictionary DB 252 associates abstracted semantic information with character information. The dictionary DB 252 may include synonyms and synonym list information. The processing of the speech recognition unit 220 and the processing of the natural language processing unit 222 are not clearly divided into stages, and the speech recognition unit 220 receives the processing result of the natural language processing unit 222 and corrects the recognition result. It may be done by influencing each other. The reply information generation unit 224 generates information (reply information) to be returned to the terminal device 100 based on the processing result of the natural language processing unit 222 . The reply information may be anything, for example, voice converted into another language. Note that the natural language processing unit 222 may be omitted, and the server apparatus 200 may simply output character information converted into text.

[Preprocessing]
The preprocessing executed by the preprocessing unit in the terminal device 100 will be described below. FIG. 6 is a diagram for explaining processing executed by a preprocessing unit; The sound picked up by the microphone 10 is supplied to, for example, the preprocessing unit 112-1 and at least the communication control unit 150-3. Note that the example shown in FIG. 6 is merely an example, and the sound picked up by the microphone 10 may be supplied to each preprocessing unit in parallel.

Pre-processing unit 112-1 outputs processing result (1) including processed sound (1) obtained by performing processing such as beamforming, noise cancellation, equalizing, etc. on the input sound to communication control unit 150-1. output to Further, preprocessing unit 112-1 may perform simple speech recognition on processed speech (1), and include the result in processing result (1) and output it to communication control unit 150-1. Communication control unit 150-1 causes in-vehicle communication device 60 to transmit processing result (1) to server device 200-1. Processed speech (1) is output to preprocessing section 112-2.

The preprocessing unit 112-2 performs, for example, the processing including the processed sound (2) obtained by performing the insufficient processing on the input processed sound (1) only by the processing of the preprocessing unit 112-1. Result (2) is output to communication control section 150-2. Further, preprocessing section 112-2 may perform simple speech recognition on processed speech (2), and include the result in processing result (2) and output it to communication control section 150-1. Communication control unit 150-2 causes in-vehicle communication device 60 to transmit processing result (2) to server device 200-2.

The preprocessing unit 122-3 may be omitted, and the voice picked up by the microphone 10 is input to the communication control unit 150-3 regardless of whether the preprocessing unit 122-3 exists or not. be. If the preprocessing unit 122-3 exists, the preprocessing unit 122-3 performs processing such as beamforming, noise cancellation, and equalizing on the input audio, and the processing result (3) including the processed audio (3) is 3) is output to communication control section 150-3. Further, preprocessing unit 112-1 may perform simple speech recognition on processed speech (3) and output the result to communication control unit 150-3 while including it in processing result (3). Communication control unit 150-3 instructs general-purpose communication device 70 via pairing application execution unit 152 to transmit at least the sound picked up by microphone 10 to server device 200-3. Communication control section 150-3 may further transmit processing result (3) to server device 200-3. Server device 200-3 is an example of a “first server device”.

Thus, according to the terminal device 100 of the embodiment, two or more preprocessing units that perform preprocessing corresponding to each of two or more server devices 200 among a plurality of server devices 200 having a speech recognition function are provided, The voice preprocessed by each of the two or more preprocessing units is transmitted to the corresponding server device 200 by the communication control unit 150 . As a result, it is possible to transmit the processing results that have been preprocessed according to the characteristics of the server devices 200 to the respective server devices 200 . For example, if one server device 200 has a strong noise tolerance and another server device 200 has a weak noise tolerance, data close to the voice picked up by the microphone 10 is transmitted to the former, and stronger data is transmitted to the latter. By transmitting the noise-cancelled data, it is possible to transmit data close to the requested quality to any of the server devices 200 . As a result, it is possible to effectively utilize a plurality of different server devices 200 each having a speech recognition function.

In the terminal device 100, the communication control unit 150-3 transmits at least the voice that has not been preprocessed by the preprocessing unit 122-3 to the server device 200-3. As a result, appropriate data for speech recognition can be transmitted to server device 200-3 requesting speech that has not been preprocessed.

Further, in the terminal device 100, a first OS for realizing a preprocessing unit that performs preprocessing according to the server device 200 other than the server device 200-3, and a preprocessing unit for transmitting to the server device 200-3 and a second OS for retrieving audio that has not been preprocessed. As a result, it is possible to eliminate the troublesomeness of arbitration or the like that is required when processing is performed using the same software.

In the terminal device 100, the preprocessing units 112-1 and 112-2 sequentially (serially, continuously) perform processing. Accordingly, when there is a process common to both, only the preprocessing unit 112-1 needs to implement the function related to the common process, so memory resources can be saved. In addition, since processing is not performed in parallel, concerns about exclusive processing results occurring in both are eliminated.

According to the terminal device 100 of the embodiment described above, it is possible to effectively utilize a plurality of different server devices each having a speech recognition function.

As described above, the mode for carrying out the present invention has been described using the embodiments, but the present invention is not limited to such embodiments at all, and various modifications and replacements can be made without departing from the scope of the present invention. can be added.

10 microphone 20 display/operation device 30 speaker unit 60 in-vehicle communication device 70 general-purpose communication device 100 terminal device 110 first management units 112-1, 112-2, 122-3 preprocessing units 116, 126 display control units 118, 128 voice Control unit 120 Second management unit 150-1, 150-2, 150-3 Communication control unit 152 Pairing application execution unit 200-1, 200-2, 200-3 Server device

Claims (8)

A terminal device mounted on a vehicle,
two or more preprocessing units that perform preprocessing corresponding to each of two or more server devices out of a plurality of server devices having a voice recognition function, on voice in the vehicle interior picked up by a microphone;
a communication control unit that transmits the audio preprocessed by each of the two or more preprocessing units to the corresponding server device using the communication unit ;
The communication control unit transmits at least the audio that has not been preprocessed by the preprocessing unit to a first server device among the plurality of server devices,
Among the two or more preprocessing units, a first OS that realizes a preprocessing unit that performs preprocessing according to a server device other than the first server device, and a preprocessing unit that does not perform preprocessing Equipped with a second OS for extracting the sound
Terminal equipment. A terminal device mounted on a vehicle,
two or more preprocessing units that perform preprocessing corresponding to each of two or more server devices out of a plurality of server devices having a voice recognition function, on voice in the vehicle interior picked up by a microphone;
a communication control unit that transmits the audio preprocessed by each of the two or more preprocessing units to the corresponding server device using the communication unit ;
Some or all of the two or more preprocessing units sequentially process,
Terminal equipment. The communication control unit transmits at least the audio that has not been preprocessed by the preprocessing unit to a first server device among the plurality of server devices.
3. The terminal device according to claim 2 . comprising a plurality of the communication control units corresponding to each of the plurality of server devices,
The terminal device according to any one of claims 1 to 3 . A control method for a terminal device mounted on a vehicle,
Each of the two or more preprocessing units provided in the terminal device responds to each of two or more server devices out of a plurality of server devices having a voice recognition function with respect to the voice in the vehicle interior picked up by the microphone. pretreatment,
transmitting the audio preprocessed by each of the two or more preprocessing units to the corresponding server device using the communication unit;
transmitting at least the audio that has not been preprocessed by the preprocessing unit to a first server device among the plurality of server devices;
Among the two or more preprocessing units, a first OS that implements a preprocessing unit that performs preprocessing according to a server device other than the first server device, and a preprocessing unit that does not perform preprocessing. Equipped with a second OS for extracting the sound
Terminal device control method. A control method for a terminal device mounted on a vehicle,
Each of the two or more preprocessing units provided in the terminal device responds to each of two or more server devices out of a plurality of server devices having a voice recognition function with respect to the voice in the vehicle interior picked up by the microphone. pretreatment,
transmitting the audio preprocessed by each of the two or more preprocessing units to the corresponding server device using the communication unit;
Some or all of the two or more preprocessing units sequentially process,
Terminal device control method. A program executed by a terminal device mounted on a vehicle,
to the terminal device,
Using each of the two or more preprocessing units provided in the terminal device, each of the two or more server devices out of a plurality of server devices having a voice recognition function for the voice in the vehicle cabin picked up by the microphone Perform pretreatment according to
transmitting the audio preprocessed by each of the two or more preprocessing units to the corresponding server device using the communication unit ;
transmitting at least the audio that has not been preprocessed by the preprocessing unit to a first server device among the plurality of server devices;
Among the two or more preprocessing units, a first OS that realizes a preprocessing unit that performs preprocessing according to a server device other than the first server device, and a preprocessing unit that does not perform preprocessing Equipped with a second OS for extracting the sound
program. A program executed by a terminal device mounted on a vehicle,
to the terminal device,
Using each of the two or more preprocessing units provided in the terminal device, each of the two or more server devices out of a plurality of server devices having a voice recognition function for the voice in the vehicle cabin picked up by the microphone Perform pretreatment according to
Some or all of the two or more pretreatment units are sequentially processed,
program.

Images