JP2023027697A - Terminal device, transmission method, transmission program and information processing system - Google Patents

Abstract

To appropriately transmit information about speech collected by a terminal device to other devices, even when the terminal device performs speech recognition.SOLUTION: A terminal device is a terminal device used by a user and performs speech recognition in the device itself, which includes a collection unit and a transmission unit. The collection unit collects speech sound uttered by the user and recognition results from speech recognition of the utterances in a storage unit in the self device in association with each other. When the speech sounds collected by the collection unit meets predetermined conditions, the transmission unit transmits information about the speech sound to a server device according to user's permission.SELECTED DRAWING: Figure 1

Description

The present invention relates to a terminal device, transmission method, transmission program, and information processing system.

Voice recognition for recognizing voice uttered by a user is used in various services. For example, there is provided a technique for recognizing speech using a model created by learning using utterance information and utterance content indicated by the utterance information (see, for example, Patent Document 1).

JP 2021-081527 A

However, with the conventional technology described above, it may be difficult to obtain information about the voice uttered by the user. For example, in the above-described prior art, since the server device recognizes the user's utterance (voice) and transmits the recognition result to the terminal device used by the user, the server device inevitably recognizes the user's utterance. (Voice) data can be acquired. On the other hand, in the case of so-called on-device type speech recognition, in which speech recognition is performed on the terminal device itself, speech recognition is completed without sending the user's utterance (voice) data from the terminal device to the server device. It is difficult for other devices such as a server device, which is a device other than the terminal device used by the user, to collect information about the voice uttered by the user.

The present application has been made in view of the above, and an object of the present invention is to appropriately transmit information on speech collected by a terminal device to another device even when the terminal device performs speech recognition. .

A terminal device according to the present application is a terminal device that is used by a user and performs voice recognition on its own device, and associates a voice uttered by the user with a recognition result of the utterance obtained by the voice recognition, a collection unit that collects data in a storage unit; and a transmission unit that transmits information about the sound to a server device in accordance with permission from the user when the sound collected by the collection unit satisfies a predetermined condition. It is characterized by

According to one aspect of the embodiments, even when the terminal device performs speech recognition, it is possible to appropriately transmit information about speech collected by the terminal device to another device.

FIG. 1 is an explanatory diagram showing an outline of information processing according to the embodiment. FIG. 2 is a diagram illustrating a configuration example of an information processing system according to the embodiment; FIG. 3 is a diagram illustrating a configuration example of a terminal device according to the embodiment; FIG. 4 is a diagram illustrating an example of a model information storage unit; FIG. 5 is a diagram illustrating an example of a collected information storage unit; FIG. 6 is a diagram illustrating a configuration example of a server device according to the embodiment; FIG. 7 is a diagram illustrating an example of a model information storage unit; FIG. 8 is a diagram showing an example of a learning data information storage unit. FIG. 9 is a flow chart showing a processing procedure according to the embodiment. FIG. 10 is a diagram illustrating an example of a hardware configuration;

Hereinafter, modes for implementing a terminal device, a transmission method, a transmission program, and an information processing system according to the present application (hereinafter referred to as "embodiments") will be described in detail with reference to the drawings. Note that the terminal device, transmission method, transmission program, and information processing system according to the present application are not limited to this embodiment. Also, in the following embodiments, the same parts are denoted by the same reference numerals, and overlapping descriptions are omitted.

(embodiment)
[1. information processing]
First, with reference to FIG. 1, an overview of information processing performed by an information processing system 1 according to an embodiment will be described. FIG. 1 is an explanatory diagram showing an outline of information processing according to the embodiment. In FIG. 1 , the server device 100 transmits to the terminal device 10 information on conditions for designating voice requesting information transmission from the terminal device 10 , and the terminal device 10 receives information on voice that satisfies the conditions received from the server device 100 . A case of transmitting to the server apparatus 100 will be described as an example. Note that if the server device 100 can acquire the information about the voice from the terminal device 10, the terminal device 10 can transmit the information about the voice to the server device 100 based on the preset conditions without specifying the conditions. may be sent.

Moreover, below, the terminal device 10 may be described as a user. That is, hereinafter, the user can also be read as the terminal device 10 . In FIG. 1, a case where the terminal device 10 is a smartphone will be described as an example. device (equipment), which will be described later in detail. In FIG. 1, a case where the terminal device 10 transmits voice data to the server device 100 as information about voice will be described as an example. The information is not limited to this and may be various kinds of information, but the details of this point will be described later.

An example of information processing will be described below with reference to FIG. For the sake of explanation, FIG. 1 shows a case where one utterance is targeted for transmission. In some cases, transmission may be performed with the user's transmission permission.

FIG. 1 shows a case where the user is identified by a user ID "U1" (hereinafter sometimes referred to as "user U1"). The terminal device 10 used by the user U1 performs speech recognition within itself using the model M1, which is a speech recognition model, and provides the user U1 with a service according to the speech recognition result. In FIG. 1, for the sake of explanation, the case where the model M1 converts voice information of the user's utterance into characters is taken as an example. ) to perform various functions.

First, the server device 100 transmits to the terminal device 10 information indicating a condition CN1 for designating a voice to be provided to the terminal device 10 (step S11). The terminal device 10 stores the information indicating the received condition CN1 in the storage unit 120 (see FIG. 3). For example, the terminal device 10 compares the condition CN1 stored in the storage unit 120 with information about voice, determines whether the voice satisfies the condition CN1, and transmits voice data that satisfies the condition CN1 to the server device 100. do. To simplify the explanation, it is assumed that the condition CN1 includes a specific word (hereinafter referred to as "new word NX"), and other conditions will be exemplified later.

First, user U1 speaks "XXXX". It is assumed that "XXXX" is an utterance containing specific content. The terminal device 10 detects the utterance PA of the user U1, and receives as an input voice data of "XXXX" which is the utterance PA of the user U1 (step S12).

Then, the terminal device 10 performs speech recognition processing using the speech data of "XXXX" received as input and the model M1 (step S13). The terminal device 10 inputs voice data of "XXXX" to the model M1 and causes the model M1 to output character data, thereby performing processing (voice recognition processing) of converting the voice into characters. In FIG. 1, the model M1 to which voice data of "XXXX" is input outputs character data of "XXXX". Note that the model M1 may output a score indicating the accuracy of speech recognition along with the characters corresponding to the input speech. It is also assumed that the character data of "XXXX" includes the new word NX.

Then, the terminal device 10 associates the voice uttered by the user with the recognition result of the voice recognition of the utterance and collects them in the storage unit DB (step S14). In FIG. 1 , the terminal device 10 associates an utterance PA uttered by the user U1 with the recognition result of the utterance PA and collects them in the storage unit 120 . For example, the terminal device 10 associates the voice data of "XXXX", which is the utterance PA, with the character data of "XXXX", which is the recognition result of the voice data, and stores the collected information storage unit 142 (FIG. 3) in the terminal device 10. reference).

Then, the terminal device 10 determines whether or not the collected voice satisfies the conditions, and if it determines that the conditions are satisfied, the terminal device 10 notifies the user (step S15). In FIG. 1, the terminal device 10 determines whether or not the collected utterance PA satisfies the condition CN1. For example, the terminal device 10 compares the character data of the utterance PA with the condition CN1, and determines whether the utterance PA satisfies the condition CN1. In this manner, the terminal device 10 determines whether or not the utterance PA includes predetermined content using the condition CN1, which is a condition regarding content (also referred to as a "content condition"). Since the character data of the collected utterance PA includes the new word NX, the terminal device 10 determines that the condition CN1 is satisfied.

Therefore, the terminal device 10 notifies the user U1 of the utterance PA as information that is a candidate for transmission to the server device 100 . For example, the terminal device 10 displays the utterance PA as a candidate for transmission to the server device 100 . For example, the terminal device 10 displays on the screen the character string "XXXX" and an explanation indicating that it is a candidate. In this case, the terminal device 10 may display information indicating that the utterance PA is a candidate as well as information for accepting information indicating whether or not user U1's permission is granted. For example, the terminal device 10 displays information indicating that the utterance PA is a candidate, and a button for the user to specify whether or not to transmit it. For example, the terminal device 10 displays information indicating that the utterance PA is a candidate, an approval button stating "permit transmission" or the like and a rejection button stating "not permitting transmission" or the like. good too. In this way, the terminal device 10 confirms the permission from the user when the conditions are satisfied.

In this case, the terminal device 10 accepts user U1's permission to transmit the utterance PA to the server device 100 by pressing the permission button. For example, when the user U1 selects the consent button, the terminal device 10 determines that the user U1 has permitted transmission of the utterance PA to the server device 100 . Moreover, when the user U1 selects the reject button, the terminal device 10 determines that the user U1 did not approve the transmission of the utterance PA to the server device 100 . Note that the above is merely an example, and the terminal device 10 may receive notifications and approvals in various modes (modals) without being limited to receiving notifications and approvals through displays. For example, the terminal device 10 may notify (output) to the user U1 that it is a candidate for transmitting the utterance PA to the server device 100 by voice. In addition, the terminal device 10 may accept whether or not the user U1 permits the transmission of the utterance PA to the server device 100 by voice.

Terminal device 10 accepts permission from user U1 to transmit utterance PA to server device 100 . (Step S16). For example, terminal device 10 accepts user U1's permission to transmit utterance PA to server device 100 by user U1's operation of selecting a consent button.

Then, the terminal device 10 transmits, to the server device 100, information regarding the utterance PA, for which transmission is permitted by the user U1 (step S17). In FIG. 1 , the terminal device 10 transmits voice data of the utterance PA and its recognition result to the server device 100 . That is, the terminal device 10 transmits to the server device 100 the voice data of "XXXX" which is the utterance PA and the character data of "XXXX" which is the recognition result of the voice data. Note that the above is merely an example, and the terminal device 10 may transmit only the voice data of the utterance PA to the server device 100 .

The server device 100 adds the information about the voice received from the terminal device 10 to the data used for learning (step S18). In FIG. 1, a combination of voice data of "XXXX", which is the utterance PA, and character data of "XXXX", which is the recognition result of the voice data (hereinafter also referred to as "new data PDT"), is received from the terminal device 10. Server device 100 adds the received new data PDT to data set DS1, which is a learning data set. For example, the server device 100 stores the new data PDT in which the character data of "XXXX" as a label is associated with the voice data of "XXXX", which is the utterance PA, as the data of the data set DS1 for learning data information storage unit 122 ( (see FIG. 8).

Then, the server device 100 learns the model M1 using the data set DS1 to which the new data PDT has been added (step S19). The server device 100 learns (updates) parameters such as weights of the model M1 using the data set DS1. Any method can be adopted for the learning process of the model M1.

For example, the server device 100 learns by a method such as back propagation so that the character data output by the model M1 approaches the correct data (label) corresponding to the speech data input to the model M1. process. For example, the server device 100 adjusts the value of the weight (that is, the connection coefficient) that is taken into account when values are transmitted between nodes by learning processing. In this way, the server device 100 learns the model M1 by processing such as back propagation that corrects the parameters (connection coefficients) so as to reduce the error between the output of the model M1 and the correct data corresponding to the input. For example, the server device 100 generates the model M1 by performing processing such as back propagation so as to minimize a predetermined loss function. As a result, the server device 100 can perform learning processing for learning the parameters of the model M1.

As described above, the information processing system 1 can appropriately transmit information about voices collected by the terminal device 10 to the server device 100 even when the terminal device 10 performs voice recognition. Therefore, the information processing system 1 can collect data necessary for learning a model even when the terminal device 10 performs speech recognition.

For example, the information processing system 1 saves the uttered voice and the recognition result in the terminal device 10 ( accumulate. For example, the information processing system 1 stores (accumulates) the uttered voice and the recognition result in the terminal device 10 when using the application on the terminal device 10 in which any application such as a car navigation application or a shopping application is installed. do. Then, when the accumulated (collected) speech satisfies a predetermined criterion, the information processing system 1 confirms with the user permission for transmission to the server device 100, and if the user's permission is obtained, the permission is obtained. The information received is transmitted from the terminal device 10 to the server device 100 . With such a process, the information processing system 1 can appropriately transmit information about voices collected by the terminal device 10 to the server device 100 even when the terminal device 10 performs voice recognition.

[1-1. Other examples]
Note that the processing shown in FIG. 1 is merely an example, and the information processing system 1 may transmit various information from the terminal device 10 to the server device 100 using various conditions. In this regard, examples of each element are described below.

[1-1-1. Information to be sent]
In FIG. 1, the case where voice data is transmitted from the terminal device 10 to the server device 100 has been described as an example. may be

The terminal device 10 may transmit voice data to the server device 100 . The terminal device 10 may transmit voice waveform data to the server device 100 . The terminal device 10 may transmit data obtained by compressing voice data to the server device 100 . The terminal device 10 may transmit feature information extracted from the voice to the server device 100 .

As described above, the information transmitted from the terminal device 10 to the server device 100 may be various information such as voice waveforms or compressed versions thereof. The information to be transmitted from the terminal device 10 to the server device 100 may be a feature quantity extracted from the voice waveform. The feature amount here is, for example, data whose size is smaller than that of the original voice data, and may be spectrum information or the like that contains as little information about individuality as possible. The information to be transmitted from the terminal device 10 to the server device 100 may be data of compressed voice such as voice in which the size is reduced as much as possible by irreversibly compressing the utterance content to a level that allows listening confirmation.

[1-1-2. Information conditions]
In FIG. 1, the condition CN1, which is whether or not the voice contains a specific word, has been described as an example. Not limited to this, various conditions may be used.

The transmission information condition may be a condition related to the number of collected utterances (speech) (also referred to as a “number condition”). For example, the transmission information condition may be that the number of collected utterances (speech) exceeds a certain threshold. In this case, the terminal device 10 determines whether or not the number of collected voices is equal to or greater than a predetermined number (eg, 50, 100, etc.). For example, when the number of collected voices reaches or exceeds a predetermined number, the terminal device 10 determines that the condition is satisfied, and transmits information about voices to the server device 100 according to the user's permission.

For example, the transmission information condition may be that the score related to voice recognition of collected utterances (speech) satisfies a predetermined condition (also referred to as "score condition"). In this case, the terminal device 10 determines whether or not the speech recognition score of the collected speech is equal to or greater than a predetermined threshold value (eg, 0.5, 0.7, etc.). For example, when the voice recognition score of the collected voice is equal to or greater than a predetermined threshold, the terminal device 10 determines that the condition is satisfied, and transmits information about the voice to the server device 100 according to the user's permission.

In addition, for example, the terminal device 10 determines whether or not the score regarding voice recognition of the collected voice is less than a predetermined threshold (for example, 0.5, 0.7, etc.). For example, when the voice recognition score of the collected voice is less than a predetermined threshold, the terminal device 10 determines that the condition is satisfied, and transmits information about the voice to the server device 100 according to the user's permission.

Further, for example, the transmission information condition may be that the collected voice satisfies a noise condition (also referred to as a “noise condition”). In this case, the terminal device 10 determines whether or not the signal-to-noise ratio (SN ratio) of the collected voice is equal to or greater than a predetermined value. For example, when the SN ratio of the collected voice is equal to or greater than a predetermined value, the terminal device 10 determines that the condition is satisfied, and transmits information about the voice to the server device 100 according to the user's permission.

Also, for example, the terminal device 10 determines whether or not the signal-to-noise ratio (SN ratio) of the collected voice is less than a predetermined value. For example, when the SN ratio of the collected voice is less than a predetermined value, the terminal device 10 determines that the condition is satisfied, and transmits information about the voice to the server device 100 according to the user's permission.

Note that the above is merely an example, and the information processing system 1 may appropriately use various conditions. The transmission information condition may be the type of the terminal device 10 . For example, the information processing system 1 may set a condition (terminal condition) that the type of the terminal device 10 is "car navigation" when the user's utterance (voice) in the car navigation system is insufficient. The transmission information condition may be a user attribute. For example, when the child's utterance (voice) is insufficient, the information processing system 1 sets the condition (the speaker condition). Note that the above is only an example, and the information processing system 1, when collecting a child's utterance (sound), uses a discriminator for "adult/child" and "age group" to classify the child's utterance (sound). may be collected. In this case, the information processing system 1 may collect the child's utterances (speech) using a discriminator that uses not only the fundamental frequency but also the spectral information as the feature amount.

Further, for example, the information processing system 1 may use a combination of the conditions described above. For example, when the number of voices satisfying at least one of the content condition, the score condition, or the noise condition satisfies the number condition, the terminal device 10 transmits the information about the voice to the server device 100 according to the permission of the user. good too. For example, the terminal device 10 collects sounds that satisfy at least one of the content condition, score condition, or noise condition as relevant sounds. Then, when the number of corresponding sounds reaches a predetermined number (for example, 30, 150, etc.), the terminal device 10 may transmit the information about the sounds to the server device 100 according to the permission of the user.

In this way, the terminal device 10 selects utterances judged to contribute to the improvement of the speech recognition engine from among the collected voices (user utterances), and obtains transmission permission when the number of utterances exceeds a certain number. good. For example, a score given to each utterance by the speech recognition engine may be used as a criterion for determining that the speech recognition engine contributes to improvement. A signal-to-noise ratio (SNR) for each utterance may be used as a criterion for determining that it contributes to improvement of the speech recognition engine. For example, the recognition result text for each utterance may be used as a criterion for determining that it contributes to the improvement of the speech recognition engine. For example, the information processing system 1 may select utterances that contain desired words for improving the speech recognition engine.

[1-1-3. Transmission timing]
In FIG. 1, the case where the information is transmitted at the timing when the user's permission is obtained has been described as an example, but any timing can be adopted as the timing for transmitting the information to the server device 100 .

After accepting the user's permission for the voice, the terminal device 10 transmits information about the voice to the server device 100 at a predetermined timing. After receiving the user's permission for the voice, the terminal device 10 transmits information about the voice to the server device 100 while the communication environment satisfies a predetermined communication condition. For example, after accepting the user's permission for voice, the terminal device 10 transmits information about the voice to the server device 100 while communicating by Wi-Fi (registered trademark) (Wireless Fidelity).

Further, after accepting the user's permission for the voice, the terminal device 10 transmits information about the voice to the server device 100 at a timing when the usage rate of the terminal device 10 is low. For example, the terminal device 10 receives the user's permission for the voice, and then transmits information about the voice to the server device 100 at the timing when the usage rate of the processor of the terminal device 10 becomes less than a predetermined threshold. For example, the terminal device 10, after accepting the user's permission for audio, transmits information about the audio to the server device 100 while the terminal device 10 is being charged.

As described above, the terminal device 10 is connected to Wi-Fi after obtaining the user's transmission permission, or when the usage rate of the terminal device 10 by the user is low (when the CPU load is less than a predetermined value or at home late at night). The information is transmitted to the server device 100 at the time of charging, etc.).

In addition, the terminal device 10 may preferentially transmit information on high-value voices according to the value of each voice. The terminal device 10 assigns priority (ranking) according to the score of voice recognition, and transmits to the server device 100 in order from the information regarding the voice with the highest priority. For example, the terminal device 10 assigns a higher priority (order) to the one with the higher speech recognition score, and transmits to the server device 100 the information about the speech with the highest priority.

[1-1-4. Selection by User]
As described above, the terminal device 10 transmits to the server device 100 only the information about the voices that the user has approved among the candidates notified to the user. For example, when obtaining permission for transmission from the user, the terminal device 10 presents the user with a list of utterances to be transmitted (recognition result text and links to utterances, etc.). Exclude user-selected utterances from being sent.

[1-1-5. incentive]
The information processing system 1 may provide an incentive to the user in order to obtain permission for transmission from the user. For example, when the user approves the transmission of voice, the information processing system 1 may provide the user with various types of incentives such as electronic money, points, and coupons according to the voice that has been approved.

For example, the terminal device 10 may notify the user of information indicating an incentive to be provided to the user when information about voice is transmitted to the server device 100 . For example, the terminal device 10 may notify the user that the higher the value of the voice, the better the user's incentive. For example, the terminal device 10 may notify the user that the higher the voice score, the more points the user will receive.

For example, the terminal device 10 may notify the user that an incentive will be provided to the user when the user labels the voice (correct answer). For example, the terminal device 10 may notify the user that an incentive will be provided to the user when the user checks the speech recognition result whose score is less than a predetermined threshold and corrects the error. .

[2. Configuration example of information processing system]
Next, the configuration of the information processing system 1 including the server device 100 according to the embodiment will be described with reference to FIG. FIG. 2 is a diagram showing a configuration example of the information processing system 1 according to the embodiment. As shown in FIG. 2, the information processing system 1 according to the embodiment includes a plurality of terminal devices 10 and a server device 100. As shown in FIG. These various devices are communicatively connected via a network N by wire or wirelessly. The network N is, for example, a LAN (Local Area Network) or a WAN (Wide Area Network) such as the Internet.

Also, the number of devices included in the information processing system 1 shown in FIG. 2 is not limited to the illustrated one. For example, in FIG. 2, only three terminal devices 10-1, 10-2, and 10-3 are shown for simplification of illustration, but this is only an example and not a limitation, and four It may be more than one.

The terminal device 10 is an information processing device (computer) that is used by a user and provides services based on voice recognition to the user through voice recognition within the device itself. The terminal device 10 is a wireless communication network such as LTE (Long Term Evolution), 4G (4th Generation), 5G (5th Generation: fifth generation mobile communication system), Bluetooth (registered trademark), wireless LAN (Local Area Network). It is possible to connect to the network N and communicate with the server apparatus 100 via short-range wireless communication, such as.

In FIG. 2, the terminal device 10-1 is a smart phone used by a user. The terminal device 10-1 may be a smart device such as a tablet terminal, a feature phone, a PC (Personal Computer), a PDA (Personal Digital Assistant), a car navigation system, a smart watch, or a head device, as long as the device is used by the user. It may be a wearable device such as a mount display, smart glasses, or the like.

In FIG. 2, the terminal device 10-2 is a smart speaker used by the user. Note that the terminal device 10-2 may be any IoT (Internet of Things) such as a television or a refrigerator as long as it is a device used by the user.

In FIG. 2, the terminal device 10-3 is a moving object such as a vehicle such as an automobile equipped with a car navigation (car navigation) function. Note that the terminal device 10-3 may be a car navigation device that provides a car navigation function provided in a moving body.

The server device 100 is an information processing device (computer) that acquires information about voice from the terminal device 10 . Also, the server device 100 is a learning device that learns a model used for speech recognition by machine learning using information about acquired speech. The server device 100 also transmits the model to the terminal device 10 . In addition, the server device 100 transmits to the terminal device 10 information indicating conditions for designating data to be transmitted to the terminal device 10 . Note that the server device 100 may have only a function of acquiring information about voice from the terminal device 10 and accumulating the acquired information. In this case, the information processing system 1 may include a device (learning device) that performs model learning using information accumulated by the server device 100 . That is, in the information processing system 1, a device (for example, a server device 100) that receives and accumulates information related to speech such as speech data from the terminal device 10, holds a speech recognition model and performs learning or updating with collected data, It may be separate from a device (for example, a learning device) that transmits (distributes) to the terminal device 10 . In this case, the learning device may hold a speech recognition model, use data acquired from the server device 100 to learn and update the model, and transmit the model to the terminal device 10 used by the user. For example, in the information processing system 1, information related to speech such as speech data collected by the server device 100 is separately manually listened to and given a correct text, or mechanically sorted to create a new speech recognition model. You can learn and update. Further, in the information processing system 1, the model may be distributed as a version upgrade to the terminal device 10 used by the user by attaching it to the binary of the application and via an application download service such as an application store. . Note that the configuration of the information processing system 1 is not limited to the above configuration, and any configuration can be adopted as long as the server device 100 acquires information about voice from the terminal device 10 .

[3. Configuration example of terminal device]
Next, the configuration of the terminal device 10 will be described using FIG. FIG. 3 is a diagram showing a configuration example of the terminal device 10. As shown in FIG. As shown in FIG. 3 , the terminal device 10 has a communication section 11 , an input section 12 , an output section 13 , a storage section 14 , a control section 15 and a sensor section 16 . Note that the terminal device 10 may have any configuration as long as it can collect data and provide it to the server device 100 . For example, the terminal device 10 may have any other configuration as long as it has a communication unit 11 that communicates with the server device 100 and a control unit 15 that performs processing for collecting data. Depending on the type of the terminal device 10 , for example, the terminal device 10 may not have any of the input unit 12 , the output unit 13 , the storage unit 14 and the sensor unit 16 .

It should be noted that the terminal device 10 may have any configuration, not limited to the above, depending on the mode of implementation. For example, when the terminal device 10 is a mobile object, the terminal device 10 may be configured to have a mechanism for realizing movement, such as a drive unit (motor).

(Communication unit 11)
The communication unit 11 is implemented by, for example, a NIC, a communication circuit, or the like. The communication unit 11 is connected to a network N (Internet or the like) by wire or wirelessly, and transmits and receives information to and from other devices such as the server device 100 via the network N.

(Input unit 12)
The input unit 12 receives various inputs. The input unit 12 receives a user's operation. For example, the input unit 12 receives a user's voice input via a voice sensor 161 such as a microphone. The input unit 12 receives various operations by user's speech.

Further, the input unit 12 may accept an operation (user operation) to the terminal device 10 other than the user's utterance (voice) as an operation input by the user. The input unit 12 may receive, via the communication unit 11, information regarding user operations using a remote controller (remote controller). Also, the input unit 12 may have buttons provided on the terminal device 10 or a keyboard and mouse connected to the terminal device 10 .

For example, the input unit 12 may have a touch panel capable of realizing functions equivalent to those of a remote controller, keyboard, or mouse. In this case, various information is input to the input unit 12 via the display (output unit 13). The input unit 12 receives various operations from the user via the display screen using a touch panel function realized by various sensors. That is, the input unit 12 receives various operations from the user via the display (output unit 13) of the terminal device 10. FIG. For example, the input unit 12 receives a user's operation via the display (output unit 13) of the terminal device 10. FIG.

(Output unit 13)
The output unit 13 outputs various information. The output unit 13 has a function of displaying information. The output unit 13 is provided in the terminal device 10 and displays various information. The output unit 13 is realized by, for example, a liquid crystal display or an organic EL (Electro-Luminescence) display. The output unit 13 may have a function of outputting sound. For example, the output unit 13 has a speaker that outputs audio.

(storage unit 14)
The storage unit 14 is realized by, for example, a semiconductor memory device such as a RAM or flash memory, or a storage device such as a hard disk or an optical disk. The storage unit 14 stores various information necessary for collecting data. The storage unit 14 has a model information storage unit 141 and a collected information storage unit 142 .

(Model information storage unit 141)
The model information storage unit 141 according to the embodiment stores information (model data) indicating the structure of a model (network). FIG. 4 is a diagram illustrating an example of a model information storage unit; In the example shown in FIG. 4, the model information storage unit 141 includes items such as "model ID", "usage", and "model data".

"Model ID" indicates identification information for identifying a model. "Use" indicates the use of the corresponding model. "Model data" indicates model data. FIG. 4 shows an example in which conceptual information such as "MDT1" is stored in "model data", but in reality, various types of information that make up the model, such as network information and functions included in the model, are stored. included.

In the example shown in FIG. 4, the model (model M1) identified by the model ID "M1" indicates that the application is "speech recognition". Model M1 indicates that it is a model used for speech recognition. It also indicates that the model data of the model M1 is the model data MDT1.

Note that the model information storage unit 141 may store various types of information, not limited to the above, depending on the purpose. For example, the model information storage unit 141 stores parameter information of the model learned (generated) by the learning process.

(Collected information storage unit 142)
The collected information storage unit 142 according to the embodiment stores information on the user's utterance (voice) collected by the terminal device 10 . FIG. 5 is a diagram illustrating an example of a collected information storage unit; In the example shown in FIG. 5, the collected information storage unit 142 includes items such as "voice ID", "voice", and "recognition result".

"Voice ID" indicates identification information for identifying the collected voice. "Voice" indicates the collected voice. FIG. 5 shows an example in which conceptual information such as "ADT1" is stored in "speech", but in reality, various information related to speech such as collected speech data, for example speech waveform data, is included. "Recognition result" indicates the recognition result of the corresponding voice. FIG. 5 shows an example in which conceptual information such as "RS1" is stored in "recognition result". and information indicating the contents included in the voice.

The example shown in FIG. 5 indicates that the recognition result of the voice (voice AD1) identified by the voice ID "AD1" is the recognition result RS1.

It should be noted that the collected information storage unit 142 may store various types of information, not limited to the above, depending on the purpose. For example, the collected information storage unit 142 stores each sound in association with information indicating under what circumstances the sound was detected. For example, the collected information storage unit 142 stores each sound in association with the SN ratio of the sound.

(control unit 15)
Returning to FIG. 3, the description is continued. The control unit 15 is a controller, and for example, a CPU (Central Processing Unit), an MPU (Micro Processing Unit), or the like processes information processing such as a program (for example, a transmission program related to a program stored inside the terminal device 10). program) is executed using a RAM or the like as a work area. Also, the control unit 15 is a controller, and may be realized by an integrated circuit such as an ASIC (Application Specific Integrated Circuit) or an FPGA (Field Programmable Gate Array).

As shown in FIG. 3, the control unit 15 includes a receiving unit 151, a speech recognition unit 152, a collection unit 153, a notification unit 154, a reception unit 155, a determination unit 156, and a transmission unit 157. , implements or performs the information processing functions and actions described below. Note that the internal configuration of the control unit 15 is not limited to the configuration shown in FIG. 3, and may be another configuration as long as it performs information processing to be described later.

(Receiver 151)
The receiving unit 151 receives various information. The receiving unit 151 receives various information from an external information processing device. The receiving unit 151 receives various information from other information processing devices such as the server device 100 .

Receiving unit 151 receives, from server device 100, information indicating a condition related to voice for which server device 100 requests information provision.

The receiving unit 151 receives the model learned by the server device 100 from the server device 100 . The receiving unit 151 receives the speech recognition model from the server device 100 . Receiving unit 151 receives model M1 from server device 100 .

(Voice recognition unit 152)
The speech recognition unit 152 executes various processes related to speech recognition. The speech recognition unit 152 uses the information stored in the storage unit 14 to perform speech recognition processing. The speech recognition unit 152 executes speech recognition processing using a speech recognition model. The speech recognition unit 152 uses the speech recognition model received by the reception unit 151 to perform speech recognition processing. The speech recognition unit 152 uses the model M1 received by the reception unit 151 to perform speech recognition processing. For example, the speech recognition unit 152 uses the model M1 stored in the model information storage unit 141 to perform speech recognition processing.

The voice recognition unit 152 uses the model M1 to convert the user's utterance (voice) into character information (character data), thereby converting the voice of the user's utterance into text. Also, the speech recognition unit 152 analyzes the content of the user's speech. The voice recognition unit 152 estimates the content of the user's utterance by analyzing the user's utterance using various conventional techniques as appropriate. For example, the speech recognition unit 152 may analyze the contents of the user's utterances using a function of natural language understanding (NLU) or automatic speech recognition (ASR).

(Collection unit 153)
The collection unit 153 collects various types of information. The collection unit 153 determines collection of various information. The collection unit 153 collects various types of information based on information from an external information processing device. The collection unit 153 collects various types of information based on the information stored in the storage unit 14 . The collection unit 153 collects data by sensing using the model M1 stored in the model information storage unit 141 .

The collection unit 153 collects the speech uttered by the user and the recognition result of the speech recognition in the storage unit 14 in the own device in association with each other. The collecting unit 153 associates the speech with the recognition result of the speech and stores them in the collected information storage unit 142 . The collection unit 153 associates the voice uttered by the user with the result of voice recognition by the voice recognition unit 152 and registers them in the collected information storage unit 142 .

(Notification unit 154)
The notification unit 154 executes processing related to notification to the user. The notification unit 154 notifies the user of information. The notification unit 154 notifies the user of information via the output unit 13 .

The notification unit 154 notifies the user of information about voices that are candidates for transmission to the server device 100 . The notification unit 154 displays on the output unit 13 list information of candidates to be transmitted to the server device 100 .

Notification unit 154 notifies information indicating an incentive to be provided to the user when information about voice is transmitted to server device 100 . Notification unit 154 displays on output unit 13 information indicating an incentive to be provided to the user when information about voice is transmitted to server device 100 .

(Reception unit 155)
The reception unit 155 receives various types of information. The accepting unit 155 accepts various operations by the user. For example, the reception unit 155 receives various operations by the user via the input unit 12 .

The accepting unit 155 accepts permission from the user. Accepting unit 155 accepts the user's permission for the candidate to be transmitted to server device 100 notified by notifying unit 154 .

(Determination unit 156)
The determination unit 156 determines various information. For example, the determination unit 156 determines various information based on various information received from the external device by the reception unit 151 . For example, the determination unit 156 determines various types of information based on information stored in the storage unit 14 . For example, the determination unit 156 makes a determination using information indicating conditions regarding data collection stored in the storage unit 14 .

The determination unit 156 determines information to be transmitted to the server device 100 . The determination unit 156 determines whether or not the voice satisfies a condition regarding information to be transmitted to the server device 100 . The determination unit 156 determines the timing of transmitting information to the server device 100 .

The determination unit 156 determines whether or not the voice collected by the collection unit 153 satisfies a predetermined condition. The determination unit 156 determines whether or not the number of voices collected by the collection unit 153 is equal to or greater than a predetermined number. The determination unit 156 determines whether or not the score regarding voice recognition of the voice collected by the collection unit 153 satisfies a predetermined condition. The determination unit 156 determines whether or not the sound collected by the collection unit 153 satisfies the noise condition. The determination unit 156 determines whether or not the utterance corresponding to the voice collected by the collection unit 153 includes predetermined content.

The determination unit 156 determines whether or not a predetermined condition designated by the server device 100 is satisfied. The determination unit 156 determines whether or not the predetermined condition received from the server device 100 is satisfied.

The determination unit 156 transmits to the server device 100 information about the voices approved by the user among the candidates notified by the notification unit 154 . The determination unit 156 determines whether or not it is time to transmit information to the server device 100 . The determination unit 156 determines whether the communication environment satisfies predetermined communication conditions. The determination unit 156 determines whether or not it is timing when the usage rate of the terminal device by the user is low.

(transmitting unit 157)
The transmission unit 157 transmits various types of information to an external information processing device. For example, the transmission unit 157 transmits various information to other information processing devices such as the server device 100 . Transmitter 157 transmits information stored in storage 14 .

The transmission unit 157 transmits various types of information based on information from other information processing devices such as the server device 100 . The transmitting section 157 transmits various information based on the information stored in the storage section 14 .

The transmission unit 157 transmits information to the server device 100 according to the determination result by the determination unit 156 . If the determination unit 156 determines that the information transmission condition is satisfied, the transmission unit 157 transmits the information to the server device 100 . When the voice collected by the collecting unit 153 satisfies a predetermined condition, the transmitting unit 157 transmits information about the voice to the server device 100 that learns a model related to voice recognition according to user's permission.

The transmission unit 157 transmits the voice data to the server device 100 . The transmitting unit 157 transmits the voice waveform data to the server device 100 . The transmission unit 157 transmits data obtained by compressing the voice data to the server device 100 . The transmission unit 157 transmits the feature information extracted from the voice to the server device 100 .

When the number of sounds collected by the collection unit 153 is equal to or greater than a predetermined number, the transmission unit 157 transmits information about the sound to the server device 100 in accordance with the permission. If the score regarding voice recognition of the voice collected by the collecting unit 153 satisfies a predetermined condition, the transmitting unit 157 transmits information regarding the voice to the server device 100 in response to permission. If the sound collected by the collection unit 153 satisfies the noise condition, the transmission unit 157 transmits information on the sound to the server device 100 in response to permission. When the utterance corresponding to the voice collected by the collecting unit 153 includes predetermined content, the transmitting unit 157 transmits information about the voice to the server device 100 in accordance with the permission. If a predetermined condition specified by server device 100 is satisfied, transmission unit 157 transmits information about audio to server device 100 in response to permission.

Transmitter 157 transmits to server device 100 information about the voices that have been approved by the user among the candidates notified by notifier 154 . After the user's permission is accepted by the accepting unit 155, the transmitting unit 157 transmits the information about the voice to the server device 100 at a predetermined timing. Transmitter 157 transmits information about voice to server device 100 while the communication environment satisfies a predetermined communication condition. The transmission unit 157 transmits the information about the voice to the server device 100 at a timing when the usage rate of the terminal device by the user is low. Transmitter 157 transmits information about audio that satisfies a predetermined condition received from server device 100 to server device 100 in accordance with the user's permission.

(Sensor unit 16)
The sensor unit 16 has sensors that detect various sensor information. In the example of FIG. 3 , the sensor section 16 has a voice sensor 161 .

(Audio sensor 161)
The audio sensor 161 is, for example, a microphone or the like, and detects audio. For example, the audio sensor 161 detects user speech. Note that the voice sensor 161 may have any configuration as long as it can detect the user's speech information necessary for processing.

In addition, the sensor part 16 may have not only the above but various sensors. The sensor unit 16 includes an image sensor, a position sensor, an acceleration sensor, a gyro sensor, a temperature sensor, a humidity sensor, an illuminance sensor, a pressure sensor, a proximity sensor, and sensors for receiving biometric information such as odor, sweat, heart rate, pulse, and brain waves. It may have various sensors such as sensors. Further, the sensors for detecting the above various information in the sensor unit 16 may be a common sensor, or may be implemented by different sensors.

[4. Configuration example of server device]
Next, the configuration of the server device 100 according to the embodiment will be described using FIG. FIG. 6 is a diagram showing a configuration example of the server device 100 according to the embodiment. As shown in FIG. 6, the server device 100 has a communication section 110, a storage section 120, and a control section .

(Communication unit 110)
The communication unit 110 is realized by, for example, a NIC (Network Interface Card) or the like. Also, the communication unit 110 is connected to the network N (see FIG. 2) by wire or wirelessly.

(storage unit 120)
The storage unit 120 is realized by, for example, a semiconductor memory device such as a RAM (Random Access Memory) or flash memory, or a storage device such as a hard disk or an optical disk. As shown in FIG. 6 , the storage unit 120 has a model information storage unit 121 and a learning data information storage unit 122 . Note that the storage unit 120 stores various information, not limited to the above. The storage unit 120 stores information indicating various conditions regarding transmission of information. For example, the storage unit 120 stores information (information condition information) indicating conditions of information to be transmitted to the server device 100 . For example, the storage unit 120 stores information (timing condition information) indicating a condition of timing for transmission to the server device 100 .

(Model information storage unit 121)
The model information storage unit 121 according to the embodiment stores information about models. For example, the model information storage unit 121 stores a common model (global model) learned for users and communities. 7 is a diagram illustrating an example of a model information storage unit according to the embodiment; FIG. The model information storage unit 121 shown in FIG. 7 includes items such as "model ID", "usage", and "model data".

"Model ID" indicates identification information for identifying a model. "Use" indicates the use of the corresponding model. "Model data" indicates model data. FIG. 7 shows an example in which conceptual information such as "MDT1" is stored in "model data", but in reality, various types of information that make up the model, such as network information and functions included in the model, are stored. included.

In the example shown in FIG. 7, the model (model M1) identified by the model ID "M1" indicates that the application is "speech recognition". Model M1 indicates that it is a model used for speech recognition. It also indicates that the model data of the model M1 is the model data MDT1.

Note that the model information storage unit 121 may store various types of information, not limited to the above, depending on the purpose. For example, the model information storage unit 121 stores parameter information of the model learned (generated) by the learning process. Note that the server device 100 may not have the model information storage unit 121 if another device (learning device or the like) performs model learning.

(Learning data information storage unit 122)
The learning data information storage unit 122 according to the embodiment stores various types of information regarding data used for learning. The learning data information storage unit 122 stores data sets used for learning. 8 is a diagram illustrating an example of a learning data information storage unit according to the embodiment; FIG. For example, the learning data information storage unit 122 stores various information related to various data such as learning data used for learning and evaluation data used for accuracy evaluation (calculation). FIG. 8 shows an example of the learning data information storage unit 122 according to the embodiment. In the example of FIG. 8, the learning data information storage unit 122 includes items such as "data set ID", "data ID", "data", "label", and "date and time".

"Dataset ID" indicates identification information for identifying a data set. "Data ID" indicates identification information for identifying data. "Data" indicates data corresponding to the data identified by the data ID.

"Label" indicates a label (correct label) attached to corresponding data. For example, the "label" may be information (correct answer information) indicating the recognition result of the corresponding data (voice). For example, "label" is correct answer information indicating the result of converting voice data indicating user's utterance into character data (character string).

"Date and time" indicates the time (date and time) for the corresponding data. In the example of FIG. 8, "DA1" or the like is shown, but the "date and time" may be a specific date and time such as "15:52:14 on August 8, 2021". Information indicating from which model learning the data has started to be used may be stored, such as "use of version XX also started from model learning".

In the example of FIG. 8, the data set (data set DS1) identified by the data set ID "DS1" includes a plurality of data identified by the data IDs "DID1", "DID2", "DID3", etc. indicates that For example, each data (learning data) identified by the data IDs "DID1", "DID2", "DID3", etc. is speech information (speech data) used for model learning.

For example, data DT1 identified by data ID "DID1" is labeled data with label LB1, and indicates that use has started from model learning at date and time DA1. Also, for example, data DT4 identified by the data ID "DID4" is data collected as unlabeled data and attached with the label LB4, which is a prediction label, and is used starting from model learning at date and time DA4. indicates that the

Note that the learning data information storage unit 122 may store various types of information, not limited to the above, depending on the purpose. For example, the learning data information storage unit 122 may store data such as whether each data is learning data or evaluation data so as to be identifiable. For example, the learning data information storage unit 122 stores learning data and evaluation data in a distinguishable manner. The learning data information storage unit 122 may store information for identifying whether each data is learning data or evaluation data. The server device 100 learns a model based on each data used as learning data and the correct answer information. The server device 100 calculates the accuracy of the model based on each data used as the evaluation data and the correct answer information. The server device 100 calculates the accuracy of the model by collecting the result of comparing the output result output by the model when the evaluation data is input with the correct answer information.

(control unit 130)
Returning to FIG. 6, the description is continued. The control unit 130 is a controller. For example, various programs (corresponding to an example of an information processing program) stored in a storage device inside the server device 100 are stored in the RAM by a CPU, MPU, ASIC, FPGA, or the like. , etc., as a work area.

In the example shown in FIG. 6 , the control unit 130 has an acquisition unit 131 , a determination unit 132 , a learning unit 133 and a transmission unit 134 . Note that the internal configuration of the control unit 130 is not limited to the configuration shown in FIG. 3, and may be another configuration as long as it performs information processing described later.

(Acquisition unit 131)
Acquisition unit 131 receives various types of information from an external information processing device via communication unit 110 . The acquisition unit 131 receives various information from the terminal device 10 . Acquisition unit 131 stores information about the audio received from terminal device 10 in storage unit 120 . The acquisition unit 131 registers the information about the voice received from the terminal device 10 in the learning data information storage unit 122 as data (learning data) used for model learning.

Acquisition unit 131 acquires various types of information. Acquisition unit 131 acquires various types of information from storage unit 120 . The acquisition unit 131 acquires various types of information from the model information storage unit 121 and the learning data information storage unit 122 .

(Determination unit 132)
The determination unit 132 determines various types of information. For example, the determination unit 132 determines various information based on various information acquired from the external device by the acquisition unit 131 . For example, the determination unit 132 determines various information based on various information acquired from the terminal device 10 .

The determination unit 132 determines conditions regarding information requested to be provided to the terminal device 10 . For example, the determination unit 132 determines conditions regarding information to request the terminal device 10 to provide, based on the information stored in the storage unit 120 .

(Learning unit 133)
The learning unit 133 learns various types of information. The learning unit 133 learns various types of information based on information from an external information processing device and information stored in the storage unit 120 . The learning unit 133 learns various information based on the information stored in the learning data information storage unit 122 . The learning unit 133 stores the model generated by learning in the model information storage unit 121 . The learning unit 133 stores the model updated by learning in the model information storage unit 121 .

The learning unit 133 performs learning processing. The learning unit 133 performs various types of learning. The learning unit 133 learns various information based on the information acquired by the acquisition unit 131 . The learning unit 133 learns (generates) a model. The learning unit 133 learns various information such as models. The learning unit 133 generates a model through learning. The learning unit 133 learns the model using various machine learning techniques. For example, the learning unit 133 learns model (network) parameters. The learning unit 133 learns the model using various machine learning techniques.

Learning unit 133 generates model M1. The learning unit 133 learns network parameters. For example, the learning unit 133 learns network parameters of the model M1.

The learning unit 133 performs learning processing based on the learning data (teacher data) stored in the learning data information storage unit 122 . The learning unit 133 generates the model M1 by performing learning processing using the learning data stored in the learning data information storage unit 122 . For example, the learning unit 133 generates models used for speech recognition. The learning unit 133 generates the model M1 by learning parameters of the network of the model M1.

The method of learning by the learning unit 133 is not particularly limited. You can learn. Also, for example, a technique based on DNN (Deep Neural Network) such as CNN (Convolutional Neural Network) and 3D-CNN may be used. When the learning unit 133 targets time-series data such as voice, a recurrent neural network (RNN) or an extended RNN LSTM (Long Short-Term Memory units) using a method based on good too. Note that the server device 100 does not need to have the learning unit 133 when another device (learning device or the like) performs model learning.

(Sending unit 134)
The transmission unit 134 transmits various types of information to the terminal device 10 via the communication unit 110 . The transmission unit 134 transmits to the terminal device 10 a speech recognition model used by the terminal device 10 to perform speech recognition within itself. The transmitter 134 provides the terminal device 10 with the model M1.

The transmitting unit 134 transmits to the terminal device 10 information indicating a predetermined condition for designating a voice requesting the terminal device 10 to transmit information. The transmitting unit 134 transmits information indicating the conditions determined by the determining unit 132 to the terminal device 10 . The transmission unit 134 may transmit the collected data to another device (learning device, etc.).

[5. Processing procedure]
Next, a processing procedure performed by the terminal device 10 according to the embodiment will be described with reference to FIG. 9 . FIG. 9 is a flow chart showing a processing procedure according to the embodiment.

As shown in FIG. 9, the terminal device 10 associates the voice uttered by the user with the recognition result of the voice recognition of the utterance and collects them in the storage unit 120 within the device itself (step S101).

Then, when the collected voice satisfies a predetermined condition, the terminal device 10 transmits information about the voice to the server device 100 according to the user's permission (step S102).

[6. effect〕
As described above, the terminal device 10 according to the present application is a terminal device 10 that is used by a user and performs speech recognition by itself, and has the collection unit 153 and the transmission unit 157 . The collection unit 153 collects the speech uttered by the user and the recognition result of the speech recognition in the storage unit 14 in the own device in association with each other. When the voice collected by the collecting unit 153 satisfies a predetermined condition, the transmitting unit 157 transmits information about the voice to the server device 100 in accordance with the user's permission.

In this way, the terminal device 10 performs voice recognition of the voice uttered by the user within itself, and transmits information about the voice to the server device 100 when the conditions are satisfied and the user permits. As a result, even when the terminal device 10 performs voice recognition, the terminal device 10 can appropriately transmit information about voice collected by the terminal device 10 to other devices.

Also, in the terminal device 10 , the transmission unit 157 transmits voice data to the server device 100 . In this way, the terminal device 10 can appropriately transmit the information about the voice collected by the terminal device 10 to another device by transmitting voice data to the server device 100 .

Also, in the terminal device 10 , the transmission unit 157 transmits voice waveform data to the server device 100 . In this way, the terminal device 10 can appropriately transmit the information on the voice collected by the terminal device 10 to another device by transmitting the waveform data of the voice to the server device 100 .

In the terminal device 10 , the transmission unit 157 transmits data obtained by compressing the voice data to the server device 100 . In this way, the terminal device 10 transmits the data obtained by compressing the voice data to the server device 100, thereby suppressing an increase in the amount of communication and transmitting the information about the voice collected by the terminal device 10 to another device. can be sent properly.

Also, in the terminal device 10 , the transmission unit 157 transmits feature information extracted from the voice to the server device 100 . In this way, the terminal device 10 can appropriately transmit the information on the voice collected by the terminal device 10 to other devices by transmitting the data obtained by compressing the voice data to the server device 100 .

Further, in the terminal device 10, when the number of voices collected by the collecting unit 153 is equal to or greater than a predetermined number, the transmission unit 157 transmits information about voices to the server device 100 in response to permission. In this way, when the number of collected voices reaches or exceeds a predetermined number, the terminal device 10 transmits information about voices to the server device 100 so that the information about voices collected by the terminal device 10 can be transmitted to other devices. can be properly sent to the device.

Further, in the terminal device 10, the transmitting unit 157 transmits the information regarding the voice to the server device 100 in accordance with the permission when the score related to voice recognition of the voice collected by the collecting unit 153 satisfies a predetermined condition. In this way, the terminal device 10 transmits the information about the voice to the server device 100 when the score about the voice recognition of the collected voice satisfies a predetermined condition, thereby obtaining the information about the voice collected by the terminal device 10 . It can be properly transmitted to other devices.

Further, in the terminal device 10, when the sound collected by the collection unit 153 satisfies the condition regarding noise, the transmission unit 157 transmits information regarding sound to the server device 100 in response to permission. In this way, when the collected sound satisfies the noise-related condition, the terminal device 10 transmits the sound-related information to the server device 100, thereby appropriately transmitting the sound-related information collected by the terminal device 10 to other devices. can be sent to

Further, in the terminal device 10, when the utterance corresponding to the voice collected by the collection unit 153 includes a predetermined content, the transmission unit 157 transmits information about the voice to the server device 100 in accordance with permission. In this way, the terminal device 10 transmits information about the voice to the server device 100 when the utterance corresponding to the voice includes predetermined content, thereby transmitting the information about the voice collected by the terminal device 10 to other devices. can be properly sent to

Further, in the terminal device 10, when a predetermined condition specified by the server device 100 is satisfied, the transmission section 157 transmits information about voice to the server device 100 in response to permission. In this way, the terminal device 10 transmits the information about the voice to the server device 100 when the condition specified by the server device 100 is satisfied, thereby transmitting the information about the voice collected by the terminal device 10 to another device. can be sent properly.

The terminal device 10 also has a reception unit 155 . The accepting unit 155 accepts permission from the user. In this way, the terminal device 10 receives permission from the user, and by transmitting the information about the voice only when the user gives permission, the information about the voice collected by the terminal device 10 can be sent to other devices. can be sent properly.

Further, in the terminal device 10, the transmission unit 157 transmits information about voice to the server device 100 at a predetermined timing after the user's permission is received by the reception unit 155. FIG. In this way, the terminal device 10 transmits the information about the voice to the server device 100 at a predetermined timing after the permission of the user, so that the information about the voice collected by the terminal device 10 can be sent to another device at an appropriate timing. can be sent by

Also, in the terminal device 10, the transmission unit 157 transmits information about voice to the server device 100 while the communication environment satisfies a predetermined communication condition. In this way, the terminal device 10 transmits information about voice to the server device 100 while the communication environment satisfies a predetermined communication condition, thereby transmitting information about voice collected by the terminal device 10 to other devices. can be sent at the right time.

Also, in the terminal device 10, the transmission unit 157 transmits the information about the voice to the server device 100 at a timing when the usage rate of the terminal device by the user is low. In this way, the terminal device 10 transmits the information about the voice to the server device 100 at the timing when the usage rate of the terminal device 10 by the user is low. It can be sent at the right time.

The terminal device 10 also has a notification unit 154 . The notification unit 154 notifies the user of information about voices that are candidates for transmission to the server device 100 . In this way, the terminal device 10 notifies the user of the information about the voice that is a candidate for transmission to the server device 100, thereby allowing the user to recognize what kind of information is to be transmitted to the server device 100. .

Also, in the terminal device 10 , the transmission unit 157 transmits to the server device 100 information about the voices approved by the user among the candidates notified by the notification unit 154 . In this way, the terminal device 10 notifies the user of the candidates, and transmits the information about the voices that the user has approved among them to the server device 100, thereby collecting the information about the voices collected by the terminal device 10. It can be properly transmitted to other devices.

Also, in the terminal device 10 , the notification unit 154 notifies the information indicating the incentive provided to the user when the information about the voice is transmitted to the server device 100 . In this way, the terminal device 10 can motivate the user to provide information by informing the user of the incentives provided to the user in exchange for providing information, and the information can be transmitted from the terminal device 10 to other devices. can increase the amount of information presented.

As described above, the information processing system 1 according to the present application includes the terminal device 10 and the server device 100 which are used by users and perform speech recognition on their own. The server device 100 transmits to the terminal device 10 information indicating a predetermined condition for designating a voice requesting the terminal device 10 to transmit information. The terminal device 10 transmits the information about the voice that satisfies the predetermined condition received from the server device 100 to the server device 100 in accordance with the user's permission.

In this manner, the information processing system 1 allows the terminal device 10 used by the user to perform voice recognition of the voice of the user's utterance within the own device, and the information about the voice that satisfies the conditions specified by the server device 100 is transmitted to the terminal device 10 . sends to the server. As a result, even when the terminal device 10 performs voice recognition, the information processing system 1 can appropriately transmit the information about the voice collected by the terminal device 10 to another device.

[7. Hardware configuration]
Also, the terminal device 10 and the server device 100 according to the above-described embodiments are implemented by a computer 1000 configured as shown in FIG. 10, for example. The server apparatus 100 will be described below as an example. FIG. 10 is a diagram illustrating an example of a hardware configuration; The computer 1000 is connected to an output device 1010 and an input device 1020, and an arithmetic device 1030, a primary storage device 1040, a secondary storage device 1050, an output I/F (Interface) 1060, an input I/F 1070, and a network I/F 1080 are buses. It has a form connected by 1090.

The arithmetic device 1030 operates based on programs stored in the primary storage device 1040 and the secondary storage device 1050, programs read from the input device 1020, and the like, and executes various processes. The arithmetic unit 1030 is implemented by, for example, a CPU (Central Processing Unit), MPU (Micro Processing Unit), ASIC (Application Specific Integrated Circuit), FPGA (Field Programmable Gate Array), or the like.

The primary storage device 1040 is a memory device such as a RAM (Random Access Memory) that temporarily stores data used for various calculations by the arithmetic device 1030 . The secondary storage device 1050 is a storage device in which data used for various calculations by the arithmetic device 1030 and various databases are registered. State Drive), flash memory, or the like. The secondary storage device 1050 may be an internal storage or an external storage. Also, the secondary storage device 1050 may be a removable storage medium such as a USB memory or an SD (Secure Digital) memory card. Also, the secondary storage device 1050 may be a cloud storage (online storage), a NAS (Network Attached Storage), a file server, or the like.

The output I/F 1060 is an interface for transmitting information to be output to the output device 1010 that outputs various information such as a display, a projector, and a printer. (Digital Visual Interface), HDMI (registered trademark) (High Definition Multimedia Interface), and other standardized connectors. Also, the input I/F 1070 is an interface for receiving information from various input devices 1020 such as a mouse, keyboard, keypad, buttons, scanner, etc., and is realized by, for example, USB.

Also, the output I/F 1060 and the input I/F 1070 may be wirelessly connected to the output device 1010 and the input device 1020, respectively. That is, the output device 1010 and the input device 1020 may be wireless devices.

Also, the output device 1010 and the input device 1020 may be integrated like a touch panel. In this case, the output I/F 1060 and the input I/F 1070 may also be integrated as an input/output I/F.

Note that the input device 1020 includes, for example, optical recording media such as CDs (Compact Discs), DVDs (Digital Versatile Discs), PDs (Phase change rewritable discs), magneto-optical recording media such as MOs (Magneto-Optical discs), and tapes. It may be a device that reads information from a medium, a magnetic recording medium, a semiconductor memory, or the like.

Network I/F 1080 receives data from other devices via network N and sends the data to arithmetic device 1030, and also transmits data generated by arithmetic device 1030 via network N to other devices.

Arithmetic device 1030 controls output device 1010 and input device 1020 via output I/F 1060 and input I/F 1070 . For example, arithmetic device 1030 loads a program from input device 1020 or secondary storage device 1050 onto primary storage device 1040 and executes the loaded program.

For example, when the computer 1000 functions as the server device 100 , the arithmetic device 1030 of the computer 1000 implements the functions of the control unit 130 by executing a program loaded on the primary storage device 1040 . Further, arithmetic device 1030 of computer 1000 may load a program acquired from another device via network I/F 1080 onto primary storage device 1040 and execute the loaded program. Further, the arithmetic unit 1030 of the computer 1000 may cooperate with another device via the network I/F 1080, and call functions, data, etc. of the program from another program of the other device for use.

[8. others〕
Although the embodiments of the present application have been described above, the present invention is not limited by the contents of these embodiments. In addition, the components described above include those that can be easily assumed by those skilled in the art, those that are substantially the same, and those within the so-called equivalent range. Furthermore, the components described above can be combined as appropriate. Furthermore, various omissions, replacements, or modifications of components can be made without departing from the gist of the above-described embodiments.

Further, among the processes described in the above embodiments, all or part of the processes described as being automatically performed can be manually performed, or the processes described as being performed manually can be performed manually. All or part of this can also be done automatically by known methods. In addition, information including processing procedures, specific names, various data and parameters shown in the above documents and drawings can be arbitrarily changed unless otherwise specified. For example, the various information shown in each drawing is not limited to the illustrated information.

Also, each component of each device illustrated is functionally conceptual, and does not necessarily need to be physically configured as illustrated. In other words, the specific form of distribution and integration of each device is not limited to the illustrated one, and all or part of them can be functionally or physically distributed and integrated in arbitrary units according to various loads and usage conditions. Can be integrated and configured.

For example, the server device 100 described above may be implemented by a plurality of server computers, and depending on the function, an external platform may be called using an API (Application Programming Interface), network computing, or the like. Flexible to change.

Also, the above-described embodiments and modifications can be appropriately combined within a range that does not contradict the processing contents.

Also, the above-mentioned "section, module, unit" can be read as "means" or "circuit". For example, the acquisition unit can be read as acquisition means or an acquisition circuit.

1 information processing system 10 terminal device 11 communication unit 12 input unit 13 output unit 14 storage unit 141 model information storage unit 142 collected information storage unit 15 control unit 151 reception unit 152 speech recognition unit 153 collection unit 154 notification unit 155 reception unit 156 determination Unit 157 Transmission Unit 16 Sensor Unit 161 Voice Sensor 100 Server Device 110 Communication Unit 120 Storage Unit 121 Model Information Storage Unit 122 Learning Data Information Storage Unit 130 Control Unit 131 Acquisition Unit 132 Determination Unit 133 Learning Unit 134 Transmission Unit

Claims (20)

A terminal device that is used by a user and performs speech recognition on its own device,
a collection unit that associates the voice uttered by the user with the recognition result of the utterance obtained by the voice recognition, and collects them in a storage unit within the device;
a transmitting unit configured to transmit information about the voice to a server device in response to permission from the user when the voice collected by the collecting unit satisfies a predetermined condition;
A terminal device comprising: The transmission unit
2. The terminal device according to claim 1, wherein said voice data is transmitted to said server device. The transmission unit
3. The terminal device according to claim 1, wherein the waveform data of the voice is transmitted to the server device. The transmission unit
The terminal device according to any one of claims 1 to 3, wherein data obtained by compressing the voice data is transmitted to the server device. The transmission unit
The terminal device according to any one of claims 1 to 4, wherein the feature information extracted from the voice is transmitted to the server device. The transmission unit
6. The apparatus according to any one of claims 1 to 5, wherein, when the number of said voices collected by said collecting unit is equal to or greater than a predetermined number, information about said voices is transmitted to said server device according to said permission. The terminal device described in . The transmission unit
7. The apparatus according to any one of claims 1 to 6, characterized in that, when the score relating to the speech recognition of the speech collected by the collecting unit satisfies a predetermined condition, the information relating to the speech is transmitted to the server device according to the permission. The terminal device according to any one of items 1 and 2. The transmission unit
8. The apparatus according to any one of claims 1 to 7, wherein when the voice collected by the collecting unit satisfies a condition regarding noise, the information regarding the voice is transmitted to the server device according to the permission. terminal equipment. The transmission unit
9. The apparatus according to any one of claims 1 to 8, characterized in that, when the utterance corresponding to the voice collected by the collecting unit includes predetermined content, the information about the voice is transmitted to the server device according to the permission. The terminal device according to item 1. The transmission unit
10. The terminal according to any one of claims 1 to 9, wherein, when the predetermined condition specified by the server device is satisfied, the information about the voice is transmitted to the server device according to the permission. Device. a reception unit that receives the permission from the user;
The terminal device according to any one of claims 1 to 10, characterized by comprising: The transmission unit
12. The terminal device according to claim 11, wherein the information about the voice is transmitted to the server device at a predetermined timing after the permission of the user is received by the receiving unit. The transmission unit
13. The terminal device according to claim 12, wherein the information about the voice is transmitted to the server device while the communication environment satisfies a predetermined communication condition. The transmission unit
14. The terminal device according to claim 12 or 13, wherein the information about the voice is transmitted to the server device at a timing when the usage rate of the terminal device by the user is low. a notification unit that notifies the user of information about the voice that is a candidate for transmission to the server device;
The terminal device according to any one of claims 1 to 14, further comprising: The transmission unit
16. The terminal device according to claim 15, wherein, among the candidates notified by the notification unit, information about the voice approved by the user is transmitted to the server device. The notification unit
17. The terminal device according to claim 15 or 16, wherein information indicating an incentive provided to the user is notified when the information about the voice is transmitted to the server device. A transmission method executed by a terminal device that is used by a user and performs speech recognition on its own device,
a collection step of associating the voice uttered by the user with the recognition result of the utterance obtained by the voice recognition, and collecting the results in a storage unit within the own device;
a transmitting step of transmitting information about the voice to a server device in response to permission from the user when the voice collected in the collecting step satisfies a predetermined condition;
A transmission method comprising: A transmission program that is used by a user and executed by a terminal device that performs speech recognition on its own device,
a collection procedure of associating the voice uttered by the user with the recognition result of the utterance obtained by the voice recognition, and collecting the result in a storage unit within the own device;
a transmitting step of transmitting information about the voice to a server device in accordance with the user's permission when the voice collected by the collecting step satisfies a predetermined condition;
A transmission program characterized by causing a terminal device to execute A terminal device that is used by a user and performs speech recognition on its own device;
a server device;
has
The server device
transmitting to the terminal device information indicating a predetermined condition specifying a voice requesting information transmission from the terminal device;
The terminal device
An information processing system, wherein the information about the voice that has been received from the server device and that satisfies the predetermined condition is transmitted to the server device according to the permission of the user.

Images