JP7430672B2 - Terminal device, transmission method, transmission program and information processing system - Google Patents

Description

The present invention relates to a terminal device, a transmission method, a transmission program, and an information processing system.

Speech recognition, which recognizes voices uttered by users, is used in various services. For example, there is a technology that performs speech recognition using a model created by learning using utterance information and utterance content indicated by the utterance information (for example, see Patent Document 1).

JP 2021-081527 Publication

However, with the above-mentioned conventional technology, it may be difficult to obtain information regarding the voice uttered by the user. For example, in the above-mentioned conventional technology, the server device performs voice recognition on the user's utterances (speech), and the recognition result is transmitted to the terminal device used by the user. (audio) data can be obtained. On the other hand, when so-called on-device type voice recognition is performed, in which voice recognition is performed on the terminal device itself, voice recognition is completed without transmitting the user's utterance (speech) data from the terminal device to the server device. , it is difficult for a device other than the terminal device used by the user, such as a server device, to collect information regarding the voice uttered by the user.

This application has been made in view of the above, and aims to appropriately transmit information related to voices collected by a terminal device to other devices even when the terminal device performs voice recognition. .

The terminal device according to the present application is a terminal device that is used by a user and performs voice recognition on its own, and that associates the voice uttered by the user with the recognition result of the voice recognition of the utterance and performs voice recognition within the own device. a collection unit that collects the audio into a storage unit; and a transmission unit that transmits information regarding the audio to a server device in accordance with the user's consent if the audio collected by the collection unit satisfies a predetermined condition. It is characterized by

According to one aspect of the embodiment, even when a terminal device performs speech recognition, information regarding speech collected by the terminal device can be appropriately transmitted to another device.

FIG. 1 is an explanatory diagram showing an overview of information processing according to the embodiment. FIG. 2 is a diagram illustrating a configuration example of an information processing system according to an embodiment. FIG. 3 is a diagram illustrating a configuration example of a terminal device according to an embodiment. FIG. 4 is a diagram illustrating an example of a model information storage section. FIG. 5 is a diagram illustrating an example of a collected information storage section. FIG. 6 is a diagram illustrating a configuration example of a server device according to an embodiment. FIG. 7 is a diagram illustrating an example of a model information storage section. FIG. 8 is a diagram illustrating an example of a learning data information storage section. FIG. 9 is a flowchart showing the processing procedure according to the embodiment. FIG. 10 is a diagram showing an example of a hardware configuration.

DESCRIPTION OF THE PREFERRED EMBODIMENTS Below, a terminal device, a transmission method, a transmission program, and a form for implementing an information processing system (hereinafter referred to as an "embodiment") according to the present application will be described in detail with reference to the drawings. Note that this embodiment does not limit the terminal device, transmission method, transmission program, and information processing system according to the present application. In addition, in the following embodiments, the same parts are given the same reference numerals, and redundant explanations will be omitted.

(Embodiment)
[1. Information processing〕
First, with reference to FIG. 1, an overview of information processing performed by the information processing system 1 according to the embodiment will be described. FIG. 1 is an explanatory diagram showing an overview of information processing according to the embodiment. In FIG. 1, the server device 100 transmits to the terminal device 10 information on conditions specifying a voice requesting information transmission, and the terminal device 10 transmits information regarding the voice that satisfies the conditions received from the server device 100. The case of transmitting to the server device 100 will be explained as an example. Note that if the server device 100 can acquire information regarding audio from the terminal device 10, the terminal device 10 will transmit information regarding audio to the server device 100 based on preset conditions without the server device 100 specifying the conditions. may be sent.

Further, below, the terminal device 10 may be referred to as a user. That is, in the following, the user can also be read as the terminal device 10. In addition, in FIG. 1, the case where the terminal device 10 is a smartphone will be explained as an example, but the terminal device 10 is not limited to a smartphone and can be any arbitrary device as long as it can collect the voice uttered by the user and send it to the server device 100. However, details regarding this point will be described later. Further, in FIG. 1, a case will be described as an example in which the terminal device 10 transmits audio data to the server device 100 as information regarding audio. However, the information regarding audio that the terminal device 10 transmits to the server device 100 is However, the details regarding this point will be described later.

An example of information processing will be described below with reference to FIG. Note that, for the sake of explanation, FIG. 1 shows a case in which transmission is performed with one utterance as the target, but the terminal device 10 detects when the number of collected utterances (voices) exceeds a certain threshold (for example, 50, 100, etc.). In some cases, transmission may be performed with the user's consent for transmission.

FIG. 1 shows a case where the user is identified by the user ID "U1" (hereinafter sometimes referred to as "user U1"). The terminal device 10 used by the user U1 performs voice recognition within itself using the model M1, which is a voice recognition model, and provides a service to the user U1 according to the result of the voice recognition. In FIG. 1, for the sake of explanation, an example is taken in which the model M1 converts the audio information of the user's utterance into text. However, if the model M1 performs processing related to speech recognition, ) may be a speech recognition model that performs various functions, such as the ability to identify

First, the server device 100 transmits to the terminal device 10 information indicating a condition CN1 specifying the voice that the terminal device 10 is requested to provide (step S11). The terminal device 10 stores the received information indicating the condition CN1 in the storage unit 120 (see FIG. 3). For example, the terminal device 10 compares the condition CN1 stored in the storage unit 120 with the information regarding the voice, determines whether the voice satisfies the condition CN1, and transmits the voice data that satisfies the condition CN1 to the server device 100. do. Note that, in order to simplify the explanation, the following description assumes that the condition CN1 includes a specific word (hereinafter referred to as "new word NX"), and examples of other conditions will be described later.

First, user U1 utters "XXXX". Note that "XXXX" is an utterance that includes specific content. The terminal device 10 detects the utterance PA of the user U1, and receives the audio data of "XXXX", which is the utterance PA of the user U1, as input (step S12).

Then, the terminal device 10 performs voice recognition processing using the voice data of "XXXX" received as input and the model M1 (step S13). The terminal device 10 inputs the voice data of "XXXX" into the model M1, and causes the model M1 to output character data, thereby performing a process of converting voice into characters (speech recognition process). In FIG. 1, model M1 to which voice data of "XXXX" is input outputs character data of "XXXX". Note that the model M1 may output a score indicating the accuracy of speech recognition together with the characters corresponding to the input speech. Further, it is assumed that the character data of "XXXX" includes a new word NX.

Then, the terminal device 10 associates the voice uttered by the user with the recognition result of the voice recognition of the utterance and collects them in the storage unit DB (step S14). In FIG. 1, the terminal device 10 associates and collects the utterance PA uttered by the user U1 and the recognition result of the utterance PA in the storage unit 120. For example, the terminal device 10 associates the voice data of “XXXX”, which is the utterance PA, with the character data of “XXXX”, which is the recognition result of the voice data, and associates it with the collected information storage unit 142 (see FIG. reference).

Then, the terminal device 10 determines whether or not the collected voice satisfies the condition, and if it determines that the condition is satisfied, it notifies the user (step S15). In FIG. 1, the terminal device 10 determines whether the collected utterance PA satisfies the condition CN1. For example, the terminal device 10 compares the character data of the utterance PA with the condition CN1, and determines whether the utterance PA satisfies the condition CN1. In this manner, the terminal device 10 uses the condition CN1, which is a condition related to content (also referred to as "content condition"), to determine whether or not the utterance PA includes predetermined content. Since the collected character data of the utterance PA includes the new word NX, the terminal device 10 determines that the condition CN1 is satisfied.

Therefore, the terminal device 10 notifies the user U1 of the utterance PA as information that is a candidate to be transmitted to the server device 100. For example, the terminal device 10 displays the utterance PA as a candidate to be transmitted to the server device 100. For example, the terminal device 10 displays the character string "XXXX" and an explanation indicating that it is a candidate on the screen. In this case, the terminal device 10 may display information indicating that the utterance PA is a candidate, as well as information for accepting information indicating whether permission is granted by the user U1. For example, the terminal device 10 displays information indicating that the utterance PA is a candidate, as well as a button for the user to specify whether to transmit the utterance PA. For example, the terminal device 10 displays information indicating that the utterance PA is a candidate, as well as an accept button that says "permit transmission" or the like and a reject button that says "do not authorize transmission" or the like. Good too. In this way, the terminal device 10 confirms permission from the user when the conditions are met.

In this case, the terminal device 10 accepts permission from the user U1 to transmit the utterance PA to the server device 100 using the permission button. For example, if the user U1 selects the consent button, the terminal device 10 determines that the user U1 has given permission to transmit the utterance PA to the server device 100. Furthermore, when the user U1 selects the reject button, the terminal device 10 determines that the user U1 does not approve the transmission of the utterance PA to the server device 100. Note that the above is just an example, and the terminal device 10 is not limited to accepting notifications and permissions through display, but may accept notifications and permissions in various modes (modals). For example, the terminal device 10 may notify (output) the user U1 that the user U1 is a candidate for transmitting the utterance PA to the server device 100 by voice. Further, the terminal device 10 may accept whether or not the user U1 gives permission to transmit the utterance PA to the server device 100 by voice.

The terminal device 10 receives permission from the user U1 to transmit the utterance PA to the server device 100. (Step S16). For example, the terminal device 10 receives permission from the user U1 to transmit the utterance PA to the server device 100 when the user U1 selects a permission button.

Then, the terminal device 10 transmits to the server device 100 information regarding the utterance PA that the user U1 has approved to transmit (step S17). In FIG. 1, the terminal device 10 transmits the audio data of the utterance PA and the recognition result thereof to the server device 100. That is, the terminal device 10 transmits to the server device 100 the voice data of “XXXX”, which is the utterance PA, and the character data of “XXXX”, which is the recognition result of the voice data. Note that the above is only an example, and the terminal device 10 may transmit only the audio data of the utterance PA to the server device 100.

The server device 100 adds the information regarding the voice received from the terminal device 10 to the data used for learning (step S18). In FIG. 1, a combination of voice data of "XXXX" which is the utterance PA and character data of "XXXX" which is the recognition result of the voice data (hereinafter also referred to as "new data PDT") is received from the terminal device 10. The server device 100 adds the received new data PDT to the data set DS1, which is a learning data set. For example, the server device 100 stores new data PDT in which the voice data of the utterance PA "XXXX" is associated with the character data of "XXXX" as a label in the learning data information storage unit 122 ( (see Figure 8).

Then, the server device 100 learns the model M1 using the data set DS1 to which the new data PDT has been added (step S19). The server device 100 uses the data set DS1 to learn (update) parameters such as weights of the model M1. Any method can be adopted for the learning process of the model M1.

For example, the server device 100 uses a technique such as backpropagation (error backpropagation method) to learn so that the character data output by the model M1 approaches the correct data (label) corresponding to the voice data input to the model M1. Perform processing. For example, the server device 100 adjusts the value of a weight (i.e., connection coefficient) that is taken into consideration when transmitting a value between nodes through a learning process. In this way, the server device 100 learns the model M1 through processing such as backpropagation that corrects parameters (connection coefficients) so that the error between the output of the model M1 and the correct data corresponding to the input is reduced. For example, the server device 100 generates the model M1 by performing processing such as backpropagation to minimize a predetermined loss function. Thereby, the server device 100 can perform a learning process to learn the parameters of the model M1.

As described above, even when the terminal device 10 performs speech recognition, the information processing system 1 can appropriately transmit information related to speech collected by the terminal device 10 to the server device 100. Therefore, the information processing system 1 can collect data necessary for learning a model even when the terminal device 10 performs speech recognition.

For example, when a user uses an application (also simply referred to as an "app") equipped with on-device voice recognition on the terminal device 10, the information processing system 1 stores the voice uttered and the recognition result in the terminal device 10 ( accumulate. For example, when the information processing system 1 uses an application on the terminal device 10 in which an arbitrary application such as a car navigation application or a shopping application is installed, the information processing system 1 stores (accumulates) the voice uttered and the recognition result in the terminal device 10. do. Then, when the accumulated (collected) utterances meet a predetermined standard, the information processing system 1 asks the user for permission to send the utterances to the server device 100, and if the user's permission is obtained, permission is obtained. The information is sent from the terminal device 10 to the server device 100. Through such processing, the information processing system 1 can appropriately transmit information related to speech collected by the terminal device 10 to the server device 100 even when the terminal device 10 performs speech recognition.

[1-1. Other examples]
Note that the process illustrated in FIG. 1 is only an example, and the information processing system 1 may transmit various information from the terminal device 10 to the server device 100 using various conditions. Regarding this point, examples of each element will be described below.

[1-1-1. Information to send〕
In FIG. 1, the case where audio data is transmitted from the terminal device 10 to the server device 100 has been described as an example, but the information transmitted from the terminal device 10 to the server device 100 can be any information related to audio. It may be.

The terminal device 10 may transmit audio data to the server device 100. The terminal device 10 may transmit audio waveform data to the server device 100. The terminal device 10 may transmit compressed audio data to the server device 100. The terminal device 10 may transmit the feature information extracted from the voice to the server device 100.

As described above, the information transmitted from the terminal device 10 to the server device 100 may be various information such as an audio waveform or a compressed version thereof. The information transmitted from the terminal device 10 to the server device 100 may be a feature extracted from the audio waveform. The feature amount here is, for example, data smaller in size than the original audio data, and may be spectrum information that does not include information regarding individuality as much as possible. Further, the information transmitted from the terminal device 10 to the server device 100 may be data of compressed audio, such as audio in which the utterance content is irreversibly compressed to a level that can be confirmed by listening test, thereby reducing the size as much as possible.

[1-1-2. Information conditions〕
In FIG. 1, the condition CN1, which is whether the audio includes a specific word, was explained as an example, but the condition for information to be transmitted to the server device 100 (transmission information condition) is whether the audio includes a specific word. The conditions are not limited to these, and various conditions may be used.

The transmission information condition may be a condition regarding the number of collected utterances (voices) (also referred to as "number condition"). For example, the transmission information condition may be that the number of collected utterances (voices) exceeds a certain threshold. In this case, the terminal device 10 determines whether the number of collected voices is equal to or greater than a predetermined number (for example, 50, 100, etc.). For example, when the number of collected voices exceeds a predetermined number, the terminal device 10 determines that the condition is satisfied, and transmits information regarding the voices to the server device 100 in accordance with the user's permission.

For example, the transmission information condition may be that a score related to speech recognition of collected utterances (voices) satisfies a predetermined condition (also referred to as "score condition"). In this case, the terminal device 10 determines whether the voice recognition score of the collected voice is equal to or higher than a predetermined threshold (for example, 0.5, 0.7, etc.). For example, if the score related to voice recognition of the collected voice is equal to or higher than a predetermined threshold, the terminal device 10 determines that the condition is satisfied, and transmits information regarding the voice to the server device 100 in accordance with the user's permission.

Further, for example, the terminal device 10 determines whether the score related to voice recognition of the collected voice is less than a predetermined threshold (eg, 0.5, 0.7, etc.). For example, if the score related to voice recognition of the collected voice is less than a predetermined threshold, the terminal device 10 determines that the condition is satisfied, and transmits information regarding the voice to the server device 100 in accordance with the user's permission.

Furthermore, for example, the transmission information condition may be that the collected audio satisfies a condition regarding noise (also referred to as a "noise condition"). In this case, the terminal device 10 determines whether the signal-to-noise ratio (SN ratio) of the collected audio is greater than or equal to a predetermined value. For example, if the SN ratio of the collected audio is greater than or equal to a predetermined value, the terminal device 10 determines that the condition is satisfied, and transmits information regarding the audio to the server device 100 in accordance with the user's consent.

Further, for example, the terminal device 10 determines whether the signal-to-noise ratio (SN ratio) of the collected audio is less than a predetermined value. For example, if the SN ratio of the collected audio is less than a predetermined value, the terminal device 10 determines that the condition is satisfied, and transmits information regarding the audio to the server device 100 in accordance with the user's permission.

Note that the above is only an example, and the information processing system 1 may use various conditions as appropriate. The transmission information condition may be the type of terminal device 10. For example, when the user's speech (voice) in the car navigation system is insufficient, the information processing system 1 may set the condition (terminal condition) that the type of the terminal device 10 is "car navigation system". The transmission information condition may be a user attribute. For example, when the child's utterances (voices) are insufficient, the information processing system 1 sets the conditions (speaker condition). Note that the above is just an example, and when collecting children's utterances (voices), the information processing system 1 uses a discriminator for "adult/child" and "age group" to identify children's utterances (voices). May be collected. In this case, the information processing system 1 may collect the child's utterances (voices) using a classifier that uses not only the fundamental frequency but also spectral information as a feature quantity.

Further, for example, the information processing system 1 may use a combination of the above-mentioned conditions. For example, if the number of voices satisfying at least one of the content condition, score condition, or noise condition satisfies the number condition, the terminal device 10 transmits information regarding the voice to the server device 100 according to the user's permission. Good too. For example, the terminal device 10 collects audio that satisfies at least one of the content condition, the score condition, or the noise condition as the relevant audio. Then, when the number of corresponding voices reaches a predetermined number (for example, 30, 150, etc.), the terminal device 10 may transmit information regarding the voices to the server device 100 according to the user's permission.

In this way, the terminal device 10 selects utterances that are judged to contribute to improving the speech recognition engine from among the collected voices (user utterances), and obtains permission to transmit them if the number of utterances exceeds a certain number. good. For example, a score given to each utterance by the speech recognition engine may be used as a criterion for determining that it contributes to improvement of the speech recognition engine. The signal-to-noise ratio (SNR) for each utterance may be used as a criterion for determining that it contributes to improving the speech recognition engine. For example, the recognition result text for each utterance may be used as a criterion for determining that it contributes to improving the speech recognition engine. For example, the information processing system 1 may select an utterance that includes a desired word to improve the speech recognition engine.

[1-1-3. Transmission timing]
In FIG. 1, an example is described in which the information is transmitted at the timing when the user's permission is obtained, but any timing can be adopted as the timing for transmitting the information to the server device 100.

After receiving the user's permission for the audio, the terminal device 10 transmits information regarding the audio to the server device 100 at a predetermined timing. After receiving permission from the user for the audio, the terminal device 10 transmits information regarding the audio to the server device 100 while the communication environment satisfies predetermined communication conditions. For example, after receiving permission from the user for audio, the terminal device 10 transmits information regarding the audio to the server device 100 while communicating via Wi-Fi (registered trademark) (Wireless Fidelity).

Further, after receiving the user's permission for the audio, the terminal device 10 transmits information regarding the audio to the server device 100 at a timing when the usage rate of the terminal device 10 is low. For example, after receiving the user's permission for the audio, the terminal device 10 transmits information regarding the audio to the server device 100 at a timing when the usage rate of the processor of the terminal device 10 becomes less than a predetermined threshold. For example, after receiving the user's permission for the audio, the terminal device 10 transmits information regarding the audio to the server device 100 while the terminal device 10 is being charged.

As mentioned above, after obtaining the user's transmission permission, the terminal device 10 is connected to Wi-Fi, or when the usage rate of the user's terminal device 10 is low (when the CPU load is below a predetermined value or when the user is at home late at night). (such as during charging), the information is transmitted to the server device 100.

Further, the terminal device 10 may perform preferential transmission of information regarding voices having high value, depending on the value of each voice. The terminal device 10 prioritizes (ranks) the information according to the voice recognition score, and transmits information to the server device 100 in order of priority. For example, the terminal device 10 assigns a higher priority (rank) in descending order of voice recognition score, and transmits information regarding voices with higher priority to the server device 100 in order.

[1-1-4. User selection]
As described above, the terminal device 10 transmits to the server device 100 only information related to voices approved by the user among the candidates notified to the user. For example, when obtaining transmission permission from the user, the terminal device 10 presents the user with a list of utterances to be transmitted (recognition result text and links to the utterances, etc.), and if the user selects an utterance that the user does not want to transmit, Exclude the utterance selected by the user from being sent.

[1-1-5. Incentive]
The information processing system 1 may provide an incentive to the user in order to obtain transmission permission from the user. For example, when the user approves the transmission of audio, the information processing system 1 may provide the user with various types of incentives, such as electronic money, points, and coupons, depending on the audio that the user has approved.

For example, the terminal device 10 may notify the user of information indicating an incentive to be provided to the user when information regarding audio is transmitted to the server device 100. For example, the terminal device 10 may notify the user that the higher the value of the voice, the better the incentive will be provided to the user. For example, the terminal device 10 may notify the user that the higher the audio score, the more points will be provided to the user.

For example, the terminal device 10 may notify the user that an incentive will be provided to the user when the user attaches a label (correct answer) to the audio. For example, the terminal device 10 may notify the user that an incentive will be provided to the user if the user checks the recognition result of a voice whose score is less than a predetermined threshold and corrects it if it is incorrect. .

[2. Configuration example of information processing system]
Next, the configuration of the information processing system 1 including the server device 100 according to the embodiment will be described using FIG. 2. FIG. 2 is a diagram showing a configuration example of the information processing system 1 according to the embodiment. As shown in FIG. 2, the information processing system 1 according to the embodiment includes a plurality of terminal devices 10 and a server device 100. These various devices are connected via a network N so that they can communicate by wire or wirelessly. The network N is, for example, a LAN (Local Area Network) or a WAN (Wide Area Network) such as the Internet.

Furthermore, the number of devices included in the information processing system 1 shown in FIG. 2 is not limited to what is illustrated. For example, in FIG. 2, only three terminal devices 10-1, 10-2, and 10-3 are shown to simplify the illustration, but this is just an example and is not limited to four terminal devices. It may be more than one.

The terminal device 10 is an information processing device (computer) that is used by a user and provides a service based on voice recognition to the user through voice recognition within the device itself. The terminal device 10 is connected to a wireless communication network such as LTE (Long Term Evolution), 4G (4th Generation), or 5G (5th Generation), Bluetooth (registered trademark), or wireless LAN (Local Area Network). It is possible to connect to the network N and communicate with the server device 100 via short-range wireless communication such as the like.

In FIG. 2, the terminal device 10-1 is a smartphone used by a user. Note that the terminal device 10-1 may be a smart device such as a tablet terminal, a feature phone, a PC (Personal Computer), a PDA (Personal Digital Assistant), a car navigation system, a smart watch, or a headphone, as long as it is a device used by the user. It may also be a wearable device such as a mounted display, smart glasses, or the like.

In FIG. 2, the terminal device 10-2 is a smart speaker used by a user. Note that the terminal device 10-2 may be any IoT (Internet of Things) device such as a television or a refrigerator as long as it is used by a user.

In FIG. 2, the terminal device 10-3 is a mobile object such as a vehicle such as an automobile equipped with a car navigation function. Note that the terminal device 10-3 may be a car navigation device that is placed in a moving object and provides a car navigation function.

The server device 100 is an information processing device (computer) that acquires information regarding audio from the terminal device 10. Further, the server device 100 is a learning device that learns a model used for speech recognition through machine learning using information regarding acquired speech. Additionally, the server device 100 transmits the model to the terminal device 10. Furthermore, the server device 100 transmits to the terminal device 10 information indicating conditions for specifying data that the terminal device 10 wants to transmit. Note that the server device 100 may have only the function of acquiring information regarding audio from the terminal device 10 and accumulating the acquired information. In this case, the information processing system 1 may include a device (learning device) that performs model learning using information accumulated by the server device 100. That is, in the information processing system 1, a device (e.g., server device 100) that receives and stores voice information such as voice data from the terminal device 10, and a voice recognition model is stored and learned or updated using the collected data. It may be separate from the device (for example, a learning device) that transmits (distributes) to the terminal device 10. In this case, the learning device may hold the speech recognition model, perform learning and updating of the model using data acquired from the server device 100, and transmit the model to the terminal device 10 used by the user. For example, in the information processing system 1, the information on speech such as speech data collected by the server device 100 is separately manually listened to and given a correct answer text, or mechanically selected to create a new speech recognition model. May be learned or updated. Further, in the information processing system 1, the model may be distributed as an upgrade to the terminal device 10 used by the user via an application download service such as an app store, attached to the binary of the application. . Note that the configuration of the information processing system 1 is not limited to the above, and any configuration can be adopted as long as the server device 100 acquires information regarding audio from the terminal device 10.

[3. Configuration example of terminal device]
Next, the configuration of the terminal device 10 will be explained using FIG. 3. FIG. 3 is a diagram showing a configuration example of the terminal device 10. As shown in FIG. As shown in FIG. 3, the terminal device 10 includes a communication section 11, an input section 12, an output section 13, a storage section 14, a control section 15, and a sensor section 16. Note that the terminal device 10 may have any device configuration as long as it can collect data and provide it to the server device 100. For example, the terminal device 10 may have any other configuration as long as it has a communication unit 11 that communicates with the server device 100 and a control unit 15 that performs a process of collecting data. Depending on the type of the terminal device 10, for example, the terminal device 10 may not include the input section 12, the output section 13, the storage section 14, or the sensor section 16.

Note that the terminal device 10 may have any configuration other than the above, depending on the implementation mode. For example, when the terminal device 10 is a moving object, the terminal device 10 may have a structure that includes a mechanism for realizing movement, such as a drive unit (motor).

(Communication Department 11)
The communication unit 11 is realized by, for example, a NIC, a communication circuit, or the like. The communication unit 11 is connected to a network N (such as the Internet) by wire or wirelessly, and transmits and receives information to and from other devices such as the server device 100 via the network N.

(Input section 12)
The input unit 12 receives various inputs. The input unit 12 receives user operations. For example, the input unit 12 receives voice input from the user via a voice sensor 161 such as a microphone. The input unit 12 accepts various operations based on user's utterances.

Further, the input unit 12 may accept an operation (user operation) on the terminal device 10 other than the user's utterance (voice) as an operation input by the user. The input unit 12 may receive information regarding a user's operation using a remote controller (remote controller) via the communication unit 11 . Furthermore, the input unit 12 may include buttons provided on the terminal device 10 or a keyboard or mouse connected to the terminal device 10.

For example, the input unit 12 may include a touch panel that can realize functions equivalent to a remote control, a keyboard, and a mouse. In this case, various information is input to the input unit 12 via the display (output unit 13). The input unit 12 receives various operations from the user via the display screen using touch panel functions realized by various sensors. That is, the input unit 12 receives various operations from the user via the display (output unit 13) of the terminal device 10. For example, the input unit 12 receives user operations via the display (output unit 13) of the terminal device 10.

(Output section 13)
The output unit 13 outputs various information. The output unit 13 has a function of displaying information. The output unit 13 is provided in the terminal device 10 and displays various information. The output unit 13 is realized by, for example, a liquid crystal display, an organic EL (Electro-Luminescence) display, or the like. The output unit 13 may have a function of outputting audio. For example, the output unit 13 includes a speaker that outputs audio.

(Storage unit 14)
The storage unit 14 is realized by, for example, a semiconductor memory device such as a RAM or a flash memory, or a storage device such as a hard disk or an optical disk. The storage unit 14 stores various information necessary for data collection. The storage unit 14 includes a model information storage unit 141 and a collected information storage unit 142.

(Model information storage unit 141)
The model information storage unit 141 according to the embodiment stores information (model data) indicating the structure of a model (network). FIG. 4 is a diagram illustrating an example of a model information storage section. In the example shown in FIG. 4, the model information storage unit 141 includes items such as "model ID", "use", and "model data".

"Model ID" indicates identification information for identifying a model. “Application” indicates the use of the corresponding model. "Model data" indicates data of a model. Figure 4 shows an example in which conceptual information such as "MDT1" is stored in "model data," but in reality, various information constituting the model, such as network information and functions included in the model, is stored. included.

In the example shown in FIG. 4, the model (model M1) identified by the model ID "M1" indicates that the application is "speech recognition." Model M1 indicates a model used for speech recognition. Further, the model data of model M1 is model data MDT1.

Note that the model information storage unit 141 is not limited to the above, and may store various information depending on the purpose. For example, the model information storage unit 141 stores parameter information of a model learned (generated) through learning processing.

(Collected information storage unit 142)
The collected information storage unit 142 according to the embodiment stores information regarding the user's utterances (voices) collected by the terminal device 10. FIG. 5 is a diagram illustrating an example of a collected information storage section. In the example shown in FIG. 5, the collected information storage unit 142 includes items such as "voice ID,""voice," and "recognition result."

“Voice ID” indicates identification information for identifying the collected voice. "Audio" indicates collected audio. Although FIG. 5 shows an example in which conceptual information such as "ADT1" is stored in "audio," in reality, various information related to speech is included, such as collected audio data, for example, audio waveform data. “Recognition result” indicates the recognition result of the corresponding voice. Although Fig. 5 shows an example in which conceptual information such as "RS1" is stored in "recognition result," in reality, it is the result of speech recognition, such as the result of converting speech data into character data (character string). It also includes information indicating the content included in the audio.

The example shown in FIG. 5 shows that the recognition result of the voice (voice AD1) identified by the voice ID "AD1" is the recognition result RS1.

Note that the collected information storage unit 142 is not limited to the above, and may store various information depending on the purpose. For example, the collected information storage unit 142 stores each sound in association with information indicating under what circumstances the sound was detected. For example, the collected information storage unit 142 stores each sound in association with its SN ratio.

(Control unit 15)
Returning to FIG. 3, the explanation will be continued. The control unit 15 is a controller that processes information such as programs stored inside the terminal device 10 (for example, transmission programs related to This is realized by executing a program (program) using a RAM or the like as a work area. Further, the control unit 15 is a controller, and may be realized by, for example, an integrated circuit such as an ASIC (Application Specific Integrated Circuit) or an FPGA (Field Programmable Gate Array).

As shown in FIG. 3, the control unit 15 includes a receiving unit 151, a voice recognition unit 152, a collecting unit 153, a notification unit 154, a receiving unit 155, a determining unit 156, and a transmitting unit 157. , realizes or executes the information processing functions and operations described below. Note that the internal configuration of the control unit 15 is not limited to the configuration shown in FIG. 3, and may be any other configuration as long as it performs information processing to be described later.

(Receiving unit 151)
The receiving unit 151 receives various information. The receiving unit 151 receives various information from an external information processing device. The receiving unit 151 receives various information from other information processing devices such as the server device 100.

The receiving unit 151 receives from the server device 100 information indicating a condition regarding the voice for which the server device 100 requests information provision.

The receiving unit 151 receives the model learned by the server device 100 from the server device 100. The receiving unit 151 receives the speech recognition model from the server device 100. The receiving unit 151 receives the model M1 from the server device 100.

(Voice recognition unit 152)
The speech recognition unit 152 executes various processes related to speech recognition. The speech recognition unit 152 uses the information stored in the storage unit 14 to execute speech recognition processing. The speech recognition unit 152 executes speech recognition processing using a speech recognition model. The speech recognition unit 152 executes speech recognition processing using the speech recognition model received by the receiving unit 151. The speech recognition unit 152 executes speech recognition processing using the model M1 received by the receiving unit 151. For example, the voice recognition unit 152 uses the model M1 stored in the model information storage unit 141 to execute voice recognition processing.

The speech recognition unit 152 converts the user's utterance (speech) into text information (text data) using the model M1, thereby converting the user's utterance into text. Furthermore, the speech recognition unit 152 analyzes the content of the user's utterance. The speech recognition unit 152 estimates the content of the user's utterance by analyzing the user's utterance using various conventional techniques as appropriate. For example, the speech recognition unit 152 may analyze the content of the user's utterance using a natural language understanding (NLU) or automatic speech recognition (ASR) function.

(Collection unit 153)
The collection unit 153 collects various information. The collection unit 153 decides to collect various information. The collection unit 153 collects various information based on information from an external information processing device. The collection unit 153 collects various information based on the information stored in the storage unit 14. The collection unit 153 collects data through sensing using the model M1 stored in the model information storage unit 141.

The collection unit 153 associates the voice uttered by the user with the recognition result of the voice recognition of the utterance, and collects them in the storage unit 14 within the own device. The collection unit 153 stores the voice and the recognition result of the voice in the collected information storage unit 142 in association with each other. The collection unit 153 associates the voice uttered by the user with the result of voice recognition by the voice recognition unit 152 and registers the result in the collected information storage unit 142 .

(Notification section 154)
The notification unit 154 executes processing related to notifications to users. The notification unit 154 notifies the user of information. The notification unit 154 notifies the user of information via the output unit 13.

The notification unit 154 notifies the user of information regarding voices that are candidates for transmission to the server device 100. The notification unit 154 displays list information of candidates to be transmitted to the server device 100 on the output unit 13.

The notification unit 154 notifies information indicating an incentive provided to the user when information regarding audio is transmitted to the server device 100. The notification unit 154 displays on the output unit 13 information indicating incentives to be provided to the user when information regarding audio is transmitted to the server device 100.

(Reception Department 155)
The reception unit 155 receives various information. The reception unit 155 receives various operations by the user. For example, the reception unit 155 accepts various operations by the user via the input unit 12.

The accepting unit 155 accepts permission from the user. The accepting unit 155 accepts permission from the user for the candidate to be transmitted to the server device 100 notified by the notifying unit 154.

(Determination unit 156)
The determination unit 156 determines various information. For example, the determination unit 156 determines various information based on various information received by the reception unit 151 from an external device. For example, the determination unit 156 determines various information based on the information stored in the storage unit 14. For example, the determination unit 156 makes the determination using information stored in the storage unit 14 that indicates conditions regarding data collection.

The determination unit 156 determines information to be transmitted to the server device 100. The determination unit 156 determines whether the voice satisfies conditions regarding information to be transmitted to the server device 100. The determining unit 156 determines the timing of transmitting information to the server device 100.

The determining unit 156 determines whether the audio collected by the collecting unit 153 satisfies a predetermined condition. The determining unit 156 determines whether the number of voices collected by the collecting unit 153 is greater than or equal to a predetermined number. The determining unit 156 determines whether the score related to voice recognition of the voice collected by the collecting unit 153 satisfies a predetermined condition. The determining unit 156 determines whether the audio collected by the collecting unit 153 satisfies the noise-related conditions. The determining unit 156 determines whether the utterance corresponding to the voice collected by the collecting unit 153 includes predetermined content.

The determination unit 156 determines whether a predetermined condition specified by the server device 100 is satisfied. The determining unit 156 determines whether a predetermined condition received from the server device 100 is satisfied.

The determination unit 156 transmits to the server device 100 information regarding the voice that the user has approved from among the candidates notified by the notification unit 154. The determining unit 156 determines whether it is the timing to transmit information to the server device 100. The determination unit 156 determines whether the communication environment satisfies predetermined communication conditions. The determination unit 156 determines whether the timing is such that the usage rate of the terminal device by the user is low.

(Transmission unit 157)
The transmitter 157 transmits various information to an external information processing device. For example, the transmitter 157 transmits various information to other information processing devices such as the server device 100. The transmitting unit 157 transmits the information stored in the storage unit 14.

The transmitter 157 transmits various information based on information from other information processing devices such as the server device 100. The transmitting unit 157 transmits various information based on the information stored in the storage unit 14.

The transmitting unit 157 transmits information to the server device 100 according to the determination result by the determining unit 156. The transmitting unit 157 transmits information to the server device 100 when the determining unit 156 determines that the information transmitting conditions are satisfied. If the voice collected by the collection unit 153 satisfies a predetermined condition, the transmitting unit 157 transmits information regarding the voice to the server device 100 that learns a model regarding voice recognition, in accordance with the user's permission.

The transmitter 157 transmits audio data to the server device 100. The transmitter 157 transmits audio waveform data to the server device 100. The transmitter 157 transmits compressed audio data to the server device 100. The transmitter 157 transmits the feature information extracted from the voice to the server device 100.

If the number of voices collected by the collection unit 153 is greater than or equal to a predetermined number, the transmitting unit 157 transmits information regarding the voices to the server device 100 in response to permission. If the score related to voice recognition of the voice collected by the collection unit 153 satisfies a predetermined condition, the transmitting unit 157 transmits information regarding the voice to the server device 100 in response to permission. If the audio collected by the collecting unit 153 satisfies the noise-related conditions, the transmitting unit 157 transmits information regarding the audio to the server device 100 in response to permission. If the utterance corresponding to the voice collected by the collection unit 153 includes predetermined content, the transmitting unit 157 transmits information regarding the voice to the server device 100 in response to permission. If a predetermined condition specified by the server device 100 is satisfied, the transmitter 157 transmits information regarding the audio to the server device 100 in accordance with the permission.

The transmitting unit 157 transmits to the server device 100 information regarding the voice that the user has approved from among the candidates notified by the notifying unit 154 . The transmitting unit 157 transmits information regarding the audio to the server device 100 at a predetermined timing after the accepting unit 155 accepts the user's permission. The transmitter 157 transmits information regarding audio to the server device 100 while the communication environment satisfies predetermined communication conditions. The transmitting unit 157 transmits information regarding audio to the server device 100 at a timing when the usage rate of the terminal device by the user is low. The transmitting unit 157 transmits information regarding the audio that satisfies a predetermined condition received from the server device 100 to the server device 100 in accordance with the user's permission.

(Sensor part 16)
The sensor section 16 includes sensors that detect various sensor information. In the example of FIG. 3, the sensor section 16 includes an audio sensor 161.

(Audio sensor 161)
The audio sensor 161 is, for example, a microphone, and detects audio. For example, the audio sensor 161 detects a user's utterance. Note that the audio sensor 161 may have any configuration as long as it can detect the user's utterance information necessary for processing.

Note that the sensor section 16 is not limited to the above, and may include various sensors. The sensor unit 16 includes an image sensor, a position sensor, an acceleration sensor, a gyro sensor, a temperature sensor, a humidity sensor, an illuminance sensor, a pressure sensor, a proximity sensor, and a sensor for receiving biological information such as odor, sweat, heartbeat, pulse, and brain waves. It may have various sensors such as a sensor. Further, the sensors that detect the above-mentioned various information in the sensor section 16 may be a common sensor, or may be realized by different sensors.

[4. Configuration example of server device]
Next, the configuration of the server device 100 according to the embodiment will be described using FIG. 6. FIG. 6 is a diagram showing a configuration example of the server device 100 according to the embodiment. As shown in FIG. 6, the server device 100 includes a communication section 110, a storage section 120, and a control section 130.

(Communication Department 110)
The communication unit 110 is realized by, for example, a NIC (Network Interface Card). Further, the communication unit 110 is connected to the network N (see FIG. 2) by wire or wirelessly.

(Storage unit 120)
The storage unit 120 is realized by, for example, a semiconductor memory element such as a RAM (Random Access Memory) or a flash memory, or a storage device such as a hard disk or an optical disk. As shown in FIG. 6, the storage unit 120 includes a model information storage unit 121 and a learning data information storage unit 122. Note that the storage unit 120 stores various information not limited to the above information. The storage unit 120 stores information indicating various conditions regarding information transmission. For example, the storage unit 120 stores information indicating conditions for information to be transmitted to the server device 100 (information condition information). For example, the storage unit 120 stores information indicating timing conditions for transmitting to the server device 100 (timing condition information).

(Model information storage unit 121)
The model information storage unit 121 according to the embodiment stores information regarding models. For example, the model information storage unit 121 stores a common model (global model) learned for users and communities. FIG. 7 is a diagram illustrating an example of a model information storage unit according to the embodiment. The model information storage unit 121 shown in FIG. 7 includes items such as "model ID", "use", and "model data".

"Model ID" indicates identification information for identifying a model. “Application” indicates the use of the corresponding model. "Model data" indicates data of a model. Although Figure 7 shows an example in which conceptual information such as "MDT1" is stored in "model data," in reality, various information constituting the model, such as network information and functions included in the model, is stored. included.

In the example shown in FIG. 7, the model identified by the model ID "M1" (model M1) indicates that the application is "speech recognition." Model M1 indicates a model used for speech recognition. Further, the model data of model M1 is model data MDT1.

Note that the model information storage unit 121 is not limited to the above, and may store various information depending on the purpose. For example, the model information storage unit 121 stores parameter information of a model learned (generated) through learning processing. Note that if another device (such as a learning device) performs model learning, the server device 100 does not need to include the model information storage unit 121.

(Learning data information storage unit 122)
The learning data information storage unit 122 according to the embodiment stores various information regarding data used for learning. The learning data information storage unit 122 stores data sets used for learning. FIG. 8 is a diagram illustrating an example of a learning data information storage unit according to the embodiment. For example, the learning data information storage unit 122 stores various information regarding various data such as learning data used for learning and evaluation data used for accuracy evaluation (calculation). FIG. 8 shows an example of the learning data information storage unit 122 according to the embodiment. In the example of FIG. 8, the learning data information storage unit 122 includes items such as "data set ID", "data ID", "data", "label", and "date and time".

"Data set ID" indicates identification information for identifying a data set. "Data ID" indicates identification information for identifying data. Further, "data" indicates data corresponding to data identified by the data ID.

“Label” indicates a label (correct label) attached to corresponding data. For example, the "label" may be information (correct information) indicating the recognition result of the corresponding data (speech). For example, the "label" is correct information indicating the result of converting audio data indicating the user's utterance into character data (character string).

Moreover, "date and time" indicates the time (date and time) regarding the corresponding data. In addition, in the example of FIG. 8, it is illustrated as "DA1" etc., but "date and time" may be a specific date and time such as "August 8, 2021 15:52:14", Information indicating from which model learning the data started to be used, such as "Version XX also started using from model learning" may be stored.

In the example of FIG. 8, the data set (data set DS1) identified by the data set ID "DS1" includes a plurality of data identified by data IDs "DID1", "DID2", "DID3", etc. Show that. For example, each data (learning data) identified by data IDs "DID1", "DID2", "DID3", etc. is audio information (sound data) used for model learning.

For example, the data DT1 identified by the data ID "DID1" is labeled data with a label LB1 attached thereto, indicating that its use started with model learning at date and time DA1. Furthermore, for example, data DT4 identified by data ID "DID4" is data that is collected as unlabeled data and is attached with label LB4, which is a predicted label, and its use starts from model learning at date and time DA4. indicates that it has been done.

Note that the learning data information storage unit 122 is not limited to the above, and may store various information depending on the purpose. For example, the learning data information storage unit 122 may store information such as whether each piece of data is learning data or evaluation data in such a manner that it can be specified. For example, the learning data information storage unit 122 stores learning data and evaluation data in a distinguishable manner. The learning data information storage unit 122 may store information that identifies whether each data is learning data or evaluation data. The server device 100 learns a model based on each data used as learning data and correct answer information. The server device 100 calculates the accuracy of the model based on each data used as evaluation data and the correct answer information. The server device 100 calculates the accuracy of the model by collecting the results of comparing the output result output by the model when the evaluation data is input and the correct answer information.

(Control unit 130)
Returning to FIG. 6, the explanation will be continued. The control unit 130 is a controller, and various programs (corresponding to an example of an information processing program) stored in a storage device inside the server device 100 are stored in a RAM by, for example, a CPU, MPU, ASIC, FPGA, etc. This is realized by using a storage area such as the following as a work area.

In the example shown in FIG. 6, the control unit 130 includes an acquisition unit 131, a determination unit 132, a learning unit 133, and a transmission unit 134. Note that the internal configuration of the control unit 130 is not limited to the configuration shown in FIG. 3, and may be any other configuration as long as it performs information processing to be described later.

(Acquisition unit 131)
The acquisition unit 131 receives various information from an external information processing device via the communication unit 110. The acquisition unit 131 receives various information from the terminal device 10. The acquisition unit 131 stores information regarding the audio received from the terminal device 10 into the storage unit 120. The acquisition unit 131 registers information regarding the audio received from the terminal device 10 in the learning data information storage unit 122 as data used for model learning (learning data).

The acquisition unit 131 acquires various information. The acquisition unit 131 acquires various information from the storage unit 120. The acquisition unit 131 acquires various information from the model information storage unit 121 and the learning data information storage unit 122.

(Decision unit 132)
The determining unit 132 determines various information. For example, the determining unit 132 determines various pieces of information based on various pieces of information acquired by the acquiring unit 131 from an external device. For example, the determining unit 132 determines various information based on various information acquired from the terminal device 10.

The determining unit 132 determines conditions regarding information that the terminal device 10 is requested to provide. For example, the determining unit 132 determines conditions regarding information that the terminal device 10 is requested to provide, based on the information stored in the storage unit 120.

(Learning Department 133)
The learning unit 133 learns various information. The learning unit 133 learns various information based on information from an external information processing device and information stored in the storage unit 120. The learning unit 133 learns various information based on the information stored in the learning data information storage unit 122. The learning unit 133 stores the model generated through learning in the model information storage unit 121. The learning unit 133 stores the model updated through learning in the model information storage unit 121.

The learning unit 133 performs learning processing. The learning unit 133 performs various types of learning. The learning unit 133 learns various information based on the information acquired by the acquisition unit 131. The learning unit 133 learns (generates) a model. The learning unit 133 learns various information such as models. The learning unit 133 generates a model through learning. The learning unit 133 learns the model using various machine learning techniques. For example, the learning unit 133 learns parameters of a model (network). The learning unit 133 learns the model using various machine learning techniques.

The learning unit 133 generates a model M1. The learning unit 133 learns network parameters. For example, the learning unit 133 learns the parameters of the network of model M1.

The learning unit 133 performs learning processing based on the learning data (teacher data) stored in the learning data information storage unit 122. The learning unit 133 generates the model M1 by performing a learning process using the learning data stored in the learning data information storage unit 122. For example, the learning unit 133 generates a model used for speech recognition. The learning unit 133 generates the model M1 by learning the parameters of the network of the model M1.

The learning method by the learning unit 133 is not particularly limited, but for example, training data in which labels and data (images) are linked is prepared, and the learning data is input to a calculation model based on a multilayer neural network. You can learn it. Further, for example, a method based on a DNN (Deep Neural Network) such as a CNN (Convolutional Neural Network) or a 3D-CNN may be used. When processing time series data such as audio, the learning unit 133 uses a method based on a recurrent neural network (RNN) or an extended RNN called LSTM (Long Short-Term Memory units). Good too. Note that if another device (such as a learning device) performs model learning, the server device 100 does not need to include the learning unit 133.

(Transmission unit 134)
The transmitting unit 134 transmits various information to the terminal device 10 via the communication unit 110. The transmitter 134 transmits to the terminal device 10 a voice recognition model used by the terminal device 10 to perform voice recognition within itself. The transmitter 134 provides the model M1 to the terminal device 10.

The transmitting unit 134 transmits to the terminal device 10 information indicating a predetermined condition specifying a voice requesting the terminal device 10 to transmit information. The transmitting unit 134 transmits information indicating the conditions determined by the determining unit 132 to the terminal device 10. The transmitter 134 may transmit the collected data to another device (such as a learning device).

[5. Processing procedure]
Next, a processing procedure by the terminal device 10 according to the embodiment will be described using FIG. 9. FIG. 9 is a flowchart showing the processing procedure according to the embodiment.

As shown in FIG. 9, the terminal device 10 associates the voice uttered by the user with the recognition result of the voice recognition of the utterance and collects them in the storage unit 120 within the terminal device 10 (step S101).

Then, if the collected voice satisfies a predetermined condition, the terminal device 10 transmits information regarding the voice to the server device 100 in accordance with the user's permission (step S102).

[6. effect〕
As described above, the terminal device 10 according to the present application is used by a user and performs voice recognition on its own, and includes a collection section 153 and a transmission section 157. The collection unit 153 associates the voice uttered by the user with the recognition result of the voice recognition of the utterance, and collects them in the storage unit 14 within the own device. If the audio collected by the collecting unit 153 satisfies a predetermined condition, the transmitting unit 157 transmits information regarding the audio to the server device 100 in accordance with the user's permission.

In this way, the terminal device 10 performs voice recognition of the user's utterance within itself, and transmits information regarding the voice to the server device 100 if the conditions are met and the user gives permission. Thereby, even if the terminal device 10 performs voice recognition, the terminal device 10 can appropriately transmit information regarding the voice collected by the terminal device 10 to another device.

Furthermore, in the terminal device 10 , the transmitter 157 transmits audio data to the server device 100 . In this manner, by transmitting voice data to the server device 100, the terminal device 10 can appropriately transmit information regarding the voice collected by the terminal device 10 to other devices.

Furthermore, in the terminal device 10 , the transmitter 157 transmits audio waveform data to the server device 100 . In this way, by transmitting voice waveform data to the server device 100, the terminal device 10 can appropriately transmit information regarding the voice collected by the terminal device 10 to other devices.

Furthermore, in the terminal device 10 , the transmitter 157 transmits compressed audio data to the server device 100 . In this way, the terminal device 10 transmits compressed audio data to the server device 100, thereby suppressing an increase in communication volume and transmitting information regarding the audio collected by the terminal device 10 to other devices. Can be sent appropriately.

Furthermore, in the terminal device 10 , the transmitter 157 transmits feature information extracted from the voice to the server device 100 . In this way, the terminal device 10 can appropriately transmit information regarding the voice collected by the terminal device 10 to other devices by transmitting compressed voice data to the server device 100.

Further, in the terminal device 10, when the number of voices collected by the collecting unit 153 is equal to or greater than a predetermined number, the transmitting unit 157 transmits information regarding the voice to the server device 100 in response to permission. In this way, when the number of collected voices exceeds a predetermined number, the terminal device 10 transmits the information about the voices to the server device 100, thereby transmitting the information about the voices collected by the terminal device 10 to other voices. can be sent to the device appropriately.

Furthermore, in the terminal device 10, if the score regarding the voice recognition of the voice collected by the collection unit 153 satisfies a predetermined condition, the transmitter 157 transmits information regarding the voice to the server device 100 in response to permission. In this way, the terminal device 10 transmits the information related to the voice collected by the terminal device 10 by transmitting the information related to the voice to the server device 100 when the score related to voice recognition of the collected voice satisfies a predetermined condition. It can be properly transmitted to other devices.

Furthermore, in the terminal device 10, when the audio collected by the collecting unit 153 satisfies the condition regarding noise, the transmitting unit 157 transmits information regarding the audio to the server device 100 in response to permission. In this way, the terminal device 10 transmits the information regarding the audio to the server device 100 when the collected audio satisfies the noise-related conditions, so that the information regarding the audio collected by the terminal device 10 can be appropriately transmitted to other devices. can be sent to.

Furthermore, in the terminal device 10, if the utterance corresponding to the voice collected by the collection unit 153 includes predetermined content, the transmitting unit 157 transmits information regarding the voice to the server device 100 in accordance with permission. In this way, when the utterance corresponding to the voice includes predetermined content, the terminal device 10 transmits the information about the voice to the server device 100, thereby transmitting the information about the voice collected by the terminal device 10 to other devices. can be sent appropriately.

Furthermore, in the terminal device 10, if a predetermined condition specified by the server device 100 is satisfied, the transmitter 157 transmits information regarding the audio to the server device 100 in response to permission. In this way, the terminal device 10 transmits information regarding the audio collected by the terminal device 10 to other devices by transmitting the information regarding the audio to the server device 100 when the conditions specified by the server device 100 are met. Can be sent appropriately.

The terminal device 10 also includes a reception section 155. The accepting unit 155 accepts permission from the user. In this way, the terminal device 10 receives permission from the user and transmits the information regarding the audio only when the user has given permission, thereby transmitting the information regarding the audio collected by the terminal device 10 to other devices. Can be sent appropriately.

Further, in the terminal device 10, the transmitting unit 157 transmits information regarding audio to the server device 100 at a predetermined timing after the accepting unit 155 accepts the user's permission. In this way, the terminal device 10 transmits the information regarding the audio to the server device 100 at a predetermined timing after the user's permission, so that the information regarding the audio collected by the terminal device 10 can be transmitted to other devices at an appropriate timing. can be sent by.

Furthermore, in the terminal device 10, the transmitter 157 transmits information regarding audio to the server device 100 while the communication environment satisfies predetermined communication conditions. In this way, the terminal device 10 transmits the information regarding the audio to the server device 100 while the communication environment satisfies the predetermined communication conditions, so that the information regarding the audio collected by the terminal device 10 can be transmitted to other devices. can be sent at the appropriate time.

Furthermore, in the terminal device 10, the transmitter 157 transmits information regarding audio to the server device 100 at a timing when the usage rate of the terminal device by the user is low. In this way, the terminal device 10 transmits the information regarding the audio collected by the terminal device 10 to other devices by transmitting the information regarding the audio to the server device 100 at a timing when the usage rate of the terminal device 10 by the user is low. It can be sent at the right time.

The terminal device 10 also includes a notification section 154. The notification unit 154 notifies the user of information regarding voices that are candidates for transmission to the server device 100. In this way, the terminal device 10 can make the user aware of what kind of information is to be transmitted to the server device 100 by notifying the user of information regarding voices that are candidates to be transmitted to the server device 100. .

Furthermore, in the terminal device 10 , the transmitting unit 157 transmits to the server device 100 information regarding the voice that the user has approved from among the candidates notified by the notifying unit 154 . In this way, the terminal device 10 notifies the user of the candidates, and transmits the information regarding the voices to the server device 100 for the voices that the user has approved, so that the information regarding the voices collected by the terminal device 10 can be transmitted to the server device 100. It can be properly transmitted to other devices.

Furthermore, in the terminal device 10, the notification unit 154 notifies the user of information indicating an incentive provided to the user when information regarding audio is transmitted to the server device 100. In this way, the terminal device 10 can motivate the user to provide information by informing the user of the incentives provided to the user as compensation for providing information, and the terminal device 10 can transmit information to other devices. The amount of information displayed can be increased.

As described above, the information processing system 1 according to the present application includes a terminal device 10 that is used by a user and performs voice recognition on its own, and a server device 100. The server device 100 transmits to the terminal device 10 information indicating a predetermined condition specifying a voice requesting the terminal device 10 to transmit information. The terminal device 10 transmits information regarding the audio that satisfies a predetermined condition received from the server device 100 to the server device 100 according to the user's permission.

In this way, in the information processing system 1, the terminal device 10 used by the user performs voice recognition of the voice of the user's utterance within its own device, and transmits information about the voice that satisfies the specified conditions from the server device 100 to the terminal device 10. sends it to the server. Thereby, even if the terminal device 10 performs voice recognition, the information processing system 1 can appropriately transmit information regarding the voice collected by the terminal device 10 to other devices.

[7. Hardware configuration]
Further, the terminal device 10 and the server device 100 according to the embodiments described above are realized by, for example, a computer 1000 having a configuration as shown in FIG. 10. The following will explain the server device 100 as an example. FIG. 10 is a diagram showing an example of a hardware configuration. The computer 1000 is connected to an output device 1010 and an input device 1020, and a calculation device 1030, a primary storage device 1040, a secondary storage device 1050, an output I/F (Interface) 1060, an input I/F 1070, and a network I/F 1080 are connected to a bus. 1090.

The arithmetic unit 1030 operates based on programs stored in the primary storage device 1040 and the secondary storage device 1050, programs read from the input device 1020, and performs various processes. The arithmetic device 1030 is realized by, for example, a CPU (Central Processing Unit), an MPU (Micro Processing Unit), an ASIC (Application Specific Integrated Circuit), an FPGA (Field Programmable Gate Array), or the like.

The primary storage device 1040 is a memory device such as a RAM (Random Access Memory) that temporarily stores data used by the calculation device 1030 for various calculations. Further, the secondary storage device 1050 is a storage device in which data used by the calculation device 1030 for various calculations and various databases are registered, and includes a ROM (Read Only Memory), an HDD (Hard Disk Drive), and an SSD (Solid Disk Drive). This is realized using flash memory, etc. The secondary storage device 1050 may be a built-in storage or an external storage. Further, the secondary storage device 1050 may be a removable storage medium such as a USB memory or an SD (Secure Digital) memory card. Further, the secondary storage device 1050 may be a cloud storage (online storage), a NAS (Network Attached Storage), a file server, or the like.

The output I/F 1060 is an interface for transmitting information to be output to the output device 1010 that outputs various information such as a display, a projector, and a printer. (Digital Visual Interface) and HDMI (registered trademark) (High Definition Multimedia Interface). Further, the input I/F 1070 is an interface for receiving information from various input devices 1020 such as a mouse, keyboard, keypad, button, scanner, etc., and is realized by, for example, a USB or the like.

Further, the output I/F 1060 and the input I/F 1070 may be wirelessly connected to the output device 1010 and the input device 1020, respectively. That is, output device 1010 and input device 1020 may be wireless devices.

Moreover, the output device 1010 and the input device 1020 may be integrated like a touch panel. In this case, the output I/F 1060 and the input I/F 1070 may also be integrated as an input/output I/F.

Note that the input device 1020 is, for example, an optical recording medium such as a CD (Compact Disc), a DVD (Digital Versatile Disc), or a PD (Phase change rewritable disk), a magneto-optical recording medium such as an MO (Magneto-Optical disk), or a tape. It may be a device that reads information from a medium, a magnetic recording medium, a semiconductor memory, or the like.

Network I/F 1080 receives data from other devices via network N and sends it to computing device 1030, and also sends data generated by computing device 1030 to other devices via network N.

Arithmetic device 1030 controls output device 1010 and input device 1020 via output I/F 1060 and input I/F 1070. For example, the arithmetic device 1030 loads a program from the input device 1020 or the secondary storage device 1050 onto the primary storage device 1040, and executes the loaded program.

For example, when the computer 1000 functions as the server device 100, the arithmetic unit 1030 of the computer 1000 realizes the functions of the control unit 130 by executing a program loaded onto the primary storage device 1040. Further, the arithmetic device 1030 of the computer 1000 may load a program obtained from another device via the network I/F 1080 onto the primary storage device 1040, and execute the loaded program. Further, the arithmetic device 1030 of the computer 1000 may cooperate with other devices via the network I/F 1080, and may call and use program functions, data, etc. from other programs of other devices.

[8. others〕
Although the embodiments of the present application have been described above, the present invention is not limited to the contents of these embodiments. Furthermore, the above-mentioned components include those that can be easily assumed by those skilled in the art, those that are substantially the same, and those that are in a so-called equivalent range. Furthermore, the aforementioned components can be combined as appropriate. Furthermore, various omissions, substitutions, or modifications of the constituent elements can be made without departing from the gist of the embodiments described above.

Further, among the processes described in the above embodiments, all or part of the processes described as being performed automatically can be performed manually, or the processes described as being performed manually can be performed manually. All or part of this can also be performed automatically using known methods. In addition, the processing procedures, specific names, and information including various data and parameters shown in the above documents and drawings may be changed arbitrarily, unless otherwise specified. For example, the various information shown in each figure is not limited to the illustrated information.

Further, each component of each device shown in the drawings is functionally conceptual, and does not necessarily need to be physically configured as shown in the drawings. In other words, the specific form of distributing and integrating each device is not limited to what is shown in the diagram, and all or part of the devices can be functionally or physically distributed or integrated in arbitrary units depending on various loads and usage conditions. Can be integrated and configured.

For example, the server device 100 described above may be realized by a plurality of server computers, and depending on the function, it may be realized by calling an external platform etc. using an API (Application Programming Interface), network computing, etc. Can be changed flexibly.

Furthermore, the above-described embodiments and modifications can be combined as appropriate within a range that does not conflict with the processing contents.

Further, the above-mentioned "section, module, unit" can be read as "means", "circuit", etc. For example, the acquisition unit can be read as an acquisition means or an acquisition circuit.

1 Information processing system 10 Terminal device 11 Communication unit 12 Input unit 13 Output unit 14 Storage unit 141 Model information storage unit 142 Collection information storage unit 15 Control unit 151 Receiving unit 152 Voice recognition unit 153 Collection unit 154 Notification unit 155 Reception unit 156 Judgment Section 157 Transmission section 16 Sensor section 161 Voice sensor 100 Server device 110 Communication section 120 Storage section 121 Model information storage section 122 Learning data information storage section 130 Control section 131 Acquisition section 132 Determination section 133 Learning section 134 Transmission section

Claims (20)

A terminal device that is used by a user and performs voice recognition on its own,
a collection unit that associates the voice uttered by the user with the recognition result of the voice recognition of the utterance and collects it in a storage unit within the own device;
a transmitting unit that transmits information regarding the voice to a server device in accordance with the user's consent if the voice collected by the collection unit satisfies a predetermined condition;
A terminal device comprising: The transmitter includes:
The terminal device according to claim 1, wherein the audio data is transmitted to the server device. The transmitter includes:
The terminal device according to claim 1 or 2, wherein the audio waveform data is transmitted to the server device. The transmitter includes:
The terminal device according to any one of claims 1 to 3, wherein data obtained by compressing the audio data is transmitted to the server device. The transmitter includes:
The terminal device according to any one of claims 1 to 4, characterized in that the feature information extracted from the voice is transmitted to the server device. The transmitter includes:
If the number of the voices collected by the collection unit is equal to or greater than a predetermined number, information regarding the voices is transmitted to the server device in accordance with the permission. The terminal device described in . The transmitter includes:
If the score regarding the voice recognition of the voice collected by the collection unit satisfies a predetermined condition, the information regarding the voice is transmitted to the server device in accordance with the permission. The terminal device according to any one of the items. The transmitter includes:
According to any one of claims 1 to 7, when the voice collected by the collection unit satisfies a condition regarding noise, information regarding the voice is transmitted to the server device in accordance with the permission. terminal device. The transmitter includes:
Any one of claims 1 to 8, characterized in that when the utterance corresponding to the voice collected by the collection unit includes predetermined content, information regarding the voice is transmitted to the server device in accordance with the permission. The terminal device according to item 1. The transmitter includes:
The terminal according to any one of claims 1 to 9, wherein if the predetermined condition specified by the server device is satisfied, information regarding the audio is transmitted to the server device in accordance with the permission. Device. a reception unit that receives the permission from the user;
The terminal device according to any one of claims 1 to 10, comprising: The transmitter includes:
The terminal device according to claim 11, wherein the information regarding the audio is transmitted to the server device at a predetermined timing after the permission from the user is accepted by the reception unit. The transmitter includes:
The terminal device according to claim 12, wherein the information regarding the voice is transmitted to the server device while a communication environment satisfies predetermined communication conditions. The transmitter includes:
The terminal device according to claim 12 or 13, wherein the information regarding the audio is transmitted to the server device at a timing when the usage rate of the terminal device by the user is low. a notification unit that notifies the user of information regarding the voice that is a candidate to be transmitted to the server device;
The terminal device according to any one of claims 1 to 14, further comprising the following. The transmitter includes:
16. The terminal device according to claim 15, wherein information regarding the voice approved by the user among the candidates notified by the notification unit is transmitted to the server device. The notification section is
The terminal device according to claim 15 or 16, wherein information indicating an incentive provided to the user is notified when the information regarding the voice is transmitted to the server device. A transmission method executed by a terminal device that is used by a user and performs voice recognition on its own device,
a collection step of associating the voice uttered by the user with the recognition result of the voice recognition of the utterance and collecting it in a storage unit within the own device;
If the voice collected in the collecting step satisfies a predetermined condition, a transmitting step of transmitting information regarding the voice to a server device according to the user's permission;
A transmission method characterized by comprising: A transmission program executed by a terminal device used by a user and performing voice recognition on its own device,
a collection procedure of associating the voice uttered by the user with the recognition result of the voice recognition of the utterance and collecting it in a storage unit within the own device;
a transmission procedure of transmitting information regarding the voice to a server device in accordance with the user's consent if the voice collected by the collection procedure satisfies a predetermined condition;
A transmission program that causes a terminal device to execute. A terminal device that is used by a user and performs voice recognition on its own,
a server device;
has
The server device includes:
transmitting to the terminal device information indicating a predetermined condition specifying a voice requesting the terminal device to transmit information;
The terminal device is
An information processing system characterized in that information regarding audio that satisfies the predetermined condition received from the server device is transmitted to the server device in accordance with permission from the user.

Images