WO2020079918A1

WO2020079918A1 - Information processing device and information processing method

Info

Publication number: WO2020079918A1
Application number: PCT/JP2019/029985
Authority: WO
Inventors: 暦本　純一
Original assignee: ソニー株式会社
Priority date: 2018-10-18
Filing date: 2019-07-31
Publication date: 2020-04-23
Also published as: CN112840397A

Abstract

Provided are an information processing device and an information processing method which make it possible to obtain acoustic information intended by a user without uttering voice. The information processing device is provided with a control unit (110) which converts a plurality of time-series images acquired by ultrasonic echo indicating the state of the inside of an oral cavity into information corresponding to the state of the inside of the oral cavity on the basis of an algorithm acquired by machine learning.

Description

Information processing apparatus and information processing method

The present disclosure relates to an information processing device and an information processing method.

In recent years, with the improvement of voice recognition accuracy, devices that can be controlled by voice commands have become popular. For example, in smartphones, car navigation devices, and the like, it has become common to use a search function using voice commands. Further, it has become possible to create a document by documenting the contents input by voice. Further, a speaker-type voice interface device, which is called a smart speaker and operates by a voice command, has become widespread.

However, situations in which voice commands are used may be limited. For example, operating a smartphone or the like by voice in a public space such as on a train or in a library is difficult for people around to accept. In addition, speaking out confidential information such as personal information in a public space poses a risk of personal information leakage. Therefore, the voice interface using voice commands is limited to use in places where the influence of utterance on the surroundings is clear, such as smart speakers used in homes and car navigation devices used in vehicles. It tends to be done.

For example, if the above devices can be operated without actually producing a voice, the above devices can be used regardless of the location. Specifically, if the wearable computer has a function capable of operating the device without producing a voice, by always wearing the wearable computer, it becomes possible to always obtain the service regardless of the place. Therefore, research on recognition technology for unvoiced utterances that can perform voice recognition without producing a voice is under way.

Related to the above-mentioned recognition technology of unvoiced speech, for example, Patent Document 1 below discloses a technology of detecting a movement and a place of a voice organ by an electromagnetic wave to identify a voice. Further, in addition to the technique disclosed in Patent Document 1 below, research on a pharyngeal microphone and a microphone to be attached to the throat for surely acquiring sound in a noisy environment is also under way.

Tokuyo 2000-504848

However, the recognition technology for non-voiced utterances mentioned above requires only a whispering amount of voice, so its use in public spaces is still limited. In addition, if the volume of whispering is reduced in order to bring the voice closer to the non-voiced state, the recognition accuracy may decrease.

Therefore, the present disclosure proposes a new and improved information processing apparatus and information processing method that allow a user to obtain intended acoustic information without vocalization.

According to the present disclosure, a plurality of time-series images showing an intraoral state acquired by ultrasonic echo, a control unit that converts information corresponding to the intraoral state based on an algorithm acquired by machine learning, An information processing apparatus including:

Further, according to the present disclosure, it is possible to convert a plurality of time-series images showing an intraoral state acquired by ultrasonic echo into information corresponding to the intraoral state based on an algorithm acquired by machine learning. An information processing method executed by a processor is provided, including:

As described above, according to the present disclosure, it is possible to obtain intended acoustic information without the user speaking. Note that the above effects are not necessarily limited, and in addition to or in place of the above effects, any of the effects shown in this specification, or other effects that can be grasped from this specification. May be played.

It is a figure showing an example of composition of a voiceless utterance system concerning an embodiment of this indication. It is a figure which shows the echo image which concerns on the same embodiment. It is a figure which shows the outline | summary of the function of the voiceless utterance system which concerns on the same embodiment. It is a block diagram showing an example of functional composition of a voiceless utterance system concerning the embodiment. It is a figure which shows the example of a production | generation of the acoustic feature-value which concerns on the same embodiment. It is a figure which shows the structure of the 2nd neural network which concerns on the same embodiment. It is a flow chart which shows a flow of machine learning which acquires the 1st neural network concerning the embodiment. It is a flow chart which shows a flow of machine learning which acquires the 2nd neural network concerning the embodiment. It is a flow chart which shows a flow of processing in a personal digital assistant concerning the embodiment. It is a block diagram which shows the hardware structural example of the information processing apparatus which concerns on the same embodiment.

Hereinafter, preferred embodiments of the present disclosure will be described in detail with reference to the accompanying drawings. In the present specification and the drawings, constituent elements having substantially the same functional configuration are designated by the same reference numerals, and duplicate description will be omitted.

The description will be given in the following order.
1. Embodiment of the present disclosure 1.1. Overview 1.2. Configuration of voiceless speech system 1.3. Function of voiceless speech system 1.4. Processing of voiceless speech system 2. Modification 3. Application example 4. Hardware configuration example 5. Summary

<< 1. Embodiments of the present disclosure >>
<1.1. Overview>
In recent years, with the improvement of voice recognition accuracy, devices that can be controlled by voice commands have become widespread. For example, in smartphones, car navigation devices, and the like, it has become common to use a search function using voice commands. Further, it has become possible to create a document by documenting the contents input by voice. Further, a speaker-type voice interface device, which is called a smart speaker and operates by a voice command, has become widespread.

Related to the recognition technology for unvoiced utterances described above, for example, a technology for detecting the movement and location of a voice organ by electromagnetic waves to identify a voice is disclosed. In addition, research on a pharyngeal microphone and a microphone to be attached to the throat for surely acquiring sound in a noisy environment is also under way.

The embodiment of the present disclosure is conceived with the above points in mind, and proposes a technology that enables the user to obtain the intended acoustic information without uttering. Hereinafter, the present embodiment will be sequentially described in detail.

<1.2. Configuration of voiceless speech system>
First, the configuration of the voiceless speech system according to the embodiment of the present disclosure will be described. FIG. 1 is a diagram illustrating a configuration example of a voiceless speech system according to an embodiment of the present disclosure. As illustrated in FIG. 1, the voiceless speech system 1000 according to the present embodiment includes a mobile terminal 10, an ultrasonic echo device 20, and a voice input / output device 30. Various devices may be connected to the mobile terminal 10. For example, the ultrasonic echo device 20 and the voice input / output device 30 are connected to the mobile terminal 10, and information cooperation is performed between the devices. The ultrasonic echo device 20 and the voice input / output device 30 are wirelessly connected to the mobile terminal 10 according to the present embodiment. For example, the mobile terminal 10 performs short-distance wireless communication with the ultrasonic echo device 20 and the voice input / output device 30 using Bluetooth (registered trademark). The ultrasonic echo device 20 and the audio input / output device 30 may be connected to the mobile terminal 10 by wire or may be connected via a network.

(1) Mobile terminal 10
The mobile terminal 10 is an information processing device capable of recognition processing based on machine learning. The recognition process according to the present embodiment is, for example, a voice recognition process. The voice recognition process is performed, for example, on information about a voice generated from an image (still image / moving image). Specifically, the mobile terminal 10 converts an image (hereinafter also referred to as an echo image) showing the state of the inside of the oral cavity of the user 12 into information regarding voice, and performs voice recognition processing on the converted information regarding voice. I do.

In the present embodiment, a plurality of time-series images showing a time-series change in the intraoral state when the user 12 changes the intraoral state without uttering a voice is converted into information regarding voice. It Thereby, the mobile terminal 10 according to the present embodiment can realize voice recognition without vocalization. The plurality of time-series images are echo images showing changes in the state of the oral cavity when the user moves at least one of the mouth and tongue without uttering. In addition, hereinafter, the plurality of time-series images showing the time-series changes in the oral cavity state of the user 12 are also referred to as time-series echo images.

Information related to voice is, for example, information that can be recognized by a voice recognition device (hereinafter, also referred to as acoustic information). The acoustic information is, for example, a spectrogram that three-dimensionally shows a time-series change in the characteristics of the voice such as the pitch, strength, and the like of the voice according to frequency, amplitude, and time.

Information related to audio is converted from images using algorithms acquired by machine learning. The machine learning according to this embodiment is performed by, for example, deep learning. The algorithm acquired by the machine learning is, for example, a neural network (NN: Neural Network). An image is used as an input for the machine learning. Therefore, the machine learning is performed using a convolutional neural network (CNN: Convolutional Neural Network) suitable for deep learning of image processing. In this embodiment, a time-series echo image when the user 12 makes a voice is used for machine learning.

There are two types of algorithms (neural networks) according to this embodiment. The first algorithm is a first neural network that performs a process of converting a time-series echo image into acoustic information (first acoustic information) when the user 12 changes the state of the oral cavity without producing a sound. It is a network (hereinafter, also referred to as NN1). The second algorithm is a second neural network (hereinafter, also referred to as NN2) that performs a process of converting acoustic information converted by NN1 into more accurate acoustic information (second acoustic information). is there. The more accurate acoustic information is, for example, acoustic information obtained by converting the uttered voice that is the voice when the user 12 actually utters. The details of NN1 and NN2 will be described later.

Note that, as described above, there are two types of time-series echo images according to the present embodiment, a time-series echo image converted into acoustic information by the NN1 and a time-series echo image used for machine learning. The time-series echo image converted into acoustic information is a time-series echo image in the oral cavity when the user 12 changes the state in the oral cavity without making a sound, and therefore, in the following, a non-voiced time-series echo. Also called an image. Further, since the time-series echo image used for machine learning is the time-series echo image in the oral cavity when the user 12 utters, it is also referred to as a vocalization time-series echo image below.

Also, as described above, the acoustic information according to the present embodiment has a plurality of acoustic information. The acoustic information (first acoustic information) converted by the NN1 is a spectrogram obtained by converting an unvoiced time-series echo image, and is hereinafter referred to as an unvoiced image spectrogram. Further, since the acoustic information (second acoustic information) converted by the NN2 is a spectrogram with higher accuracy obtained by converting the unvoiced image spectrogram, it is hereinafter referred to as a high-accuracy unvoiced image spectrogram.

Note that machine learning is performed in each of NN1 and NN2, but the learning information used for each machine learning is different. The learning information (first learning information) used for the machine learning of NN1 is a vocalization time-series echo image and vocalized voice. The learning information (second learning information) used for machine learning of the NN2 includes acoustic information (third acoustic information) obtained by converting the vocal image spectrogram via the NN1 and acoustic information corresponding to the vocal sound (fourth acoustic information). Acoustic information).

The acoustic information (third acoustic information) obtained by converting the vocal image spectrogram through the NN1 is a spectrogram obtained by converting the vocal time-series echo image by the NN1, and is hereinafter referred to as a vocal image spectrogram. The acoustic information (fourth acoustic information) corresponding to the uttered voice is a spectrogram corresponding to the voice when the user 12 actually utters, and is hereinafter referred to as a uttered voice spectrogram. The vocal image spectrogram (third acoustic information) is used as an input of the NN2 machine learning, and the vocalized speech spectrogram (fourth acoustic information) is used as an output of the NN2 machine learning.

The mobile terminal 10 also has a function of controlling the overall operation of the voiceless speech system 1000. For example, the mobile terminal 10 controls the overall operation of the voiceless speech system 1000 based on the information associated with each device. Specifically, the mobile terminal 10 controls the process related to voice recognition in the mobile terminal 10 and the operation of the voice input / output device 30 based on the information received from the ultrasonic echo device 20 and the voice input / output device 30. Further, the mobile terminal 10 may control the operation of the ultrasonic echo device 20.

The mobile terminal 10 is realized by, for example, a smartphone as shown in FIG. The mobile terminal 10 is not limited to a smartphone. For example, the mobile terminal 10 may be a terminal device such as a tablet terminal, a PC, a wearable terminal, or an agent device that has the functions of the mobile terminal 10 implemented as an application. That is, the mobile terminal 10 can be realized as an arbitrary terminal device.

(2) Ultrasonic echo device 20
The ultrasonic echo device 20 is a device that acquires an echo image in the oral cavity of the user 12. The ultrasonic echo device 20 acquires an echo image by using an ultrasonic examination technique widely used in medicine. The ultrasonic echo device 20 includes an ultrasonic output device capable of outputting ultrasonic waves, and causes the ultrasonic output device attached to the body surface of the user 12 to output ultrasonic waves into the body of the user 12 and An echo image is acquired based on the ultrasonic waves reflected by the organ. Then, the ultrasonic echo device 20 transmits the acquired echo image to the mobile terminal 10.

The ultrasonic echo device 20 according to the present embodiment is realized as a neckband type device as shown in FIG. 1, for example. The ultrasonic wave output unit 22 of the ultrasonic wave echo device 20 is provided with an ultrasonic wave output device. The ultrasonic echo device 20 shown in FIG. 1 is provided with two ultrasonic wave output parts 22 of ultrasonic

wave output parts

22a and 22b because of its structure as a neckband type device. The number of ultrasonic wave output units 22 is not limited to two, and the ultrasonic wave echo device 20 may include at least one ultrasonic wave output unit 22.

When the user 12 wears the ultrasonic echo device 20 so that the ultrasonic wave output unit 22 is located under the chin, ultrasonic waves are output toward the oral cavity of the user 12. Thereby, the ultrasonic echo device 20 can acquire an echo image in the oral cavity of the user 12. The voice is uttered after the vibration due to the vocal cords is adjusted by the opening of the tongue and the mouth. Therefore, it can be said that the echo image in the oral cavity of the user 12 acquired by the ultrasonic echo device 20 has effective information as an image to be converted into acoustic information.

Here, an echo image acquired by the ultrasonic echo device 20 will be described. FIG. 2 is a diagram showing an echo image according to this embodiment. The echo image 40 is an echo image in the oral cavity of the user 12 acquired by the ultrasonic echo device 20. In the echo image 40 shown in FIG. 2, the tongue tip 402, the tongue surface 404, and the tongue root 406 are shown. In the present embodiment, the ultrasonic echo device 20 continuously acquires the echo images 40, thereby acquiring a plurality of time-series images (time-series echo images) that show time-series changes in the intraoral state of the user 12. To do.

(3) Voice input / output device 30
The voice input / output device 30 is a device capable of inputting / outputting voice. The voice input / output device 30 acquires, for example, the voice uttered by the user 12. Then, the voice input / output device 30 transmits the acquired voice to the mobile terminal 10. The voice input / output device 30 also receives, for example, voice data indicating the content recognized by the mobile terminal 10 from the mobile terminal 10. Then, the voice input / output device 30 outputs the received voice data as voice.

The voice input / output device 30 according to the present embodiment is realized by, for example, a wearable terminal. Specifically, the voice input / output device 30 is preferably a wearable terminal such as an earphone or a bone conduction earphone capable of inputting / outputting voice. Since the voice input / output device 30 is an earphone, a bone conduction earphone, or the like, the amount of voice leaked to the outside can be reduced.

Furthermore, it is more preferable that the voice input / output device 30 has a structure that allows the user 12 to listen to externally generated voice in addition to the voice output from the voice input / output device 30. For example, as shown in FIG. 1, the voice input / output device 30 has an opening 32. Therefore, the user 12 can listen to external sound through the opening 32 even when the sound input / output device 30 is worn. Therefore, even if the user 12 always wears the voice input / output device 30 having the structure, he / she can comfortably spend his / her daily life without any trouble. Further, even when the voice indicating the content recognized by the mobile terminal 10 is output from the speaker such as the smart speaker instead of the voice input / output device 30, the user 12 can hear the voice.

In the present embodiment, an example in which a voice output function for outputting voice and a voice input function for acquiring voice are realized by one device will be described. However, the voice output function and the voice input function are independent devices. May be realized by

<1.3. Functions of voiceless speech system>
The configuration of the voiceless speech system 1000 has been described above. Next, the function of the voiceless speech system 1000 will be described.

<13.1. Overview of functions>
FIG. 3 is a diagram showing an outline of functions of the voiceless speech system according to the present embodiment. The voiceless speech system 1000 first acquires NN1 and NN2 in advance by machine learning based on the vocalization time series echo image and the vocalized voice. When the user 12 changes the state of the oral cavity without making a sound, the ultrasonic echo device 20 acquires the unvoiced time series echo image 42. Then, the acquired unvoiced time-series echo image 42 is converted to the unvoiced image spectrogram 72 via the first neural network 122 (NN1). The unvoiced image spectrogram 72 is a combination of a plurality of acoustic feature quantities 70 in chronological order. Details of the acoustic feature amount 70 will be described later.

After the conversion processing by NN1, the converted unvoiced image spectrogram 72 is converted into a high-accuracy unvoiced image spectrogram 74 via the second neural network 124 (NN2). After the conversion processing by NN2, the converted high-accuracy unvoiced image spectrogram 74 is input to the recognition unit 114 of the mobile terminal 10. Then, the recognition unit 114 performs voice recognition processing based on the input high-accuracy unvoiced image spectrogram 74.

<1.3.2. Functional configuration example>
FIG. 4 is a block diagram showing a functional configuration example of the voiceless speech system according to the present embodiment.

(1) Mobile terminal 10
As shown in FIG. 4, the mobile terminal 10 includes a communication unit 100, a control unit 110, and a storage unit 120. The information processing apparatus according to this embodiment has at least the control unit 110.

(1-1) Communication unit 100
The communication unit 100 has a function of communicating with an external device. For example, the communication unit 100 outputs the information received from the external device to the control unit 110 in the communication with the external device. Specifically, the communication unit 100 outputs the echo image received from the ultrasonic echo device 20 to the control unit 110. Further, the communication unit 100 outputs the voice received from the voice input / output device 30 to the control unit 110.

The communication unit 100 transmits information input from the control unit 110 to an external device in communication with the external device. Specifically, the communication unit 100 transmits the information regarding the acquisition of the echo image input from the control unit 110 to the ultrasonic echo device 20. Further, the communication unit 100 transmits information regarding the input / output of the voice input from the control unit 110 to the voice input / output device 30.

(1-2) Control unit 110
The control unit 110 has a function of controlling the operation of the mobile terminal 10. For example, the control unit 110 converts a plurality of time-series images indicating the intraoral state acquired by ultrasonic echo into information corresponding to the intraoral state based on an algorithm acquired by machine learning. The algorithm has a first neural network, and the control unit 110 converts a plurality of input time-sequential images in the unvoiced state into first acoustic information via the first neural network. For example, the control unit 110 inputs the unvoiced time series echo image input from the communication unit 100 to the NN1. The NN1 converts the input unvoiced time series echo image into an unvoiced image spectrogram. The control unit 110 can perform voice recognition processing by converting the spectrogram into a voice waveform. Therefore, the control unit 110 can perform voice recognition processing based on the unvoiced time-series echo image and control a device that can be operated by voice even if the user 12 does not make a voice.

Further, the algorithm further has a second neural network, and the control unit 110 converts the first acoustic information into second acoustic information corresponding to the voice at the time of utterance, via the second neural network. . For example, the control unit 110 inputs the unvoiced image spectrogram output from NN1 to NN2. The NN2 converts the input unvoiced image spectrogram into a high-precision unvoiced image spectrogram corresponding to the uttered voice. Specifically, it is assumed that the voice indicated by the unvoiced image spectrogram output from the NN1 is “Ulay music.” And the voice indicated by the high-precision unvoiced image spectrogram corresponding to the voiced voice is “Play music.”. At this time, the unvoiced image spectrogram indicating the voice "Ulay music." Is converted to a high-precision unvoiced image spectrogram indicating the voice "Play music." . That is, the NN2 has a role of correcting the voice indicated by the unvoiced image spectrogram converted from the unvoiced time series echo image by the NN1.

In order to realize the above functions, the control unit 110 according to the present embodiment has a machine learning unit 112, a recognition unit 114, and a processing control unit 116, as shown in FIG.

・ Machine learning unit 112
The machine learning unit 112 has a function of performing machine learning using learning information. The machine learning unit 112 acquires an algorithm for converting an echo image into a spectrogram by machine learning. Specifically, the machine learning unit 112 acquires NN1 which is an algorithm for converting the unvoiced time-series echo image into an unvoiced image spectrogram. The machine learning unit 112 also acquires NN2, which is an algorithm for converting the unvoiced image spectrogram into a high-accuracy unvoiced image spectrogram.

NN1 is obtained by machine learning using the first learning information that includes a voice when uttered and a plurality of time-series images when uttered. For example, the NN1 is obtained by machine learning using the voice uttered by the user 12 and the utterance time-series echo image when the user 12 utters the voice as the first learning information. Thereby, the control part 110 can convert an echo image into a spectrogram via NN1.

Note that the first learning information is acquired, for example, by causing the user 12 to read a text or the like. Accordingly, it is possible to acquire an echo image showing a time series change and a speech waveform corresponding to the echo image. The speech waveform can be converted into an acoustic feature amount.

When the control unit 110 inputs a plurality of time-series images when no voice is input to the NN1, the NN1 generates a plurality of acoustic feature amounts per unit time from the plurality of time-series images when no voice is input. , The first acoustic information is generated by synthesizing the generated plural acoustic feature amounts in time series. For example, the NN1 generates a plurality of acoustic feature amounts per unit time from the unvoiced time-series echo image input by the control unit 110, and synthesizes the generated acoustic feature amounts in time-series order to generate a unvoiced image. Generate a spectrogram.

Here, the acoustic feature amount generated by NN1 will be described. FIG. 5 is a diagram showing an example of generation of the acoustic feature quantity according to the present embodiment. The NN1 selects a time-series image at the central time of the unit time from among a plurality of time-series images when no voice is acquired in the unit time, and selects the acoustic feature amount per unit time from the selected time-series image. To generate. For example, the NN1 selects an echo image at the central time of the unit time from the unvoiced time-series echo images acquired in the unit time, and generates an acoustic feature amount per unit time from the selected echo image. The unit time according to the present embodiment is, for example, the time when the number of acquired echo images is any of 5 to 13. In the present embodiment, the unit time is the time when 13 echo images are acquired. Specifically, as shown in FIG. 5, the NN 1 selects the echo image 424 at the center of the unvoiced time series echo image 422 acquired in a unit time from the unvoiced time series echo image 42, and selects the echo image 424. The acoustic feature amount 70 is generated from 424. Then, the NN1 repeats the generation processing of the acoustic feature amount 70 by shifting the start time of the unit time, and synthesizes the plurality of generated acoustic feature amounts 70 to acquire the unvoiced image spectrogram 78.

With this, the NN1 can learn the movement of the mouth corresponding to the smallest unit of pronunciation such as th. Furthermore, the recognition unit 114 can recognize the pronunciation more accurately.

Note that the acoustic features are reduced by processing them with a mel-scale spectrogram, MFCC (mel frequency cepstrum coefficient), short-time FFT (SFFT), and a voice waveform with a neural network (auto encoder). You can use the expressions you have made. The technology related to the auto encoder is disclosed in a paper by Jesse Engel and 6 others ("Neural Audio Synthesis of Musical Notes with WaveNet Autoencoders", URL: https://arxiv.org/abs/1704.01279). . These can be interconverted with acoustic waveforms. For example, a melscale spectrogram can be combined with an acoustic waveform by using a technique called Griffin Lim algorithm. As the acoustic feature amount, other expressions capable of dividing the speech waveform into a vector expression in a short time can be used. In the present embodiment, the number of dimensions of the acoustic feature amount is about 64, but the number of dimensions may be changed according to the sound quality of the reproduced voice.

The NN2 outputs second learning information including third acoustic information generated by inputting a plurality of time-series images at the time of utterance to the NN1 and fourth acoustic information corresponding to the voice at the time of utterance. Obtained by machine learning used. For example, NN2 is obtained by machine learning using a vocalization image spectrogram generated by inputting a vocalization time-series echo image to NN1 and a vocalization spectrogram corresponding to a vocalization as second learning information. As a result, the control unit 110 can convert the unvoiced image spectrogram output from the NN1 via the NN2 into a more accurate spectrogram.

NN2 may convert the unvoiced image spectrogram into a spectrogram of the same length. For example, if the unvoiced image spectrogram is a spectrogram corresponding to the command issued by the user 12 to the smart speaker or the like, the NN 2 converts the unvoiced image spectrogram into a spectrogram of the same length.

Note that a fixed value may be set for the length of the spectrogram input to NN2. However, when the spectrogram having a length shorter than the fixed value is about to be input to the NN2, the control unit 110 inserts the voiceless part having the insufficient length into the spectrogram and then inputs the spectrogram to the NN2. May be. Further, in NN2, since the root mean square error is used as the loss function, NN2 learns so that the input to NN2 matches the output as much as possible.

The purpose of using NN2 is to adjust the unvoiced image spectrogram generated from the unvoiced time series echo image corresponding to the command so as to be closer to the uttered voice spectrogram generated from the voice when the command is actually uttered. is there. In NN1, since only the specified number of unvoiced time-series echo images is input, it is not possible to capture the context in a time width longer than the time width corresponding to the specified number. In NN2, conversion can be performed including the context of the command.

Here, the specific structure of NN2 will be described. FIG. 6 is a diagram showing the structure of the second neural network according to the present embodiment. FIG. 6 shows an example in which an unvoiced image spectrogram 72 having a length of 184 and a dimension number of 64 and having a prescribed length is converted into a high-precision unvoiced image spectrogram 74 of the same length. The first-stage 1D-Convolution Bank 80 is a one-dimensional CNN, but is composed of eight different NNs with filter sizes ranging from 1 to 8. Features with different time widths are extracted by using multiple filter sizes. The features are, for example, phonetic symbol level features, word level features, and the like. The output from the filter is converted by an NN called U-Network, which is a combination of 1D-Convolution and 1D-Deconvolution. The U-Network can recognize global information from the information converted by the Convolution / Deconvolution. However, since local information tends to be lost, U-Network has a structure for guaranteeing the local information.

For example, as shown in FIG. 6, in the U-Network, an acoustic feature amount 802 having a length of 184 and a dimensionality of 128 is set as an acoustic feature amount 804 having a length of 96 and a dimensionality of 256. Next, the U-Network sets the acoustic feature amount 804 as the acoustic feature amount 806 having a length of 46 and a dimension number of 512. Further, the U-Network sets the acoustic feature amount 806 as the acoustic feature amount 808 having a length of 23 and a dimension number of 1024. As a result, the local feature is extracted by increasing the spatial depth instead of decreasing the spatial size.

After extracting local features, U-Network restores the size and number of dimensions of acoustic features in the reverse order of the extraction of local features. At this time, the U-Network also integrates the information in which the input is directly copied to the output into the NN. For example, as shown in FIG. 6, in the U-Network, an acoustic feature amount 808 having a length of 23 and a dimension number of 1024 is set as an acoustic feature amount 810 of a length of 46 and a dimension number of 512, and the acoustic feature amount 806 Is integrated with the copied acoustic feature amount 812. Next, the U-Network integrates the acoustic feature amount 810 integrated with the acoustic feature amount 812 into the acoustic feature amount 814 having a length of 96 and a dimension number of 256, and integrates the acoustic feature amount 804 with the copied acoustic feature amount 816. To do. Further, the U-Network integrates the acoustic feature amount 814 integrated with the acoustic feature amount 816 into an acoustic feature amount 818 having a length 184 and a dimensionality of 128, and integrates the acoustic feature amount 802 with a copied acoustic feature amount 820. To do.

The method using the U-Network described above is a method that is generally used for NN that learns conversion of a two-dimensional image (for example, conversion from a monochrome image to a color image). In the present embodiment, the method is used. Is applied to a one-dimensional acoustic feature quantity sequence.

Note that the number of the second learning information used in the NN2 is the number of combinations of the uttered voice and the utterance time-series echo image created by the user 12 for the learning information. For example, when the user 12 speaks 300 times to create learning information, 300 combinations of input and output are created. However, the amount of 300 may not be enough to train NN2. Therefore, if the amount of the second learning information is not sufficient, data extension (Data Augmentation) may be performed. The data expansion can increase the number of the second learning information by perturbing the input acoustic feature quantity with a random number while fixing the output.

Note that the machine learning for NN1 and NN2 can be performed more effectively by relying on a specific speaker. Therefore, it is desirable that the machine learning be performed depending on a specific speaker. It should be noted that NN1 may be made to depend only on a specific speaker, and NN2 may be made to learn the information of a plurality of speakers all at once, so that complex learning may be performed.

・ Recognition unit 114
The recognition unit 114 has a function of performing recognition processing. For example, the recognition unit 114 accesses the storage unit 120 and performs conversion processing using NN1. Specifically, the recognition unit 114 inputs the unvoiced time series echo image acquired by the ultrasonic echo device 20 input from the communication unit 100 to the NN1. The recognition unit 114 also accesses the storage unit 120 and performs conversion processing using NN2. Specifically, the recognition unit 114 inputs the unvoiced image spectrogram output from NN1 to NN2. The recognition unit 114 also performs a voice recognition process based on the high-accuracy unvoiced image spectrogram output from the NN2. Then, the recognition unit 114 outputs the result of the voice recognition process to the process control unit 116.

Note that the recognition unit 114 may perform voice recognition processing using only NN1. For example, the recognition unit 114 may access the storage unit 120, perform conversion processing using NN1, and perform voice recognition processing based on the unvoiced image spectrogram output from NN1. As described above, in the present embodiment, it is possible to perform the voice recognition process based on the unvoiced image spectrogram output from the NN1. However, the high-accuracy unvoiced image spectrogram output from NN2 is more accurate than the unvoiced image spectrogram output from NN1. Therefore, the recognition unit 114 can perform the voice recognition process with higher accuracy by performing the voice recognition process using not only the NN1 but also the NN2.

・ Processing control unit 116
The processing control unit 116 has a function of controlling the processing in the control unit 110. For example, the process control unit 116 determines the process to be executed based on the result of the voice recognition process by the recognition unit 114. Specifically, when the result of the voice recognition process indicates that the process performed by the control unit 110 by the user 12 is specified by the user 12, the process control unit 116 executes the process specified by the user 12. Further, when the result of the voice recognition process indicates that the question is made by the user 12, the process control unit 116 executes a process of answering the question.

When the process executed by the process control unit 116 is a process of outputting a voice to the user, the process control unit 116 transmits the voice to the voice input / output device 30 worn by the user, and outputs the voice. The input / output device 30 can output sound. As a result, the voiceless speech system 1000 according to the present embodiment can perform voice communication with the user 12 without leaking voice to the outside.

(1-3) Storage unit 120
The storage unit 120 has a function of storing data regarding processing in the mobile terminal 10. For example, the storage unit 120 stores a first neural network 122 and a second neural network 124, which are algorithms generated by machine learning in the control unit 110. The control unit 110 accesses the storage unit 120 and uses the first neural network 122 when converting the unvoiced time-series echo image into the unvoiced image spectrogram. In addition, the control unit 110 accesses the storage unit 120 and uses the second neural network 124 when converting the unvoiced image spectrogram into a high-accuracy unvoiced image spectrogram.

The storage unit 120 may store learning information used by the control unit 110 for machine learning. The data stored in the storage unit 120 is not limited to the above example. For example, the storage unit 120 may store programs such as various applications.

(2) Ultrasonic echo device 20
As shown in FIG. 4, the ultrasonic echo device 20 includes a communication unit 200, a control unit 210, and an echo acquisition unit 220.

(2-1) Communication unit 200
The communication unit 200 has a function of communicating with an external device. For example, the communication unit 200 outputs the information received from the external device to the control unit 210 in the communication with the external device. Specifically, the communication unit 200 outputs information regarding the acquisition of the echo image received from the mobile terminal 10 to the control unit 210.

The communication unit 200 also transmits information input from the control unit 210 to an external device in communication with the external device. Specifically, the communication unit 200 transmits the echo image input from the control unit 210 to the mobile terminal 10.

(2-2) Control unit 210
The control unit 210 has a function of controlling the overall operation of the ultrasonic echo device 20. For example, the control unit 210 controls the echo image acquisition processing by the echo acquisition unit 220. Further, the control unit 210 controls the process of transmitting the echo image acquired by the echo acquisition unit 220 to the mobile terminal 10 by the communication unit 200.

(2-3) Echo acquisition unit 220
The echo acquisition unit 220 has a function of acquiring an echo image. For example, the echo acquisition unit 220 acquires an echo image using the ultrasonic output device provided in the ultrasonic output unit 22. Specifically, the echo acquisition unit 220 causes the ultrasonic wave output device to output ultrasonic waves into the body of the user 12 and acquires an echo image based on the ultrasonic waves reflected by the organs inside the body of the user 12. The echo acquisition unit 220 may acquire an echo image showing the state of the inside of the oral cavity of the user 12 by causing the ultrasonic wave output device to output ultrasonic waves from under the chin of the user 12 toward the inside of the oral cavity of the user 12. it can.

(3) Voice input / output device 30
As shown in FIG. 4, the voice input / output device 30 includes a communication unit 300, a control unit 310, a voice input unit 320, and a voice output unit 330.

(3-1) Communication unit 300
The communication unit 300 has a function of communicating with an external device. For example, the communication unit 300 outputs the information received from the external device to the control unit 310 in the communication with the external device. Specifically, the communication unit 300 outputs the audio data received from the mobile terminal 10 to the control unit 310.

The communication unit 300 also transmits information input from the control unit 310 to the external device in communication with the external device. Specifically, the communication unit 300 transmits the voice data input from the control unit 310 to the mobile terminal 10.

(3-2) Control unit 310
The control unit 310 has a function of controlling the overall operation of the voice input / output device 30. For example, the control unit 310 controls the voice acquisition process by the voice input unit 320. In addition, the control unit 310 controls the process in which the communication unit 300 transmits the voice acquired by the voice input unit 320 to the mobile terminal 10. The control unit 310 also controls the audio output processing by the audio output unit 330. For example, the communication unit 300 causes the audio output unit 330 to output the audio data received from the mobile terminal 10 as audio.

(3-3) Voice input unit 320
The voice input unit 320 has a function of acquiring a voice generated outside. The voice input unit 320 acquires, for example, a voiced voice that is a voice when the user 12 speaks. Then, the voice input unit 320 outputs the acquired vocalized voice to the control unit 310. The voice input unit 320 can be realized by, for example, a microphone.

(3-4) Audio output unit 330
The voice output unit 330 has a function of outputting a voice received from an external device. The voice output unit 330 receives, for example, voice data generated based on the result of the voice recognition process in the mobile terminal 10 from the control unit 310, and outputs a voice corresponding to the input voice data. The audio output unit 330 can be realized by, for example, a speaker.

<1.4. Processing of voiceless speech system>
The functions of the voiceless speech system 1000 according to the present embodiment have been described above. Next, the processing of the voiceless speech system 1000 will be described.

(1) Machine Learning Flow for Acquiring First Neural Network FIG. 7 is a flowchart showing the flow of machine learning for acquiring the first neural network according to the present embodiment. First, the mobile terminal 10 acquires the vocalization time series echo image from the ultrasonic echo device 20 as learning information (S100). Further, the mobile terminal 10 acquires the vocalized voice as the learning information from the voice input / output device 30 (S102). Next, the mobile terminal 10 performs machine learning using the acquired learning information (S104). Then, the mobile terminal 10 sets the algorithm generated by the machine learning to NN1 (S106).

(2) Machine Learning Flow for Acquiring Second Neural Network FIG. 8 is a flowchart showing the flow of machine learning for acquiring the second neural network according to the present embodiment. First, the mobile terminal 10 inputs the utterance time series echo image to the NN1 (S200). Next, the mobile terminal 10 acquires the vocal image spectrogram output from the NN1 as learning information (S202). In addition, the mobile terminal 10 acquires a uttered voice spectrogram from the uttered voice as learning information (S204). Next, the mobile terminal 10 performs machine learning using the acquired learning information (S206). Then, the mobile terminal 10 sets the algorithm generated by the machine learning to NN2 (S208).

(3) Processing in mobile terminal 10 FIG. 9 is a flowchart showing a flow of processing in the mobile terminal according to the present embodiment. First, the mobile terminal 10 acquires an unvoiced time series echo image (S300). Next, the mobile terminal 10 inputs the acquired unvoiced time-series echo image to the NN1 and generates a plurality of audio feature amounts from the unvoiced time-series echo image (S302). Next, the mobile terminal 10 synthesizes the plurality of generated voice feature amounts in time series to generate an unvoiced image spectrogram (S304).

After generating the unvoiced image spectrogram from the unvoiced time-series echo image, the mobile terminal 10 inputs the generated unvoiced image spectrogram into the NN2 and converts the unvoiced image spectrogram into a high-precision unvoiced image spectrogram (S306). After conversion, the mobile terminal 10 recognizes the content indicated by the high-accuracy unvoiced image spectrogram in the recognition unit 114 (S308). Then, the mobile terminal 10 executes a process based on the content recognized by the recognition unit 114 (S310).

<< 2. Modification >>
The embodiments of the present disclosure have been described above. Subsequently, a modified example of the embodiment of the present disclosure will be described. Note that the modified examples described below may be applied to the embodiment of the present disclosure alone, or may be applied to the embodiment of the present disclosure in combination. In addition, the modified example may be applied instead of the configuration described in the embodiment of the present disclosure, or may be additionally applied to the configuration described in the embodiment of the present disclosure.

In the above-described embodiment, an example in which the high-precision unvoiced image spectrogram converted by NN2 is output to the recognition unit 114 of the mobile terminal 10 has been described. However, the high-precision unvoiced image spectrogram is converted into a voice waveform. Then, it may be output as voice from a voice output device such as a speaker. Thus, the user 12 can control the information device with the voice input function such as the smart speaker via the voice output device.

The high-accuracy unvoiced image spectrogram may be output to an external voice recognition device instead of the recognition unit 114 of the mobile terminal 10. For example, the high-accuracy unvoiced image spectrogram may be input to the voice recognition unit of the smart speaker via communication. Accordingly, the user 12 can control the information device with the voice input function such as the smart speaker without causing the mobile terminal 10 to emit the sound wave into the air.

<< 3. Application example ＞＞
The modification of the embodiment of the present disclosure has been described above. Next, an application example of the voiceless speech system 1000 according to the embodiment of the present disclosure will be described.

<3.1. First application example>
First, a first application example according to this embodiment will be described. The voiceless speech system 1000 according to the present embodiment can be applied to, for example, training for a speaker to move his mouth or tongue without speaking. For example, the voiceless speech system 1000 visually feeds back the content recognized from the voiceless time series echo image acquired by the ultrasonic echo device 20 to the speaker. Thereby, the speaker can improve the way of moving the mouth and tongue based on the feedback. Specifically, the voiceless utterance system 1000 displays the voiceless time-series echo images on the display device or the like, so that the speaker can confirm the displayed image and learn how to move the mouth or tongue. . Further, by feeding back the content recognized by the voiceless utterance system 1000 from the voiceless time-series echo images by voice, the speaker recognizes how the voiceless utterance system 1000 moves by moving his mouth or tongue. You can learn what will be done. The content recognized by the voiceless speech system 1000 may be fed back as text.

<3.2. Second application example>
Next, a second application example according to this embodiment will be described. The voiceless speech system 1000 according to the present embodiment can be applied as a vocalization assisting device for a person with a defective vocal cord or a deaf person. In recent years, there has been proposed a technique related to a technique in which a button-controllable oscillator is pressed against the pharynx to substitute the vocal cords for a person who has lost the function of the vocal cords. With this technique, a person who has lost the function of the vocal cord can utter a voice without vibrating the vocal cord. However, in this technique, since the vibrator emits a loud sound, it may happen that the sound of the utterance that has passed through the oral cavity is disturbed. Further, it is difficult for the speaker to adjust the volume of the loud sound, and the loud sound may be an unpleasant sound for the speaker. On the other hand, in the voiceless speech system 1000 according to the present embodiment, the information acquired by the ultrasonic echo is converted into acoustic information, and the acoustic information is uttered as a voice waveform. No offensive sound. In addition, the speaker can also adjust the volume of the voice generated from the voiceless speech system 1000. Therefore, even a person who has lost the function of the vocal cords can more comfortably use the voiceless speech system 1000 according to the present embodiment.

Also, a person with a defective vocal cord cannot make a voice, but can move the mouth or tongue to change the condition inside the oral cavity. Therefore, the voiceless system 1000 recognizes the state of the oral cavity of a person with a defective vocal cord, and outputs the recognized content as a voice from the speaker, so that even if the person with a defective vocal cord is used, And can communicate by voice. The voiceless speech system 1000 according to the present embodiment is effective even for a person who does not have sufficient vital capacity to vibrate the vocal cords, such as an elderly person, regardless of the person who has a vocal cord defect. To do. For example, in the case of an elderly person who cannot speak with a sufficient amount of voice, it may be difficult for the elderly person to speak, but the elderly person can have speaking ability by the voiceless speech system 1000 and can easily carry out the conversation. It will be possible.

Although a hearing-impaired person can make a voice, it is difficult to confirm whether the voice made by himself or herself is accurately transmitted to others. Therefore, by utilizing the feedback of the voiceless speech system 1000 of the present embodiment described in the first application example, the deaf person can confirm how he / she is making a voice. Further, with the voiceless speech system 1000, the state of the oral cavity can be confirmed, so that the deaf person can practice speaking while confirming how to move the mouth and tongue.

<3.3. Third application example>
Then, the 3rd application example concerning this embodiment is explained. The voiceless speech system 1000 according to the present embodiment can be applied to expand the functions of a hearing aid. By installing the voiceless speech system 1000 in a hearing aid, the convenience of the user of the hearing aid can be improved.

<< 4. Hardware configuration example >>
Finally, with reference to FIG. 10, a hardware configuration example of the information processing apparatus according to the present embodiment will be described. FIG. 10 is a block diagram showing a hardware configuration example of the information processing apparatus according to the present embodiment. The information processing apparatus 900 shown in FIG. 10 can realize, for example, the mobile terminal 10, the ultrasonic echo device 20, and the voice input / output device 30 shown in FIGS. 1 and 4, respectively. Information processing by the mobile terminal 10, the ultrasonic echo device 20, and the voice input / output device 30 according to the present embodiment is realized by cooperation of software and hardware described below.

As shown in FIG. 10, the information processing apparatus 900 includes a CPU (Central Processing Unit) 901, a ROM (Read Only Memory) 902, and a RAM (Random Access Memory) 903. The information processing apparatus 900 also includes a host bus 904a, a bridge 904, an external bus 904b, an interface 905, an input device 906, an output device 907, a storage device 908, a drive 909, a connection port 910, and a communication device 911. The hardware configuration shown here is an example, and some of the components may be omitted. The hardware configuration may further include components other than the components shown here.

The CPU 901 functions as, for example, an arithmetic processing device or a control device, and controls the overall operation of each component or a part thereof based on various programs recorded in the ROM 902, the RAM 903, or the storage device 908. The ROM 902 is a means for storing a program read by the CPU 901, data used for calculation, and the like. The RAM 903 temporarily or permanently stores, for example, a program read by the CPU 901 and various parameters that change appropriately when the program is executed. These are connected to each other by a host bus 904a including a CPU bus and the like. The CPU 901, the ROM 902, and the RAM 903 can realize the functions of the control unit 110, the control unit 210, and the control unit 310 described with reference to FIG. 4, for example, in cooperation with software.

The CPU 901, the ROM 902, and the RAM 903 are mutually connected, for example, via a host bus 904a capable of high-speed data transmission. On the other hand, the host bus 904a is connected to the external bus 904b having a relatively low data transmission rate via the bridge 904, for example. The external bus 904b is also connected to various components via the interface 905.

The input device 906 is realized by a device such as a mouse, a keyboard, a touch panel, a button, a microphone, a switch, and a lever, to which information is input by the user. Further, the input device 906 may be, for example, a remote control device that uses infrared rays or other radio waves, or may be an externally connected device such as a mobile phone or PDA that corresponds to the operation of the information processing device 900. . Further, the input device 906 may include, for example, an input control circuit that generates an input signal based on the information input by the user using the above-described input means and outputs the input signal to the CPU 901. By operating the input device 906, the user of the information processing device 900 can input various data to the information processing device 900 and instruct a processing operation.

Besides, the input device 906 may be formed by a device that detects information about the user. For example, the input device 906 includes an image sensor (for example, a camera), a depth sensor (for example, a stereo camera), an acceleration sensor, a gyro sensor, a geomagnetic sensor, an optical sensor, a sound sensor, and a distance measuring sensor (for example, ToF (Time of Flight). ) Sensors), force sensors and the like. Further, the input device 906 includes information about the state of the information processing device 900, such as the posture and moving speed of the information processing device 900, information about the surrounding environment of the information processing device 900, such as brightness and noise around the information processing device 900. May be obtained. Further, the input device 906 receives a GNSS signal from a GNSS (Global Navigation Satellite System) satellite (for example, a GPS signal from a GPS (Global Positioning System) satellite) and receives position information including latitude, longitude, and altitude of the device. It may include a GNSS module to measure. Regarding the position information, the input device 906 may detect the position by transmission / reception with Wi-Fi (registered trademark), a mobile phone / PHS / smartphone, or the like, short-range communication, or the like. The input device 906 can realize the functions of the echo acquisition unit 220 and the voice input unit 320 described with reference to FIG. 4, for example.

The output device 907 is formed of a device capable of visually or audibly notifying the user of the acquired information. Such devices include CRT display devices, liquid crystal display devices, plasma display devices, EL display devices, display devices such as laser projectors, LED projectors and lamps, audio output devices such as speakers and headphones, and printer devices. . The output device 907 outputs results obtained by various processes performed by the information processing device 900, for example. Specifically, the display device visually displays the results obtained by the various processes performed by the information processing device 900 in various formats such as text, images, tables, and graphs. On the other hand, the audio output device converts an audio signal composed of reproduced audio data, acoustic data, and the like into an analog signal and outputs it audibly. The output device 907 can realize the function of the audio output unit 330 described with reference to FIG. 4, for example.

The storage device 908 is a device for data storage formed as an example of a storage unit of the information processing device 900. The storage device 908 is realized by, for example, a magnetic storage device such as an HDD, a semiconductor storage device, an optical storage device, a magneto-optical storage device, or the like. The storage device 908 may include a storage medium, a recording device that records data in the storage medium, a reading device that reads data from the storage medium, a deletion device that deletes data recorded in the storage medium, and the like. The storage device 908 stores programs executed by the CPU 901, various data, various data acquired from the outside, and the like. The storage device 908 can realize the function of the storage unit 120 described with reference to FIG. 4, for example.

The drive 909 is a reader / writer for a storage medium, and is built in or externally attached to the information processing device 900. The drive 909 reads out information recorded on a removable storage medium such as a mounted magnetic disk, optical disk, magneto-optical disk, or semiconductor memory, and outputs it to the RAM 903. The drive 909 can also write information in a removable storage medium.

The connection port 910 is, for example, a USB (Universal Serial Bus) port, an IEEE 1394 port, a SCSI (Small Computer System Interface), an RS-232C port, or a port for connecting an external device such as an optical audio terminal. .

The communication device 911 is, for example, a communication interface formed of a communication device or the like for connecting to the network 920. The communication device 911 is, for example, a communication card for wired or wireless LAN (Local Area Network), LTE (Long Term Evolution), Bluetooth (registered trademark) or WUSB (Wireless USB). The communication device 911 may be a router for optical communication, a router for ADSL (Asymmetric Digital Subscriber Line), a modem for various kinds of communication, or the like. The communication device 911 can send and receive signals and the like to and from the Internet and other communication devices, for example, according to a predetermined protocol such as TCP / IP. The communication device 911 can realize, for example, the functions of the communication unit 100, the communication unit 200, and the communication unit 300 described with reference to FIG.

The network 920 is a wired or wireless transmission path for information transmitted from a device connected to the network 920. For example, the network 920 may include a public line network such as the Internet, a telephone line network, and a satellite communication network, various LANs (Local Area Network) including Ethernet (registered trademark), WAN (Wide Area Network), and the like. Further, the network 920 may include a dedicated line network such as an IP-VPN (Internet Protocol-Virtual Private Network).

Above, an example of the hardware configuration capable of realizing the functions of the information processing apparatus 900 according to the present embodiment has been shown. Each component described above may be realized by using a general-purpose member, or may be realized by hardware specialized for the function of each component. Therefore, it is possible to appropriately change the hardware configuration to be used according to the technical level at the time of implementing the present embodiment.

<< 5. Summary >>
As described above, the mobile terminal 10 according to the present embodiment sets a plurality of time-series images showing the intraoral state acquired by ultrasonic echo to the intraoral state based on the algorithm acquired by machine learning. Convert to the corresponding information. Thereby, the mobile terminal 10 can convert an image showing the state in the oral cavity when the user intentionally moves at least one of the mouth and the tongue into acoustic information without uttering a voice.

Therefore, it is possible to provide a new and improved information processing apparatus and information processing method capable of obtaining the intended acoustic information without the user speaking.

The preferred embodiments of the present disclosure have been described in detail above with reference to the accompanying drawings, but the technical scope of the present disclosure is not limited to such examples. It is obvious that a person having ordinary knowledge in the technical field of the present disclosure can come up with various changes or modifications within the scope of the technical idea described in the claims. Of course, it is understood that the invention also belongs to the technical scope of the present disclosure.

For example, each device described in this specification may be realized as a single device, or part or all may be realized as separate devices. For example, the mobile terminal 10, the ultrasonic echo device 20, and the voice input / output device 30 illustrated in FIG. 1 may be realized as independent devices. Further, for example, the mobile terminal 10 may be realized as a server device connected to the ultrasonic echo device 20 and the voice input / output device 30 via a network or the like. Further, the function of the control unit 110 included in the mobile terminal 10 may be included in a server device connected via a network or the like.

Also, the series of processes performed by each device described in this specification may be realized using any of software, hardware, and a combination of software and hardware. The program forming the software is stored in advance in a recording medium (non-transitory medium: non-transmission media) provided inside or outside each device, for example. Then, each program is read into the RAM when it is executed by a computer, and executed by a processor such as a CPU.

Also, the processes described using the flowcharts in this specification do not necessarily have to be executed in the illustrated order. Some processing steps may be performed in parallel. In addition, additional processing steps may be adopted, and some processing steps may be omitted.

Also, the effects described in the present specification are merely explanatory or exemplifying ones, and are not limiting. That is, the technique according to the present disclosure may have other effects that are apparent to those skilled in the art from the description of the present specification, in addition to or instead of the above effects.

Note that the following configurations also belong to the technical scope of the present disclosure.
(1)
An information processing apparatus, comprising: a plurality of time-series images showing the intraoral state acquired by ultrasonic echo, and a control unit that converts the information corresponding to the intraoral state based on an algorithm acquired by machine learning. .
(2)
The algorithm has a first neural network,
The control unit converts a plurality of input time-series images when no voice is input into first acoustic information via the first neural network,
The information processing device according to (1).
(3)
The first neural network generates a plurality of acoustic feature amounts per unit time from a plurality of input time-series images when no voice is input, and synthesizes the generated plurality of acoustic feature amounts in time series order. The information processing apparatus according to (2), wherein the first acoustic information is generated by.
(4)
The first neural network selects a time-series image at a central time of the unit time from the plurality of time-series images of the unvoiced time acquired in the unit time, and selects the selected time-series image. The information processing apparatus according to (3), wherein the acoustic feature amount per unit time is generated from the.
(5)
The first neural network is obtained by the machine learning using the first learning information including a voice at the time of utterance and a plurality of time-series images at the time of utterance, (2) to (4) above. The information processing apparatus according to any one of 1.
(6)
The algorithm further comprises a second neural network,
The control unit converts the first acoustic information into second acoustic information corresponding to a voice when uttered, via the second neural network.
The information processing apparatus according to any one of (2) to (5) above.
(7)
The second neural network includes third acoustic information generated by inputting the plurality of time-series images at the time of utterance to the first neural network, and a fourth acoustic information corresponding to voice at the time of utterance. The information processing device according to (6), which is obtained by the machine learning using second learning information including acoustic information.
(8)
The information processing device according to any one of (2) to (7), wherein the acoustic information is a spectrogram.
(9)
The plurality of time-series images show changes in the state of the oral cavity when the user moves at least one of the mouth and tongue without uttering, and the time-series images are described in any one of (1) to (8) above. The information processing device described.
(10)
The information processing device according to any one of (1) to (9), wherein the machine learning is performed by deep learning.
(11)
The information processing device according to any one of (1) to (10), wherein the machine learning is performed using a convolutional neural network.
(12)
A plurality of time-series images showing the intraoral state acquired by ultrasonic echo, including conversion into information corresponding to the intraoral state based on an algorithm acquired by machine learning, including, executed by the processor. Information processing method.

10 Mobile Terminal 20 Ultrasonic Echo Device 30 Voice Input / Output Device 100 Communication Unit 110 Control Unit 112 Machine Learning Unit 114 Recognition Unit 116 Processing Control Unit 120 Storage Unit 122 First Neural Network 124 Second Neural Network 200 Communication Unit 210 Control Part 220 Echo acquisition part 300 Communication part 310 Control part 320 Voice input part 330 Voice output part 1000 Voiceless speech system

Claims

An information processing apparatus, comprising: a plurality of time-series images showing the intraoral state acquired by ultrasonic echo, and a control unit that converts the information corresponding to the intraoral state based on an algorithm acquired by machine learning. .
The algorithm has a first neural network,
The control unit converts a plurality of input time-series images when no voice is input into first acoustic information via the first neural network,
The information processing apparatus according to claim 1.
The first neural network generates a plurality of acoustic feature quantities per unit time from a plurality of input time-series images when no voice is input, and synthesizes the generated plurality of acoustic feature quantities in chronological order. The information processing apparatus according to claim 2, wherein the first acoustic information is generated by.
The first neural network selects a time-series image at a central time of the unit time from the plurality of time-series images of the unvoiced time acquired in the unit time, and selects the selected time-series image. The information processing apparatus according to claim 3, wherein the acoustic feature quantity per unit time is generated from the.
The information processing according to claim 2, wherein the first neural network is obtained by the machine learning using the first learning information including a voice at the time of utterance and a plurality of time-series images at the time of the utterance. apparatus.
The algorithm further comprises a second neural network,
The control unit converts the first acoustic information into second acoustic information corresponding to a voice when uttered, via the second neural network.
The information processing apparatus according to claim 2.
The second neural network includes third acoustic information generated by inputting the plurality of time-series images at the time of utterance to the first neural network, and a fourth acoustic information corresponding to voice at the time of utterance. The information processing device according to claim 6, which is obtained by the machine learning using the second learning information including acoustic information.
The information processing apparatus according to claim 2, wherein the acoustic information is a spectrogram.
The information processing apparatus according to claim 1, wherein the plurality of time-series images show changes in the state of the oral cavity when the user moves at least one of the mouth and tongue without uttering.
The information processing apparatus according to claim 1, wherein the machine learning is performed by deep learning.
The information processing apparatus according to claim 1, wherein the machine learning is performed using a convolutional neural network.
A plurality of time-series images showing the intraoral state acquired by ultrasonic echo, including conversion into information corresponding to the intraoral state based on an algorithm acquired by machine learning, including, executed by the processor. Information processing method.