JP2023112556A - Visualization device, visualization method and program - Google Patents

Abstract

To provide a visualization device, a visualization method and a program that contribute to smooth communication even if mouths are hidden behind masks etc., during a conversation.SOLUTION: There is provided a visualization device that has: a speech acquisition part which acquires a voice; a voice recognition part which recognizes the voice to generate recognition data; a visualization part which generates visualization data by visualizing the recognition data; and a presentation part which presents the visualization data in the form of a video.SELECTED DRAWING: Figure 1

Description

The present invention relates to a visualization device, a visualization method, and a program.

People with hearing disabilities may read the movement of the other person's mouth to assist in conversation, but as more people wear masks to prevent disease, etc., it is not possible to read the movement of the mouth. communication may be difficult.

Patent Document 1 discloses an information processing apparatus as follows. When the first actor is a voice user and the second actor is a sign language user, the device converts a message spoken by the first actor into a sign language gesture corresponding to the message. An invention is described in which the motion is converted into a moving image of a hand to be performed, and the moving image is superimposed on the first action subject appearing on a transmissive display.

WO2019/003616

In addition, each disclosure of the above prior art documents is incorporated into this document by reference. The following analysis was made by the inventors.

However, there are many cases in which even people with hearing impairments cannot communicate by sign language, and there is a desire to communicate with a few voices and the movement of the other party's mouth.

SUMMARY OF THE INVENTION It is an object of the present invention to provide a visualization device, a visualization method, and a program that enable smooth communication even when the mouth is hidden by a mask or the like during conversation.

According to a first aspect of the present invention or disclosure, a voice acquisition unit that acquires voice, a voice recognition unit that recognizes the voice and uses it as recognition data, and a visualization data that visualizes the recognition data is generated. A visualization device is provided that has a visualization unit and a presentation unit that presents the visualization data as a video.

According to a second aspect of the present invention or disclosure, a step of acquiring speech, a step of recognizing the speech to obtain recognition data, a step of generating visualization data by visualizing the recognition data, and a step of visualizing the recognition data; and presenting visualization data.

According to a third aspect of the present invention or the disclosure, a process of acquiring speech, a process of recognizing the speech to obtain recognition data, a process of generating visualization data obtained by visualizing the recognition data, and A program is provided for causing a computer to perform a process of presenting visualization data.

According to each aspect of the present invention and the disclosure, there is provided a visualization device, a visualization method, and a program that contribute to smooth communication even when the mouth is hidden by a mask or the like during conversation.

It is a block diagram showing an example of a configuration of a visualization device according to an embodiment. 1 is a block diagram showing an example of the configuration of a visualization device according to a first embodiment; FIG. FIG. 2 is a schematic diagram showing an outline of processing in the visualization device in the first embodiment; FIG. 4 is a flow chart showing an example of the operation of the visualization device according to the first embodiment; 1 is a schematic diagram showing the hardware configuration of a visualization device according to a first embodiment; FIG. FIG. 11 is a schematic diagram for showing processing of the auxiliary instrument A corresponding to the visualization device according to the second embodiment; FIG. 11 is a schematic diagram showing the configuration of an auxiliary instrument A corresponding to the visualization device according to the second embodiment; FIG. 11 is another schematic diagram for showing the processing of the auxiliary instrument A corresponding to the visualization device according to the second embodiment; 9 is a flowchart for showing details of the operation of a speech analysis unit according to the second embodiment; 9 is a flowchart for showing details of the operation of the screen display unit according to the second embodiment;

First, an overview of one embodiment will be described. It should be noted that the drawing reference numerals added to this outline are added to each element for convenience as an example to aid understanding, and the description of this outline does not intend any limitation. Also, connecting lines between blocks in each figure include both bi-directional and uni-directional. The unidirectional arrows schematically show the flow of main signals (data) and do not exclude bidirectionality. Furthermore, in the circuit diagrams, block diagrams, internal configuration diagrams, connection diagrams, etc. disclosed in the present application, an input port and an output port exist at the input end and the output end of each connection line, respectively, although not explicitly shown. The input/output interface is the same.

FIG. 1 is a block diagram showing an example of the configuration of a visualization device according to one embodiment. As shown in this figure, the visualization device 10 according to one embodiment has a speech acquisition unit 11, a speech recognition unit 12, a visualization unit 13, and a presentation unit .

The voice acquisition unit 11 acquires voice. "Voice" refers to the voice at the time of speaking, but the speaker does not necessarily have to be in front of the speaker. For example, "speech" includes the voice of an announcement at a station, in a train, or in a disaster prevention radio system. "Acquisition" also includes a form of acquiring voice data via a network or a storage medium by temporarily storing it in a storage area as voice data using an input device such as a microphone. . The format of the audio data is not particularly limited, but it takes time to develop the data and the response performance deteriorates in order to execute the visualization process while recognizing it immediately during a conversation. desirable. The stored voice data is sent to the voice recognition unit 12, which will be described later.

The voice recognition unit 12 recognizes the voice and uses it as recognition data. Speech data is input to a speech recognition program and recognition data is output. The recognition method of the speech recognition program is not particularly limited, but since there is a high need for immediate processing during conversation, a method that can process at high speed and that can simultaneously perform recognition processing while inputting speech data is desirable. . "Recognition data" generally refers to speech data output to text data in many cases, but it does not necessarily have to be text data. For example, speech data to phoneme data may be output. Furthermore, for speeding up, only vowels in phonemes may be recognized and output as data.

Further, for example, waveform data of speech may be associated with the shape of the mouth, which is an element of visualization data at the time of utterance, and the shape of the mouth may be output according to the degree of similarity between the waveform data and the speech data. By directly associating the language-independent acoustic model with the visualization data in this way, it is possible to visualize not only Japanese but also foreign language speech.

The visualization unit 13 generates visualization data by visualizing the recognition data. "Visualization" refers to converting speech features into visually recognizable forms. Examples include text, gestures such as sign language, mouth movements, spatial patterns and their movements, It also includes a process of converting to a form with a high degree of abstraction, such as color. "Generation" may mean, for example, that recognition data is associated with visualization data prepared in advance such as the shape of the mouth, or visualization data that is immediately generated according to recognition data. good. The generated visualization data is sent to the presenter.

The presentation unit 14 presents the visualization data as an image. The generated visualization data is output from an output interface such as a display device. The mode of presentation is not limited to one mode, and can be based on various methods as long as it is a visually recognizable mode.

Specific embodiments will be described in more detail below with reference to the drawings. In addition, the same code|symbol is attached|subjected to the same component in each embodiment, and the description is abbreviate|omitted.

[First embodiment]
FIG. 2 is a block diagram showing an example of the configuration of the visualization device of this embodiment. The visualization device 10 of this embodiment has the following configuration. That is, as shown in FIG. 1, the visualization device 10 according to the first embodiment includes a speech acquisition unit 11, a speech recognition unit 12, a visualization unit 13, a presentation unit 14, and have The visualization device 10 of this embodiment newly has a body recognition unit 15 in addition to the above components.

The visualization unit 13 of the present embodiment may generate visualization data that is an image representing the shape of the mouth when the recognition data recognized by the speech recognition unit 12 is uttered. FIGS. 3A to 3F are schematic diagrams showing an outline of processing in the visualization device 10 of this embodiment. The “image representing the shape of the mouth” refers to the image within the dotted line portion in FIG. 3(B). An image in which the mouth is not hidden as shown in FIG. 3(A) is shot in advance, a cropping range including the mouth is determined as shown in FIG. 3(B), and a cropped image as shown in FIG. 3(C) is generated. . At this time, the size such as the width and height of the cutout range and the position of the cutout range, for example, the relative position from both eyes, are acquired and stored in a storage area in association with the image.

Next, an image of the person wearing the mask as shown in FIG. 3(D) is shot, and the image of the cutout range shown in FIG. 3(C) is superimposed on the image. At this time, as shown in FIG. 3(E), the position of superimposition is calculated from the relative position from both eyes, for example. At this time, if the angle of view of the image differs between FIG. 3B and FIG. 3E, the image to be superimposed in FIG. 3C is enlarged or reduced to adjust the size. FIG. 3F is an example of a video presented by the presentation unit 14 .

Since the image of the cut-out range to be superimposed in FIG. 3(C) differs depending on the recognized speech, images representing the shape of the mouth corresponding to the pre-recognized speech may be collected. In addition, since it takes time and effort to shoot images of the cut-out range for all sounds, at least 1 or more images are shot that are less than the types of recognized sounds, and are deformed by image processing to capture all sounds. An image representing the shape of the mouth corresponding to the recognized speech may be generated.

The presentation unit 14 may present the visualization data on a transmissive display device, such as a transparent display device. For example, there is a mode in which the image shown in FIG. 3(C) is aligned with the actual speaker seen through the transmissive display device, and the image shown in FIG. 3(F) is presented. may In this case, preprocessing by the body recognition unit 15, which will be described later, is required.

A body recognition unit 15 generates body recognition data by recognizing body parts of a conversation partner. "Body recognition data" refers to data obtained by recognizing body parts such as eyes, mouth, nose, etc. based on their shapes and determining the positions of the recognized parts on the screen.

The body recognition unit 15 further generates body recognition data that recognizes the position of the mouth of the conversation partner, and the presentation unit 14 displays an image representing the mouth shape of the generated visualization data according to the generated body recognition data. may be used to place it at the position of the mouth of the conversation partner. For example, in the case of superimposing and presenting an image showing the shape of a mouth as shown in FIG. It is possible to position the image representing the shape of the mouth by recognizing and finding the position of the mouth and using it as a reference.

The speech recognition unit 12 may further acquire emotion data obtained by estimating the emotion of the conversation partner from the voice, and the visualization unit 13 may generate visualization data based on the acquired emotion data. For example, by comparing the loudness and intonation of the voice input during speech recognition with a normal acoustic model, emotions such as emotions can be estimated based on specific differences in feature values stored in advance. good. For example, the visualization unit transforms the image representing the shape of the mouth shown in FIG. good too. For example, when an emotion of “joy” is estimated, an image with the corners of the mouth raised may be generated by image processing.

It should be noted that the speech recognition unit 12 may further recognize the speech to generate recognition data in text format, and the presentation unit 14 may present the recognition data in text format. For example, the presentation unit 14 may display the recognition data of the text on a part of the screen of the transmissive display device, and display the recognized text like a caption while looking at the speaker through the display device. .

[Explanation of operation]
An example of the operation of the visualization device 10 of this embodiment will be described with reference to FIG. FIG. 4 is a flow chart showing an example of the operation of the visualization device 10 according to the first embodiment.

When the device starts to operate, a device such as a camera scans an image of the speaker and generates body recognition data (step S41). Note that this process may be executed only once after starting the operation of the visualization device (step S46, solid line in the case of N), and if the position of the mouth changes within the screen due to the movement of the speaker, , may be repeatedly executed during busy hours (step S46, dotted line in case of N). Next, voice is acquired by a device such as a microphone (step S42). Next, the acquired voice is recognized and used as recognition data (step S43). Visualization data is generated using the recognition data (step S44). Next, the visualization data is presented as an image based on the body recognition data (step S45). After that, it is judged whether or not it is finished (step S46). If it is finished (step S46, Y), the operation is finished. ), or return to the process of generating body recognition data (step S41).

The step of generating the body recognition data (step S41) may be performed by the step of presenting the visualization data as an image (step S45), and may be performed in parallel with other processes.

[Hardware configuration]
The visualization device 10 of this embodiment can be configured by an information processing device (computer), and has the configuration illustrated in FIG. For example, the visualization device 10 includes a CPU (Central Processing Unit) 51 , a memory 52 , an input/output interface 53 and a communication means such as a NIC (Network Interface Card) 54 , which are interconnected by an internal bus 55 .

However, the configuration shown in FIG. 5 is not meant to limit the hardware configuration of the visualization device 10 . Visualization device 10 may include hardware not shown. Also, the number of CPUs and the like included in the visualization device 10 is not limited to the example shown in FIG.

The memory 52 is a RAM (Random Access Memory), a ROM (Read Only Memory), an auxiliary storage device (hard disk, etc.).

The input/output interface 53 is a means serving as an interface for a display device and an input device (not shown). The display device is, for example, a transparent display, which is a transmissive display device, smart glasses equipped with the same, or the like. The input device is, for example, a device for receiving user operations such as a keyboard, a mouse, a sensor for receiving gesture input, or a device for acquiring target video or audio such as a microphone or camera.

The functions of the visualization device 10 include programs (processing modules) such as a speech acquisition program, a speech recognition program, a visualization program, a presentation program, and a body recognition program stored in the memory 52; , data such as visualization data for presenting the shape of the mouth. The processing module is implemented by the CPU 51 executing each program stored in the memory 52, for example. Also, the program can be downloaded via a network or updated using a storage medium storing the program. Furthermore, the processing module may be realized by a semiconductor chip. In other words, it is sufficient if there is means for executing the functions performed by the processing module by some kind of hardware and/or software.

[Hardware operation]
When the visualization device 10 starts to operate, the body recognition program is called from the memory 52 and executed by the CPU 51 . The program controls the camera, acquires images of the speaker's body, and acquires, for example, the reference position of the eyes and the relative positions of the mouth and nose as body recognition data through shape recognition. The acquired data is temporarily stored in the memory 52 .

Next, the voice acquisition program is called from the memory 52 and executed by the CPU 51 . The program controls the microphone and captures the speaker's voice. The acquired voice is temporarily stored in the memory 52 as voice data.

Next, the speech recognition program is called from the memory 52 and executed by the CPU 51 . The program reads voice data temporarily stored in the memory 52 and performs matching processing with the feature values of the acoustic model stored in the memory 52, for example. The speech recognition program can employ various existing speech recognition methods. The recognition result may be, for example, phoneme-by-phoneme or word-level using a language model. The recognition data is temporarily stored in memory 52 .

The visualization program is then called from memory 52 and is ready for execution on CPU 51 . The program reads the recognition data temporarily stored in the memory 52, acquires an image representing the shape of the mouth at the time of speaking associated with the recognition data, and sends it to the presentation program described later. Alternatively, an image representing the base shape of the mouth is deformed by image processing to generate an image, which is visualization data corresponding to the recognition result, and this is sent to the presentation program.

Next, the presentation program is called from the memory 52 and executed by the CPU 51 . The program reads visualization data from a visualization program and outputs it to a display device, such as a transparent display. Here, the same program reads the body recognition data temporarily stored in the memory 52 and executes the positioning process for displaying the visualization data. Specifically, for example, the position of the speaker's eyes reflected on the transparent display is used as a reference, and the visualization data is placed at a position separated by the relative distance from the eye position included in the body recognition data. . As a result, the position of the actual speaker's mouth that can be seen through the transparent display and the position of the visualization data as shown in FIG. It is possible to reflect the speaker on the

[Explanation of effect]
The visualization device according to the first embodiment enables smooth communication even when the mouth is hidden by a mask or the like during conversation. In addition, when the visualization data is superimposed on the image of the real speaker reflected on the transparent display, even if the speaker moves during conversation, it is possible to present mouth movements in a natural way.

[Second embodiment]
In the present embodiment, a device (auxiliary device A, corresponding to the visualization device 10) that superimposes a mouth-shaped image on the appearance of a speaker reflected on smart glasses, which is a transmissive display device, will be described. FIG. 6 is a schematic diagram showing the processing of the auxiliary instrument A corresponding to the visualization device of this embodiment. As shown in this figure, the auxiliary device A has smart glasses 601 having a transmissive display, a microphone 602, a voice analysis function, and a screen display function on the display. The smart glasses 601 are worn by the listener 603, and the voice of the speaker 604 picked up by the microphone 602 is decomposed and analyzed into single sounds by the voice analysis unit. 601 displays.

The auxiliary tool A of this embodiment has the following configuration. FIG. 7 is a schematic diagram showing the configuration of an auxiliary instrument A corresponding to the visualization device of this embodiment. FIG. 8 is another schematic diagram for showing the processing of the auxiliary tool A corresponding to the visualization device according to the second embodiment. That is, as shown in FIG. 7, for example, a glass-type wearable device includes a microphone 701 (corresponding to the voice acquisition unit 11), a transmissive display 702 (corresponding to the presentation unit 14), a voice analysis unit 703 (corresponding to the voice recognition unit 12). corresponding), and a screen display unit 704 (corresponding to the visualization unit 13 and presentation unit 14). The voice analysis unit 703 analyzes the voice input from the microphone and generates a mouth-shaped image, and the screen display unit 704 displays the generated mouth-shaped image 801 as shown in FIG. It is displayed on the type display 803 .

FIG. 9 is a flowchart for showing details of the operation of the speech analysis unit 703 according to the second embodiment. First, input speech data is accumulated for a certain period of time (step S901) (step S902), and after the certain period of time the accumulated speech data is analyzed and divided into vowel units (step S903). After that, the divided voice data is transmitted to the screen display section one sound at a time (step S904).

FIG. 10 is a flowchart for showing details of the operation of the screen display unit 704 according to the second embodiment. From the sound data for one sound received from the sound analysis unit, an image of the shape of the mouth that matches the sound is generated (step S1001), and displayed on the display (step S1002) for a certain period of time (step S1003). good.

Some or all of the above-described embodiments can also be described as the following appendices. However, each of the following supplementary notes is merely an example of the present invention, and the present invention is not limited to such cases.
[Appendix 1]
It is as the visualization device according to the first viewpoint described above.
[Appendix 2]
Preferably, the visualization device according to appendix 1, wherein the visualization unit generates visualization data that is an image representing a mouth shape when uttering the recognition data.
[Appendix 3]
Preferably, the visualization device according to appendix 1 or 2, wherein the speech recognition unit extracts vowels contained in each sound that constitutes speech and uses them as recognition data.
[Appendix 4]
4. A visualization device, preferably according to any one of Appendixes 1 to 3, wherein the presentation unit presents the visualization data on a transmissive display device.
[Appendix 5]
Preferably, any one of Appendices 1 to 4, further comprising a body recognition unit that generates body recognition data that recognizes body parts of a conversation partner, and wherein the presentation unit presents visualization data based on the body recognition data. Or one visualization device.
[Appendix 6]
The body recognition unit generates body recognition data that recognizes the position of the mouth of the conversation partner, and the presentation unit uses the body recognition data to generate an image representing the shape of the mouth of the conversation partner. A visualization device, preferably according to clause 5, placed in position.
[Appendix 7]
Preferably, the visualization device according to any one of appendices 1 to 6, wherein the speech recognition unit further recognizes the speech into textual recognition data, and the presenting unit further presents the textual recognition data.
[Appendix 8]
The speech recognition unit further obtains emotion data obtained by estimating the emotion of the conversation partner from the voice, and the visualization unit further generates visualization data based on the emotion data. visualization device.
[Appendix 9]
This is the same as the visualization method related to the second viewpoint described above.
[Appendix 10]
This is the same as the program related to the third viewpoint mentioned above.
It should be noted that Supplementary Notes 9 and 10 can be developed into Supplementary Notes 2 to 8 in the same manner as Supplementary Note 1.

The disclosures of the cited patent documents and the like are incorporated herein by reference. Within the framework of the full disclosure of the present invention (including the scope of claims), modifications and adjustments of the embodiments and examples are possible based on the basic technical concept thereof. Also, various combinations or selections of various disclosure elements (including each element of each claim, each element of each embodiment or example, each element of each drawing, etc.) within the framework of the full disclosure of the present invention (including partial deletion) is possible. That is, the present invention naturally includes various variations and modifications that can be made by those skilled in the art according to the entire disclosure including claims and technical ideas. In particular, any numerical range recited herein should be construed as specifically recited for any numerical value or subrange within that range, even if not otherwise stated.

10: Visualization device 11: Voice acquisition unit 12: Voice recognition unit 13: Visualization unit 14: Presentation unit 15: Body recognition unit 51: CPU
52: Memory 53: Input/output interface 55: Internal bus 601: Smart glasses 602: Microphone 603: Listener 604: Speaker 701: Microphone 702: Transmissive display 703: Sound analysis unit 704: Screen display unit 801: Image 802: Smart Glasses 803: Transmissive Display

Claims (10)

an audio acquisition unit that acquires audio;
a voice recognition unit that recognizes the voice and uses it as recognition data;
a visualization unit that generates visualization data obtained by visualizing the recognition data;
a presentation unit that presents the visualization data as a video;
A visualization device having The visualization unit generates visualization data that is an image representing the shape of the mouth when the recognition data is uttered.
2. The visualization device of claim 1. 3. The visualization device according to claim 1, wherein said speech recognition unit extracts vowels included in each sound constituting said speech and uses them as recognition data. The presentation unit presents the visualization data on a transmissive display device.
Visualization device according to any one of claims 1 to 3. further comprising a body recognition unit that generates body recognition data recognizing body parts of a conversation partner;
the presentation unit presents visualization data based on the body recognition data;
Visualization device according to any one of claims 1 to 4. The body recognition unit generates body recognition data that recognizes the position of the mouth of the conversation partner,
The presentation unit arranges an image representing the shape of the mouth in the visualization data at the position of the mouth of the conversation partner using the body recognition data.
6. The visualization device of claim 5. The speech recognition unit further recognizes the speech and converts it into text format recognition data,
The presentation unit further presents the recognition data in text format.
Visualization device according to any one of claims 1 to 6. The speech recognition unit further acquires emotion data obtained by estimating the emotion of a conversation partner from the speech,
the visualization unit further generates visualization data based on the emotion data;
Visualization device according to any one of claims 1 to 7. obtaining audio;
a step of recognizing the speech as recognition data;
generating visualization data that visualizes the recognition data;
presenting the visualization data;
A visualization method having a process of obtaining audio;
a process of recognizing the voice and converting it into recognition data;
a process of generating visualization data that visualizes the recognition data;
a process of presenting the visualization data;
A program that causes a computer to run

Images