JP2023120068A - Speech processing system, device and speech processing method - Google Patents

Abstract

To provide a speech processing system that lightens a processing load of conversion to text data.SOLUTION: The present invention relates to a speech processing system in which a terminal unit and a device comprising a microphone communicate with each other, and the speech processing system has: a speech synthesis part which synthesizes a first speech that the terminal unit receives from outside and a second speech that the device collects to generate a synthetic speech; and a text conversion request part which sends a request to convert the synthetic speech that the speech synthesis part generates into text data to the outside.SELECTED DRAWING: Figure 2

Description

TECHNICAL FIELD The present invention relates to an audio processing system, a device, and an audio processing method.

2. Description of the Related Art Teleconference service systems are known in which images and voices are transmitted in real time from one base to one or more other bases, and users at remote locations hold conferences using the images and voices. Also, as a method of recording the content of remote communication such as a meeting as minutes, a method of converting the voices of users at each site into text and saving the text data is known.

A technology is known that associates text data with which of a plurality of speakers has spoken (see, for example, Patent Document 1). Patent Document 1 discloses an information processing device that distinguishes a speaker who is speaking by distinguishing the voiceprint of the speaker by voiceprint identification, and adds identification information of one of the speakers to each utterance sentence. It is

However, the conventional technology has a problem that the processing load of conversion to text data is large. In other words, the conventional technology separately recognizes each voice uttered by a plurality of speakers, so there is a risk that resources for voice recognition will be pressed and recognition will take time.

SUMMARY OF THE INVENTION An object of the present invention is to provide a technique for reducing the processing load of conversion into text data.

In view of the above problems, the present invention provides a voice processing system in which a terminal device and a device equipped with a microphone communicate with each other, wherein a first voice received by the terminal device from the outside and a second voice collected by the device are provided. and a text conversion requesting unit for externally requesting conversion of the synthesized speech synthesized by the speech synthesis unit into text data.

It is possible to provide a speech processing system that reduces the processing load of conversion into text data.

FIG. 10 is a diagram illustrating an overview of creation of record information for saving a screen of an application executed during a teleconference together with a surrounding panorama image; 1 is an example of a system block diagram relating to functions of a recording information creation system; FIG. It is a figure which shows the structural example of a record information production system. It is a figure which shows the hardware constitutions of an information processing system and an example of a terminal device. 1 is an example of a hardware configuration diagram of a meeting device capable of capturing 360° moving images; FIG. FIG. 4 is a diagram for explaining an imaging range of a meeting device; FIG. It is a figure explaining the extraction of a panorama image and a speaker image. 1 is an example of a functional block diagram illustrating functions of a terminal device, a meeting device, and an information processing system in a recorded information creation system by dividing them into blocks; FIG. 4 is a diagram showing an example of moving image recording information stored in an information storage unit; FIG. It is a figure which shows an example of the meeting information memorize|stored in the meeting information storage part which the meeting information acquisition part manages. 4 is a diagram showing an example of recording information stored in a recording information storage unit; FIG. 4 is a diagram illustrating the structure of text data stored in the storage service system; FIG. FIG. 10 is a diagram showing an example of an initial screen after login displayed by an information recording application running on a terminal device; It is a figure which shows an example of the recording setting screen which an information recording application displays. It is a figure which shows an example of the meeting list screen which an information recording application displays. It is a figure which shows an example of the conference record confirmation screen which a conference record confirmation part presents and a terminal device displays. It is a figure explaining an example base determination conditions. FIG. 10 is an example of a flow chart for explaining a method of determining a site based on a site determination condition 1-1. FIG. 10 is an example of a flowchart illustrating a method of determining a site based on site determination conditions 1-2. FIG. 10 is an example of a flowchart for explaining a method of determining a site based on a site determination condition 2; FIG. 10 is a diagram for explaining a method by which a user associates site identification information E, speech recognition result D, and synthesized speech C with a cue method; FIG. 10 is a diagram illustrating a method for a user to associate base identification information E, speech recognition result D, and synthesized speech C with a time stamp method; FIG. 10 is an example of a sequence diagram showing a procedure for an information recording application to record a panorama image, a speaker image, and an application screen; FIG. 10 is an example of a sequence diagram showing a procedure for an information recording application to record a panorama image, a speaker image, and an application screen; FIG. 10 is an example of a sequence diagram illustrating processing for a user to edit base identification information E;

Hereinafter, as an example of a mode for carrying out the present invention, a recording information creation system and an audio processing method performed by the recording information creation system will be described.

<Example of how to create text data in a teleconference>
First, with reference to FIG. 1, an outline of a method of creating minutes using a panorama image and an application screen will be described. FIG. 1 is a diagram for explaining an overview of creation of recording information for saving a screen of an application executed during a teleconference together with a surrounding panoramic image. As shown in FIG. 1, a user at his own site 102 is holding a remote conference with another site 101 using a teleconference service system 90 .

The recorded information creation system 100 of this embodiment creates a horizontal panoramic image (hereinafter referred to as a panoramic image) captured by a meeting device 60 equipped with a microphone and a speaker, and an application (hereinafter referred to as an application) executed by the terminal device 10. The record information (minutes) is created by using the screen for As for the sound, the record information creation system 100 (an example of a sound processing system) synthesizes the sound received by the teleconference application 42 and the sound acquired by the meeting device 60, and includes the synthesized sound in the record information. An outline will be described below.

(1) In the terminal device 10, an information recording application 41 and a teleconference application, which will be described later, are running. In addition, an application for displaying materials may also be running. The information recording application 41 transmits audio output by the terminal device 10 (including audio received by the teleconference application from other bases; an example of first audio data) to the meeting device 60 . The meeting device 60 mixes (synthesizes) the voice (an example of the second voice data) acquired by itself and the voice of the teleconference application. The meeting device 60 determines the location of the speech based on the sound pressure of the voice output by the terminal device 10 .

(2) The meeting device 60 is equipped with a microphone, and based on the direction from which the voice is acquired, performs a process of cutting out the speaker from the panorama image and creates a speaker image. The meeting device 60 transmits both the panorama image and the speaker image to the terminal device 10 .

(3) The information recording application 41 running on the terminal device 10 can display the panoramic image 203 and the speaker image 204 . The information recording application 41 combines the screen of any application selected by the user (for example, the screen 103 of the teleconference application), the panorama image 203 and the speaker image 204 . For example, the panoramic image 203, the speaker image 204, and the application screen 103 are combined so that the panoramic image 203 and the speaker image 204 are arranged on the left side, and the remote conference application screen 103 is arranged on the right side (hereinafter referred to as the combined image 105 called). Since the process (3) is repeatedly executed, the combined image 105 becomes a moving image (hereinafter referred to as a combined image moving image). In addition, the information recording application 41 creates a moving image with sound by combining synthesized sound (hereinafter referred to as synthesized sound) with the combined image moving image.

In this embodiment, an example in which the panoramic image 203, the speaker image 204, and the application screen 103 are combined will be described, but the information recording application 41 may store them separately and arrange them on the screen during playback. .

(4) The information recording application 41 accepts editing work (cutting of unnecessary parts by the user, etc.) and completes the combined image moving image. The combined image moving image constitutes part of the recorded information.

(5) The information recording application 41 transmits the created combined image moving image (with sound) to the storage service system 70 and stores it.

(6) In the case of batch conversion of audio, the information recording application 41 extracts only the audio from the combined image moving image (or the audio before combining), and sends the extracted audio to the information processing system 50. Send to The information processing system 50 transmits the speech to a speech recognition service system 80 that converts the speech into text data, converting the speech into text. The text data also includes data on how many minutes after the recording started.

(7) In the case of real-time conversion, the meeting device 60 transmits the synthesized speech C to the information processing system 50 after determining the base. The information processing system 50 converts the data into text data in real time using the speech recognition service system, and transmits this text data to the information recording application 41 . In this embodiment, the case of real-time conversion will be mainly described.

Note that the information processing system 50 can execute charging processing according to the service used by the user. For example, the charge is calculated based on the amount of text data, the file size of the combined image moving image, the processing time, and the like.

(8) The information processing system 50 additionally stores text data in the storage service system 70 that stores the combined image moving image. The user can reproduce the combined image moving image on the terminal device 10 . Text data forms part of the recorded information.

In this way, in the combined image moving image, the surrounding panorama image including the user, the speaker image, and the screen of the application displayed during the teleconference are displayed and recorded. Since the speech recognition is performed on the synthesized speech, the processing load of the speech recognition service system can be reduced as compared with separate speech recognition. In addition, since the site where the synthetic voice was spoken is determined based on the sound pressure information, it is possible to record the site where the voice data was spoken.

<Terms>
An application (app) is software that is developed and used for a specific function or purpose, not for operating a computer itself. Applications include native apps and web apps.

A running application is an application that is in a state from when the application is started until it is terminated. The app doesn't have to be active (the one in the foreground), it just needs to run in the background.

A device is a device capable of capturing an image of the surroundings and collecting sounds of the surroundings. In this embodiment, the term meeting device 60 will be used.

The image of the surroundings acquired by the meeting device 60 refers to an image captured with a wider angle of view than the normal angle of view in the horizontal direction. In this embodiment, surrounding images are described in terms of panoramic images. A panoramic image is an image having an angle of view of approximately 180° to 360° in the horizontal direction. The meeting device 60 does not have to capture a panoramic image by itself, and may be a combination of a plurality of imaging devices with normal angles of view.

Recorded information is information recorded by the information recording application 41 . When the information recording application 41 records the screen of the teleconference application, the recorded information may be minutes. Recorded information includes, for example, combined image moving images (including audio) and text data in which audio has been recognized.

A tenant is a group of users (companies, local governments, some of these organizations, etc.) who have contracted to receive services from a service provider. Creation of record information and conversion into text data in this embodiment are executed because the tenant has a contract with the service provider.

Remote communication refers to communicating with a person at a physically distant base through voice and video using software and terminal devices. An example of remote communication is a teleconference, and the conference may be called a meeting, a meeting, a meeting, a gathering, a get-together, a gathering, and a seminar may be called a workshop, a study group, a seminar, a training session, or the like.

A base is a place where activities are based. An example of a base is a conference room. A conference room is a room set up mainly for the purpose of using it for a conference.

Speech refers to language sounds uttered by humans, ambient sounds, etc., and speech data is speech data converted into data.

<System block diagram related to functions>
FIG. 2 is a system block diagram relating to the functions of the recorded information creation system 100. As shown in FIG. The recording information creation system 100 has a self-site voice input unit 7 that acquires a self-site voice A in a teleconference, and an other-site voice input unit 8 that acquires other-site voice B of a site other than the self-site.

The local site voice input unit 7 may be, for example, a general microphone. Also, the other-site voice input unit 8 is a module that can acquire the other-site voice B separately from the own-site voice input unit 7 . The other site voice input unit 8 is, for example, a USB connector, Bluetooth (registered trademark), Wi- A wireless communication module such as Fi can be considered.

With the above configuration, the recorded information creation system 100 can produce separate voices for own site A and other site B even in a remote conference using a general remote communication system without preparing a plurality of dedicated terminals or dedicated accounts. can be obtained as

Further, the local site's voice A and the other site's voice B are input to the voice synthesizing unit 65 and output as a synthesized voice C after being subjected to synthesizing processing. The synthesized speech C is converted into text by a speech recognition technique in the speech recognition section 55 and stored as a speech recognition result D in the speech recognition result recording section 57 . Furthermore, the synthesized speech C is recorded as speech data by the speech recording unit 56 .

As a speech recognition technology, the speech recognition service system 80 can use a general speech recognition engine. As the implementation form of the speech recognition engine, it is possible to select it for each system, such as embedding it in hardware or using a cloud service.

By synthesizing the local site's voice A and the other site's voice B into the synthesized voice C, it is possible to input the voice data to the voice recognition unit 55 because the amount of data is less likely to increase even in a poor communication environment.

In addition, if voice recognition is performed without synthesizing voices from a plurality of sites, two voice recognition processes must be performed in parallel, so there is concern about performance degradation during voice recognition, and there is a possibility that real-time performance will be impaired. However, by synthesizing the voices of a plurality of bases and inputting the voice data as synthesized voice C to the voice recognition unit 55, it becomes possible to convert the voice data into text by voice recognition at low performance and low cost, and the real-time performance is also improved. Furthermore, by recognizing the synthesized speech C, it is possible to perform speech recognition by estimating the context, thereby improving the speech recognition rate.

Furthermore, since the voice recording unit 56 records the synthesized voice C, the voices uttered at the same time can be recorded as they are without loss of either of the voice data before synthesis or shifting of the timing of utterances during reproduction. Therefore, the user can check the conference voice as a conference record at a later date without any sense of incongruity.

Note that the recorded information creation system 100 may have a voice data processing unit that processes and processes voice data before voice recognition or voice synthesis.

The recorded information creation system 100 also has a site determination processing unit 64 . The base determination processing unit 64 has a sound pressure detection unit 64a that detects own base sound pressure information A' and other base sound pressure information B' from the own base sound A and the other base sound B, respectively. The base determination unit 64b compares the own base sound pressure information A' and the other base sound pressure information B' to determine whether the synthesized speech C is mainly composed of the own base sound A or the other base sound B. By doing so, the base identification information E is generated. The site identification information E is stored in the site determination result recording unit 58 .

By using a simple means of comparing sound pressure information for identifying the speaking base, it is possible to identify the speech base without using speaker identification AI such as voice recognition AI using voiceprints and face recognition AI using the face image of the speaker. , it is possible to identify bases with low performance, and it is possible to implement at low cost.

Furthermore, the recorded information creation system 100 synchronizes the site identification information E, the voice recognition result D, and the synthesized voice C from the site determination result recording unit 58, the voice recognition result recording unit 57, and the voice recording unit 56, respectively. It has a conference record confirmation section 59 that is displayed to the reading user.

As described above, the user can recognize whether the speech recognition result is a text based on the utterance of the user's own site or a text based on the utterance of another site, thereby facilitating understanding of the minutes.

<System configuration example>
Next, with reference to FIG. 3, the system configuration of the recording information creation system 100 will be described. FIG. 3 shows a configuration example of the recording information creation system 100. As shown in FIG. FIG. 3 shows one site (own site 102) among a plurality of sites where remote conferences are held. Communicate with service system 90 . A meeting device 60 is further arranged at the self-site 102, and the terminal device 10 is communicably connected to this meeting device 60 via a USB cable or the like.

At least an information recording application 41 and a teleconference application 42 operate on the terminal device 10 . The teleconference application 42 can communicate with the terminal devices 10 of the other bases 101 via the teleconference service system 90 on the network, so that users at each base can have a conference from a remote location. The information recording application 41 uses the functions of the information processing system 50 and the meeting device 60 to create recorded information in the remote conference held by the remote conference application 42 .

In this embodiment, an example of creating recorded information during a remote conference will be described, but the conference to be recorded does not have to be a conference that communicates with a remote site. In other words, the conference may be a conference in which only participants within one base participate. In this case, only the sound collected by the meeting device 60 is saved without synthesis, and the processing of the information recording application 41 remains unchanged.

The terminal device 10 has a built-in (or external) camera with a normal angle of view, and captures a front image including the user 107 operating the terminal device 10 . A normal angle of view is an image that is not a panoramic image, but in the present embodiment, it is mainly a planar image that is not a curved surface such as an omnidirectional image. In addition, the terminal device 10 has a built-in microphone (it may be externally attached), and collects the surrounding sounds of the user operating the terminal device 10 or the like. Therefore, the user can have a conventional remote conference using the remote conference application 42 without being conscious of the information recording application 41 . The information recording application 41 and the meeting device 60 do not affect the teleconference application 42 except for an increase in the processing load on the terminal device 10 . Note that the teleconference application 42 can also transmit panoramic images and speaker images captured by the meeting device 60 to the teleconference service system 90 .

The information recording application 41 communicates with the meeting device 60 to create recorded information. The meeting device 60 is a meeting device that includes a panorama image capturing device, a microphone, and a speaker. The camera of the terminal device 10 can capture only a limited range of the front, but the meeting device 60 can capture the entire surroundings (not necessarily the entire surroundings) surrounding the meeting device 60 . The meeting device 60 can always keep the multiple participants 106 shown in FIG. 3 within the angle of view.

In addition, the meeting device 60 cuts out a speaker image from a panorama image, synthesizes the voice acquired by the meeting device 60 and the voice output by the terminal device 10 (including the voice received by the teleconference application 42), and the like. conduct. Note that the meeting device 60 may be placed anywhere in the local site 102, not just on the desk. Since the meeting device 60 can capture an omnidirectional image, it may be placed on the ceiling, for example.

The information recording application 41 displays a list of applications being executed on the terminal device 10, synthesizes images for the above-described recorded information (creates a combined image moving image), reproduces the combined image moving image, accepts editing, and the like. The information recording application 41 also displays a list of remote conferences that have been held or are scheduled to be held. The list of teleconferences is used for information about recorded information and allows users to associate teleconferences with recorded information.

The teleconference application 42 performs communication connection with other bases 101, transmission/reception of images and voices with other bases 101, display of images, output of voices, and the like.

Note that the information recording application 41 and the teleconference application 42 may be web applications or native applications. A web application is an application that performs processing in cooperation with a program on a web server and a program on a web browser, and does not need to be installed on the terminal device 10 . A native application is an application that is installed in the terminal device 10 and used. In this embodiment, it is assumed that both are native applications.

The terminal device 10 may be, for example, a general-purpose information processing device having a communication function, such as a PC (Personal Computer), a smart phone, a tablet terminal, or the like. In addition, the terminal device 10 may be an electronic blackboard, a game machine, a PDA (Personal Digital Assistant), a wearable PC, a car navigation system, an industrial machine, a medical device, a network appliance, or the like. The terminal device 10 may be any device as long as the information recording application 41 and the teleconference application 42 operate.

The information processing system 50 is one or more information processing devices arranged on a network. The information processing system 50 has one or more server applications that perform processing in cooperation with the information recording application 41, and infrastructure services. This server application acquires a list of remote conferences managed by the conference management system 9, manages recording information recorded in remote conferences, and manages various settings and storage paths. The platform service performs user authentication, contracts, billing processing, and the like.

The conference management system 9 is a system for reserving conference rooms and managing conference schedules. The conference management system 9 and the information processing system 50 may be integrated.

All or part of the functions of the information processing system 50 may exist in the cloud environment or may exist in the on-premises environment. The information processing system 50 may be configured by a plurality of server devices, or may be configured by a single information processing device. For example, the server application and the infrastructure service may be provided by separate information processing devices, or there may be an information processing device for each function within the server application. The information processing system 50, the storage service system 70, and the voice recognition service system 80, which will be described later, may be integrated.

The storage service system 70 is storage means on the network, and provides a storage service that receives storage of files and the like. One Drive (registered trademark), Google Workspace (registered trademark), DropBox (registered trademark), and the like are known as the storage service system 70 . The storage service system 70 may be an on-premise NAS (Network Attached Storage) or the like.

The voice recognition service system 80 provides a service of performing voice recognition on voice data and converting it into text data. The speech recognition service system 80 may be a general-purpose commercial service, or may be part of the functions of the information processing system 50 .

<Hardware configuration example>
Hardware configurations of the information processing system 50 and the terminal device 10 according to the present embodiment will be described with reference to FIG. 4 .

<<information processing system and terminal device>>
FIG. 4 is a diagram showing an example hardware configuration of the information processing system 50 and the terminal device 10 according to this embodiment. As shown in FIG. 4, the information processing system 50 and the terminal device 10 are constructed by a computer, and include a CPU 501, a ROM 502, a RAM 503, an HD (Hard Disk) 504, an HDD (Hard Disk Drive) controller 505, and a display 506. , an external device connection I/F (Interface) 508 , a network I/F 509 , a bus line 510 , a keyboard 511 , a pointing device 512 , an optical drive 514 and a media I/F 516 .

Among these, the CPU 501 controls the operations of the information processing system 50 and the terminal device 10 as a whole. The ROM 502 stores programs used to drive the CPU 501 such as IPL. A RAM 503 is used as a work area for the CPU 501 . The HD 504 stores various data such as programs. The HDD controller 505 controls reading or writing of various data to/from the HD 504 under the control of the CPU 501 . A display 506 displays various information such as cursors, menus, windows, characters, or images. The external device connection I/F 508 is an interface for connecting various external devices. The external device in this case is, for example, a USB (Universal Serial Bus) memory, a printer, or the like. A network I/F 509 is an interface for data communication using the network N2. A bus line 510 is an address bus, a data bus, or the like for electrically connecting each component such as the CPU 501 shown in FIG.

Also, the keyboard 511 is a type of input means having a plurality of keys used for inputting characters, numerical values, various instructions, and the like. A pointing device 512 is a kind of input means for selecting and executing various instructions, selecting a processing target, moving a cursor, and the like. An optical drive 514 controls reading or writing of various data to an optical storage medium 513 as an example of a removable recording medium. Note that the optical storage medium 513 may be a CD, DVD, Blu-ray (registered trademark), or the like. A media I/F 516 controls reading or writing (storage) of data to a recording medium 515 such as a flash memory.

<<Meeting Device>>
The hardware configuration of the meeting device 60 will be described with reference to FIG. FIG. 5 is an example of a hardware configuration diagram of a meeting device 60 capable of capturing 360° moving images. In the following description, the meeting device 60 is a device that uses an imaging element to capture a 360° moving image around the device at a predetermined height, but the number of imaging elements may be one or two or more. Further, it is not always necessary to use a dedicated device, and a PC, digital camera, smartphone, or the like may have substantially the same function by attaching a post-installed 360° video imaging unit.

As shown in FIG. 5, the meeting device 60 includes an imaging unit 601, an image processing unit 604, an imaging control unit 605, a microphone 608, a sound processing unit 609, a CPU (Central Processing Unit) 611, a ROM (Read Only Memory). ) 612, SRAM (Static Random Access Memory) 613, DRAM (Dynamic Random Access Memory) 614, operation unit 615, external device connection I/F 616, communication unit 617, antenna 617a, audio sensor 618, and concave for Micro USB It is configured by a terminal 621 .

Among them, the imaging unit 601 includes a wide-angle lens (a so-called fisheye lens) 602 having a 360° field angle for forming a hemispherical image, and an imaging element 603 (image sensor) provided corresponding to each wide-angle lens. It has The image sensor 603 is an image sensor such as a CMOS (Complementary Metal Oxide Semiconductor) sensor or a CCD (Charge Coupled Device) sensor that converts an optical image formed by the fisheye lens 602 into electrical signal image data and outputs the image data. It has a timing generation circuit that generates synchronization signals, pixel clocks, etc., and a group of registers in which various commands and parameters necessary for the operation of this image sensor are set.

The imaging elements 603 (image sensors) of the imaging unit 601 are each connected to the image processing unit 604 via a parallel I/F bus. On the other hand, the imaging element 603 of the imaging unit 601 is connected to the imaging control unit 605 via a serial I/F bus (such as an I2C bus). The image processing unit 604 , imaging control unit 605 and sound processing unit 609 are connected to a CPU 611 via a bus 610 . Furthermore, the bus 610 is also connected with a ROM 612, an SRAM 613, a DRAM 614, an operation unit 615, an external device connection I/F 616, a communication unit 617, an audio sensor 618, and the like.

The image processing unit 604 takes in the image data output from the image pickup device 603 through the parallel I/F bus, performs predetermined processing on each image data, and converts the fisheye video into panorama image and speaker image data. to create Further, the image processing unit 604 synthesizes the panorama image, the speaker image, etc., and outputs one moving image.

The imaging control unit 605 generally uses the I2C bus with the imaging control unit 605 as a master device and the imaging device 603 as a slave device to set commands and the like in registers of the imaging device 603 . Necessary commands and the like are received from the CPU 611 . The imaging control unit 605 also uses the I2C bus to take in status data and the like of the register group of the imaging device 603 and send it to the CPU 611 .

Also, the imaging control unit 605 instructs the imaging devices 603a and 603b to output image data at the timing when the imaging start button of the operation unit 615 is pressed or at the timing when the imaging start instruction is received from the PC. Depending on the meeting device 60, there may be a function corresponding to a preview display function or moving image display by a display (for example, a display of a PC or a smartphone). In this case, image data is output continuously from the image sensor 603 at a predetermined frame rate (frames/minute).

The imaging control unit 605 also functions as synchronization control means for synchronizing the output timing of the image data of the imaging element 603 in cooperation with the CPU 611, as will be described later. In this embodiment, the meeting device 60 is not provided with a display, but may be provided with a display unit.

A microphone 608 converts sound into sound (signal) data. The sound processing unit 609 takes in audio data output from the microphone 608 through the I/F bus and performs predetermined processing on the audio data.

The CPU 611 controls the overall operation of the meeting device 60 and executes necessary processing. ROM 612 stores various programs for CPU 611 . An SRAM 613 and a DRAM 614 are work memories, and store programs to be executed by the CPU 611, data during processing, and the like. In particular, the DRAM 614 stores image data being processed by the image processing unit 604 and data of the processed equirectangular projection image.

The operation unit 615 is a general term for operation buttons such as the imaging start button 615a. By operating the operation unit 615, the user inputs settings such as starting image capturing and recording, power ON/OFF execution, communication connection execution, and various image capturing modes and image capturing conditions.

The external device connection I/F 616 is an interface for connecting various external devices. The external device in this case is, for example, a PC (Personal Computer) or the like. The moving image data and image data stored in the DRAM 614 are transmitted to an external terminal via the external device connection I/F 616 or recorded on an external medium.

The communication unit 617 communicates with the cloud server via the Internet using wireless communication technology such as Wi-Fi via an antenna 617a provided in the meeting device 60, and transmits the stored video data and image data to the cloud server. may Also, the communication unit 617 may communicate with nearby devices using short-range wireless communication technology such as BLE (Bluetooth Low Energy. Registered trademark) and NFC.

The audio sensor 618 is a sensor that acquires 360° audio data in order to identify from which direction a loud sound is input in 360° around the meeting device 60 (horizontal plane). The sound processing unit 609 identifies the strongest direction according to the input 360° audio parameters and outputs the 360° audio input direction.

Note that other sensors (azimuth/acceleration sensor, GPS, etc.) may calculate the azimuth, position, angle, acceleration, etc., and use them for image correction and addition of position information.

The image processing unit 604 also performs the following processing.

- The CPU 611 creates a panorama image by the following method. The CPU 611 performs predetermined camera image processing such as Bayer conversion (RGB interpolation processing) on RAW data input from an image sensor that inputs a spherical image to create a fisheye image (curved surface image). Further, DeWarp processing (distortion correction processing) is performed on the created fisheye image (curved surface image) to create a panoramic image (planar image) showing a 360° surrounding of the meeting device 60 .

- The CPU 611 creates a speaker image by the following method. The CPU 611 creates a speaker image by cutting out the speaker from a panorama image (planar image) showing 360° surroundings. The CPU 611 cuts out a speaker image from the panorama image with the audio input direction specified from 360° output by the audio sensor 618 and the sound processing unit 609 as the direction of the speaker. At this time, the method of extracting an image of a person from the voice input direction is to cut out 30° from 360° centering on the specified voice direction, and perform face detection (or person detection, etc. is also possible) in it. The CPU 611 further identifies, among the cut-out speaker images, speaker images for a specific number of people (three, etc.) who have recently spoken.

The panorama image and one or more speaker images may be individually sent to the information recording application 41, or the meeting device 60 may create one image from these and send it to the information recording application 41. . In this embodiment, it is assumed that the panoramic image and one or more speaker images are individually transmitted from the meeting device 60 to the information recording application 41 .

FIG. 6 is a diagram for explaining the imaging range of the meeting device 60. As shown in FIG. As shown in FIG. 6(a), the meeting device 60 takes an image in a horizontal 360° range. As shown in FIG. 6B, the meeting device 60 has an imaging range of 0° in a horizontal direction to the height of the meeting device 60 and a predetermined vertical angle.

FIG. 7 is a diagram for explaining the extraction of the panorama image and the speaker image. As shown in FIG. 7, the image captured by the meeting device 60 forms a portion 110 of a sphere and thus has a three-dimensional shape. As shown in FIG. 6B, the meeting device 60 divides the angle of view into predetermined vertical angles and horizontal predetermined angles to perform perspective projection conversion. A predetermined number of planar images can be obtained by performing perspective projection transformation over the entire 360° in the horizontal direction without any gaps. In addition, the meeting device 60 performs face detection in a predetermined range centering on the sound direction from the panorama image, and creates a speaker image 112 by cutting out 15° left and right from the center of the face (30° in total). do.

<About functions>
Next, with reference to FIG. 8, the functional configuration of the recorded information creation system 100 will be described. FIG. 8 is an example of a functional block diagram illustrating the functions of the terminal device 10, the meeting device 60, and the information processing system 50 in the recorded information creation system 100 by dividing them into blocks.

<< terminal device >>
The information recording application 41 operating on the terminal device 10 includes the communication unit 11, the operation reception unit 12, the display control unit 13, the application screen acquisition unit 14, the audio acquisition unit 15, the device communication unit 16, the video storage unit 17, audio data processing. It has a unit 18 , a recording/playback unit 19 , an upload unit 20 , an edit processing unit 21 and a search unit 22 . Each of these units of the terminal device 10 is realized by operating one of the components shown in FIG. It is a function or means of functioning. The terminal device 10 also has a storage unit 1000 constructed by the HD 504 or the like shown in FIG. An information storage unit 1001 is constructed in the storage unit 1000 .

The communication unit 11 communicates various information with the information processing system 50 via the network. The communication unit 11 receives, for example, a list of teleconferences from the information processing system 50 and transmits a recognition request for voice data to the information processing system 50 .

The display control unit 13 displays various screens serving as user interfaces in the information recording application 41 in accordance with screen transitions set in the information recording application 41 . The operation reception unit 12 receives various operations for the information recording application 41 .

The application screen acquisition unit 14 acquires a desktop screen or an application screen selected by the user from an OS (Operating System) or the like. When the application selected by the user is the teleconference application 42, a screen generated by the teleconference application 42 (images of each site, images of materials, etc.) is obtained.

The voice acquisition unit 15 acquires voice (including voice data received from the remote conference application 42 in the remote conference) output by the terminal device 10 from a microphone or an earphone. Even if the output sound is muted, the sound acquisition unit 15 can acquire the sound. The user does not need to perform operations such as selecting the teleconference application 42 for audio data, and the audio acquisition unit 15 can acquire audio that the terminal device 10 can output via the OS or API (Application Interface). As a result, voice data received by the teleconference application 42 from the other site 101 is also obtained. If the remote conference application 42 is not running or the remote conference is not in progress, the information recording application 41 may not be able to acquire voice data. It should be noted that the sound acquired by the sound acquisition unit 15 does not include the sound collected by the terminal device 10, but only the output sound data. This is because the meeting device 60 separately collects the sound.

The device communication section 16 communicates with the meeting device 60 using a USB cable or the like. The device communication section 16 may communicate with the meeting device 60 via a wireless LAN, Bluetooth (registered trademark), or the like. The device communication section 16 receives the panoramic image and the speaker image from the meeting device 60 and transmits the audio data acquired by the audio acquisition section 15 to the meeting device 60 . The device communication section 16 receives voice data synthesized by the meeting device 60 .

The moving image storage unit 17 combines the panorama image and the speaker image received by the device communication unit 16 and the application screen acquired by the application screen acquisition unit 14 to create a combined image. In addition, the moving image storage unit 17 creates a combined image moving image by connecting repeatedly created combined images in time series, and combines synthesized audio data with the combined image moving image to create a combined image moving image with sound.

The audio data processing unit 18 extracts the audio data combined with the combined image moving image, or requests the information processing system 50 to convert the synthesized audio data received from the meeting device 60 into text data.

The recording/playback unit 19 plays back the combined image moving image. The combined image moving image is stored in the terminal device 10 during recording, and then uploaded to the information processing system 50 .

The upload unit 20 transmits the combined image moving image to the information processing system 50 when the teleconference ends.

The editing processing unit 21 edits the combined image moving image (partial deletion, joining, etc.) according to the user's operation.

The search unit 22 receives a search for text data using a keyword, searches the text data, and displays the search results.

FIG. 9 shows moving image recording information stored in the information storage unit 1001 . The moving image recording information has items such as meeting ID, recording ID, update date and time, title, upload, and save destination. When the user logs into the information processing system 50 , the information recording application 41 downloads the meeting information from the meeting information storage section 5001 of the information processing system 50 . A meeting ID or the like included in the meeting information is reflected in the moving image recording information. The moving image record information in FIG. 9 is held by the terminal device 10 operated by a certain user.

- The conference ID is identification information for identifying the held remote conference. The conference ID is numbered when a remote conference schedule is registered in the conference management system 9 or is numbered by the information processing system 50 upon request from the information recording application 41 .

- The recording ID is identification information for identifying the combined image video recorded in the teleconference. The recording ID is numbered by the meeting device 60, but may be numbered by the information recording application 41 or the information processing system 50. FIG. A different recording ID assigned to the same conference ID indicates a case where the recording was terminated in the middle of the remote conference but restarted for some reason.

- The updated date and time is the date and time when the combined image moving image was updated (recording ended). When the combined image moving image is edited, it is the date and time when it was edited.

• The title is the conference name of the conference. It may be set when the conference is registered in the conference management system 9, or may be arbitrarily set by the user.

· Upload indicates whether or not the combined image moving image has been uploaded to the information processing system 50 .

The storage destination indicates the location (URL or file path) where the combined image moving image and the text data are stored in the storage service system 70 . Therefore, the user can arbitrarily view the uploaded combined image moving image. Note that the combined image moving image and the text data are saved with separate file names following the URL, for example.

<<Meeting Device>>
Returning to FIG. 8, description will be made. The meeting device 60 includes a communication unit 61 , a panorama image creation unit 62 , a speaker image creation unit 63 , a location determination processing unit 64 , a voice synthesis unit 65 , a text conversion request unit 66 , a device connection unit 67 , and a sound collection unit 68 . have. Each of these units of the meeting device 60 is a function implemented by hardware or a means for functioning by one of the components shown in FIG. However, each of these units may be implemented by operating according to instructions from the CPU 611 according to a program expanded from the ROM 612 to the DRAM 614 .

The communication unit 61 transmits and receives various information to and from the information processing system 50 . The communication unit 61 can also communicate with the voice recognition service system and the storage service system 70 .

The device connection unit 67 receives voice input from the terminal device 10 . The device connection unit 67 communicates with the terminal device 10 using a USB cable or the like. The device connection unit 67 may communicate with the terminal device 10 via a wireless LAN, Bluetooth (registered trademark), or the like. The meeting device 60 may include a speaker capable of outputting the other site audio B acquired by the device connection unit 67 . The device connection section 67 corresponds to the other site audio input section 8 in FIG.

A panorama image creating unit 62 creates a panorama image. A speaker image creating unit 63 creates a speaker image. These creation methods have been described with reference to FIGS. 6 and 7. FIG.

The site determination processing unit 64 divides the own site audio A and the other site audio B according to a predetermined rule, and divides the own site sound pressure information A' of the divided own site audio A and the other site sound pressure information of the other site audio B. Based on the information B', the site where the local site's voice A or the other site's voice B is uttered is determined. Details will be described later.

The voice synthesizing unit 65 synthesizes the voice transmitted from the terminal device 10 and the voice collected by the sound collecting unit 68 . As a result, the voice uttered at the other site 101 and the utterance at the own site 102 are put together.

The sound collector 68 converts the audio signal of the local site audio A acquired by the microphone of the meeting device 60 into audio data (digital). As a result, the content of what the user or the participant said at the site on the terminal device 10 side is collected. Own site audio A collected by the sound collecting unit 68 may be transmitted to the terminal device 10 via the device connecting unit 67 . This allows the user to use the meeting device 60 like a general external microphone device. The sound collection unit 68 (and the sound acquisition unit 15 may also be included) corresponds to the local site sound input unit 7 in FIG.

Also, the meeting device 60 has a sound collecting unit 68 and a device connecting unit 67, so that the local site audio A and the other site audio B can be acquired individually. Therefore, it becomes possible to separately acquire the own site sound pressure information A' and the other site sound pressure information B'.

The text conversion request unit 66 transmits the synthesized speech C to the information processing system 50 via the communication unit 61 and requests real-time speech recognition (conversion to text data).

<<Information Processing System>>
The information processing system 50 includes a communication unit 51, an authentication unit 52, a screen generation unit 53, a meeting information acquisition unit 54, a voice recognition unit 55, a voice recording unit 56, a voice recognition result recording unit 57, a location determination result recording unit 58, a conference A record confirmation unit 59 is provided. Each of these units of the information processing system 50 has a function realized by any one of the components shown in FIG. or a means of functioning. The information processing system 50 also has a storage unit 5000 constructed by the HD 504 or the like shown in FIG. A meeting information storage unit 5001 and a recording information storage unit 5002 are constructed in the storage unit 5000 .

The communication unit 51 transmits and receives various information to and from the terminal device 10 . The communication unit 51 , for example, transmits a list of teleconferences to the terminal device 10 and receives voice data recognition requests from the terminal device 10 .

The authentication unit 52 authenticates the user operating the terminal device 10 . The authentication unit 52 authenticates the user by, for example, determining whether or not the authentication information (user ID and password) included in the authentication request received by the communication unit 51 matches pre-stored authentication information. Note that the authentication information may be a card number of an IC card, biometric information such as a face or a fingerprint, or the like. Also, the authentication unit 52 may perform authentication using an external authentication system or an authentication method such as OAUTH.

The screen generation unit 53 generates screen information displayed by the terminal device 10 . When the terminal device 10 executes the native application, the terminal device 10 holds the screen information, and the information to be displayed is transmitted in XML or the like. When the terminal device 10 executes a web application, the screen information is created using HTML, XML, CSS (Cascade Style Sheet), JavaScript (registered trademark), and the like.

The conference information acquisition unit 54 acquires information about remote conferences from the conference management system 9 using each user's account or a system account assigned to the information processing system 50 . The conference information acquisition unit 54 can acquire a list of teleconferences for which the user belonging to the tenant has viewing authority. Since the conference ID is set for the remote conference, the remote conference and the recording information are associated with each other by the conference ID.

The speech recognition unit 55 converts the synthesized speech C requested by the terminal device 10 or the meeting device 60 to be converted into text data into text data using an external speech recognition service. The speech recognition unit 55 itself may convert.

The voice recording unit 56 records at least the synthesized voice C in advance. The voice recognition result recording unit 57 stores the voice recognition result D. FIG. The site determination result recording unit 58 stores the site identification information E. FIG. The site determination result recording unit 58 , the voice recognition result recording unit 57 and the voice recording unit 56 may be stored in the information processing system 50 or may be stored in the storage service system 70 . That is, the location determination result recording unit 58 , the voice recognition result recording unit 57 and the voice recording unit 56 may function as passive storage, or may function in cooperation with the recorded information storage unit 7001 of the storage service system 70 . In this embodiment, the latter is mainly explained.

Synthesized speech C, speech recognition result D, and site identification information E are stored in a divided state in association with each other. A recognition result character string is a character string generated for each segment of speech recognition. Where to break the speech recognition is appropriately set, for example, silent state, fixed time, and the like.

The conference record confirmation unit 59 synchronously acquires the base identification information E, the speech recognition result D, and the synthesized speech C, and presents them to the user, thereby accepting confirmation of the base determination.

With such a configuration, the meeting device 60 can use a remote conference using a widely-used remote communication system, without using multiple dedicated microphones for collecting voices and identifying speaking bases. It is possible to pick up the voice in a natural form and identify the speaking base.

<<meeting information storage>>
FIG. 10 shows an example of conference information stored in the conference information storage unit 5001 managed by the conference information acquisition unit 54. As shown in FIG. The conference information acquisition unit 54 can use the above account to acquire a list of teleconferences for which the user belonging to the tenant has viewing authority. In this embodiment, a remote conference is taken as an example, but the list of remote conferences also includes conferences held in only one conference room.

The conference information is managed by a conference ID, and is associated with participants, title (meeting name), start time, end time, location, and the like. These are examples of conference information, and the conference information may include other information.

- Meeting ID is the identification information of a meeting.

- Tenant ID is the identification information of the tenant where the conference is held.

• The title is the subject or agenda of the meeting.

・The organizer is the organizer of the conference.

• Participants is a list of participants invited to the conference.

- The users with viewing authority is a list of users who can access the conference resource including the combined image video.

• Ad hoc participants is a list of guest participants.

- The location is information about the conference room, such as the name of the conference room.

• The start time is the scheduled time at which the conference will start.

• The end time is the scheduled time at which the conference will end.

- The conference creator is the user ID or the like that registered the conference information.

- The password is the password for the participant to log in to the conference.

- The location is the place where the conference is held, such as a conference room, a branch office name, a building, and the like.

As shown in FIGS. 9 and 10, the combined image moving image recorded at the conference is specified by the conference ID.

FIG. 11 shows recording information stored in the recording information storage unit 5002. As shown in FIG. The recording information has a list of combined image videos recorded by all users belonging to the tenant. The recording information has items such as meeting ID, recording ID, date and time of update, title, storage destination, and the like. These items may be the same as those in FIG.

<<Storage Service System>>
The storage service system 70 may be any service system that stores recorded information. The recorded information storage unit 7001 stores recorded information (combined image moving image, text data). FIG. 12 illustrates the structure of text data saved in the recording information storage unit 7001. FIG.

FIG. 12 is a diagram illustrating the structure of text data stored in the storage service system 70. As shown in FIG. As shown in FIG. 12, the items ID, time, recognition result character string, voice data, and site identification information E are associated with the text data. Note that the text data is stored in association with the conference ID.

- ID is identification information numbered when own site audio A and other site audio B are divided according to a predetermined rule. Predetermined rules are set in the meeting device 60 (and at least one of the speech recognition service system 80), and for example, when a silent state continues for a certain period of time, it is forcibly separated at a certain period of time even if there is no silent state. , segment each sentence detected by morphological analysis, and so on.

• time is the utterance time based on the duration from the start of recording. Since the so-called time is also saved at the start of recording, the time (absolute time) when the text was said can also be known.

・The recognition result character string is a part of the text data obtained by converting the divided synthesized speech C by speech recognition. Synthesized speech C is speech data that is the source of conversion of the recognition result character string.

・Speech data is the synthesized speech C (divided) obtained by synthesizing the self-site speech A and the other-site speech B after the determination of the site.

The base identification information E is the identification information of the base where the voice data was uttered, determined based on the sound pressure of the own base sound pressure information A' and the other base sound pressure information B'. As for the base identification information E, for example, 1 indicates the own base and 2 indicates the other base.

In this way, since the time and the recognition result character string are associated with each other, if the text data matches the search, the information recording application 41 can reproduce the combined image moving image from the reproduction time corresponding to this time.

<Screen transition>
Next, some screens displayed by the terminal device 10 during the teleconference will be described with reference to FIGS. 13 to 16. FIG. FIG. 13 shows an initial screen 200 after login displayed by the information recording application 41 operating on the terminal device 10 . A user of the terminal device 10 connects the information recording application 41 to the information processing system 50 . When the user enters the authentication information and successfully logs in, the initial screen 200 of FIG. 13 is displayed.

The initial screen 200 includes a fixed display button 201, a change front button 202, a panorama image 203, one or more speaker images 204a to 204c (hereinafter referred to as speaker images 204 when not distinguished), and a recording start button 205. have. When the meeting device 60 has already started up at the time of login and is capturing an image of the surroundings, a panorama image 203 created by the meeting device 60 and a speaker image 204 are displayed on the initial screen 200 . Therefore, the user can decide whether or not to start recording while viewing these. When the meeting device 60 is not activated (not capturing an image), the panorama image 203 and the speaker image 204 are not displayed.

Note that the information recording application 41 may display the speaker images 204 of all the participants based on all the faces detected from the panoramic image 203, or display only the N speaker images 204 who spoke most recently. may be displayed. FIG. 13 shows an example in which up to three speaker images 204 are displayed. Until the participant speaks, there may be no speaker images 204 (the number increases one by one according to the speech), or the speaker images 204 of three participants in a predetermined direction may be displayed ( (Replaced according to the statement).

When no one is speaking, such as immediately after the meeting device 60 is activated, the speaker image 204 is created in a predetermined direction (0°, 120°, 240°, etc.) within the horizontal 360°. If fixed display, which will be described later, is set, priority is given to the setting of fixed display.

A fixed display button 201 is a button for the user to perform an operation to fix and close up an area of the panorama image 203 as the speaker image 204 .

The change front button 202 is a button for the user to change the front of the panorama image 203 (because the panorama image is horizontally oriented at 360°, the directions of the right end and the left end are the same). The user can slide the panorama image 203 left and right with the pointing device to determine the participants in the front view. The user's operation is transmitted to the meeting device 60 , and the meeting device 60 changes the front angle of 360° in the horizontal direction to create a panorama image and transmits it to the terminal device 10 .

When the user presses the recording start button 205, the information recording application 41 displays the recording setting screen 210 of FIG.

FIG. 14 is an example of a recording setting screen 210 displayed by the information recording application 41. As shown in FIG. On the recording setting screen 210, the user can set whether to record the panorama image and the speaker image created by the meeting device 60, and the desktop screen of the terminal device 10 or the screen of the operating application (whether to include them in the recording). If the information recording application 41 does not record the panorama image, the speaker image, the desktop screen, or the screen of the application that is running, only the sound (the sound output by the terminal device 10 + the sound collected by the meeting device 60) is displayed. Recorded.

The camera toggle button 211 is a button for switching on and off recording of the panorama image and the speaker image created by the meeting device 60 . The camera toggle button 211 may be set to individually record the panorama image and the speaker image.

The PC screen toggle button 212 is a button for switching on and off recording of the desktop screen of the terminal device 10 and the screen of the application running on the terminal device 10 . With the PC screen toggle button 212 on, the desktop screen is recorded.

When the user wants to record the screen of an application, the application is further selected in the application selection field 213 . The name of the application being executed by the terminal device 10 is displayed in the application selection field 213 in a pull-down format. Users can choose which apps to record. This application name is acquired by the information recording application 41 from the OS. The information recording application 41 can display only applications having a UI (screen) among the applications being executed. Teleconferencing application 42 may be included among the selected applications. Therefore, the information recording application 41 can also record materials displayed by the teleconference application 42, participants at each base, etc., as moving images. In addition, applications displayed in the pull-down menu include presentation applications, word processor applications, spreadsheet applications, web browser applications, and the like. Therefore, the user can flexibly select the screens of the application to be included in the combined image video.

Also, when recording by app, users can select multiple apps. The information recording application 41 can record screens of all selected applications.

When both the camera toggle button 211 and the PC screen toggle button 212 are off, the message "Audio only is recorded" is displayed in the recorded content confirmation window 214. FIG. This sound is the sound output by the terminal device 10 (the sound received by the teleconference application 42 from the other site 101) and the sound collected by the meeting device 60. FIG. In other words, if a remote conference is being held, the audio from the remote conference application 42 and the audio from the meeting device 60 are saved regardless of image recording. However, the user may be able to selectively stop saving the audio of the teleconference application 42 and the audio of the meeting device 60 in user settings.

A combined image moving image is recorded as follows according to the combination of ON and OFF of the camera toggle button 211 and the PC screen toggle button 212 . Also, the combined image moving image is displayed in real time in the recorded content confirmation window 214 .

When the camera toggle button 211 is on and the PC screen toggle button 212 is off, the panorama image captured by the meeting device 60 and the speaker's image are displayed in the recorded content confirmation window 214 .

When the camera toggle button 211 is off and the PC screen toggle button 212 is on (the screen is also selected), the desktop screen and the screen of the selected application are displayed in the recorded content confirmation window 214 .

・When the camera toggle button 211 is on and the PC screen toggle button 212 is on, the panoramic image and speaker image captured by the meeting device 60, the desktop screen, and the screen of the selected application are displayed in the recorded content confirmation window 214. displayed side by side.

Therefore, the panoramic image, the speaker image, and the application screen may not be recorded, or the panoramic image, the speaker image, and the application screen may not be recorded at all. The image created by is called a combined image movie.

Additionally, the recording settings screen 210 has a check box 215 with the message "Automatically transcribe recording after upload." The recording setting screen 210 also has a start recording now button 216 . When the user checks the check box 215, the text data obtained by converting the remarks made during the teleconference is attached to the recorded moving image. In this case, the information recording application 41 uploads the voice to the information processing system 50 together with a request for conversion into text data after the recording is finished. Also, when the user presses the recording start button 216 now, the recording screen is displayed.

FIG. 15 is an example of a conference list screen 230 displayed by the information recording application 41. As shown in FIG. The conference list screen 230 is a list of conferences, and can display a list of recorded information recorded in remote conferences. It also includes conferences held only in a certain conference room, regardless of remote conferences. The conference list screen 230 displays conference information for which the log-in user has viewing authority in the conference information storage unit 5001 . Information of the moving image recording information stored in the information storage unit 1001 may be further integrated.

The conference list screen 230 is displayed when the user selects the conference list tab 231 on the initial screen 200 of FIG. The conference list screen 230 displays a list 236 of record information that this user has viewing authority. The conference creator (minutes creator) can set viewing authority for the participants. The conference list may be a list of stored recording information, a list of conference schedules, or a list of conference data.

The conference list screen 230 has check boxes 232 , update dates and times 233 , titles 234 , and statuses 235 .

• Check box 232 accepts selection of a recording file. Check box 232 is used when the user wants to delete recorded files in bulk.

The update date/time 233 indicates the start and end times of recording of the combined image moving image. If it has been edited, the date and time of editing may be used.

・The title is 234, the title of the meeting (agenda, etc.). It may be transcribed from the meeting information, or may be set by the user.

· The status 235 indicates whether or not the combined image moving image has been uploaded to the information processing system 50 . If it has not been uploaded, "Local PC" is displayed, and if it has been uploaded, "Uploaded" is displayed. If not already uploaded, an upload button will be displayed. If there is an unuploaded combined image moving image, the information recording application 41 may automatically upload it when the user logs into the information processing system 50 .

When the user selects an arbitrary title or the like from the combined image video list 236 with a pointing device, the information recording application 41 displays a recording/playback screen, but this is omitted in this embodiment. On the recording/playback screen, it is possible to play back a combined image moving image.

In addition, it is desirable that the user can narrow down the conferences based on update dates, titles, keywords, and the like. In addition, if it is difficult to find the appropriate meeting due to the large number of displayed meetings, the user can enter words and phrases in the search function to narrow down the recorded information based on the words and phrases included in the meeting remarks and titles. It is desirable to be able to The search function allows the user to find desired recorded information in a short time even when there is a large amount of recorded information. Also, on the conference list screen, the user may be able to sort the conferences by update date and time or by title.

<Screen example presented by the meeting record confirmation section>
FIG. 16 is an example of a conference record confirmation screen 240 presented by the conference record confirmation unit 59 and displayed by the terminal device 10 .

The conference record confirmation screen 240 has an arbitrary number of segment display sections 130-1 to 130-n. An arbitrary segment display portion among the segment display portions 130-1 to 130-n will be simply referred to as a segment display portion 130 hereinafter.

Each segment display section 130 has at least a speech recognition result display section 133 , a site identification information display section 131 and a synthesized speech reproduction section 132 . The speech recognition result display unit 133 displays the speech recognition result D. FIG. The site identification information display section 131 displays the site identification information E corresponding to the speech recognition result D. FIG. The synthesized speech reproducing unit 132 has at least a play button for playing back the speech, and the user presses the play button to reproduce the synthesized speech C corresponding to the speech recognition result D displayed on the speech recognition result display unit 133. can be played. In addition, the synthesized speech reproduction unit 132 may have functions such as pause, double-speed reproduction, reproduction position change function using a seek bar, arbitrary time skip function, and the like.

By displaying the voice recognition result D and the base identification information E corresponding to the voice recognition result D in association with each other, the user can be promoted to understand the conference record, and misrecognition due to the difference in speaking bases can be prevented. . By being able to reproduce the synthesized speech C as voice, the user can more easily check the minutes.

In FIG. 16, the point identification information display section 131 and the synthesized speech reproduction section 132 on the right side and the point identification information display section 131 and the synthesized speech reproduction section 132 on the left side indicate the difference between the points. For example, the site identification information display section 131 and the synthesized speech reproduction section 132 are displayed on the right side when the statement is from the own site, and are displayed on the left side when the statement is from another site. By doing this, the user can determine the base at a glance.

It is preferable that the information displayed in the speech recognition result display section 133 and the base identification information display section 131 can be edited arbitrarily by the user later. In addition, it is preferable that an arbitrary portion of the segment display portions 130-1 to 130-n can be deleted by the user's operation. This allows the user to arbitrarily modify or delete the recognition result character string when there is an error in the speech recognition result or when there is unnecessary text data in the minutes.

When the user edits (changes) the site identification information display section 131, the display control section 13 moves the location of the site identification information display section 131 and the synthesized speech reproduction section 132 from left to right or from right to left depending on the edited site. to the left to display. The screen generation unit 53 may perform the change processing regarding this screen display. When the user edits the base identification information E, the conference record confirmation unit 59 receives the edited content from the terminal device 10 and changes the base identification information E of the text data shown in FIG.

<Judgment of base>
FIG. 17 is a diagram for explaining base determination conditions. 17A, 17B, and 17C show examples of sound pressure information of own site audio A and other site audio B, respectively. The sound pressure is the amplitude of the waveforms shown in FIGS. 17A, 17B, and 17C, and is obtained by converting the input voltage V _IN into decibel full scale (dBFS) using Equation (1). That is, the decibel full scale is the sound pressure. 17(a), (b) and (c) are obtained in this way.

デシベルフルスケールを用いることにより、音声データの音圧を信号レベル(入力電圧)から換算ですることができる。自拠点音圧情報Ａ'、他拠点音圧情報Ｂ'の導出方法としては音圧(dBFS)の実効値(RMS値)が好ましいが、ピーク値でもよく、システムごとに最適な方法を選択可能である。

By using the decibel full scale, the sound pressure of audio data can be converted from the signal level (input voltage). As a method of deriving own site sound pressure information A' and other site sound pressure information B', the effective value (RMS value) of the sound pressure (dBFS) is preferable, but the peak value is also acceptable, and the optimum method can be selected for each system. is.

The peak value is a method of using the peak value (maximum value) of an audio waveform as a representative value of sound pressure information of certain audio data. The effective value (RMS value) is the value obtained by converting the effective value V _IN -rms of the waveform of the input voltage V _IN of a certain sound into sound pressure (dBFS) using the above equation (1), which is representative of the sound pressure information. This is the method used as a value. The rms value can be calculated using equation (2).

ピーク値を用いた場合は、例えば図１７（ｂ）のように自拠点で話者が発言しており、他拠点からの発言はないが他拠点で突発的で大きな物音がした際には突発的で大きな物音に影響され拠点判断として拠点識別情報Ｅが他拠点と判断されてしまう可能性がある。しかし、実効値(RMS値)を用いることで、音圧が平均化され突発的な音の影響を受けにくくなる。

When the peak value is used, for example, as shown in FIG. There is a possibility that the site identification information E may be determined to be another site as a site determination due to the influence of a loud loud noise. However, by using the effective value (RMS value), the sound pressure is averaged, making it less susceptible to sudden sounds.

Next, an example of conditions for the site determination processing unit 64 to determine the site will be shown. As a condition for site determination, condition 2 is preferable to condition 1 described below, but the condition is not limited to these, and an optimum condition can be selected for each system. In the site determination conditions below, the own site sound pressure information A' is simply described as A', and the other site sound pressure information B' is simply described as B'. ) etc. can be used.

<<Base Judgment Conditions 1-1>>
(1) The site determination processing unit 64 compares the values of A' and B', and determines that the statement was made at its own site if A' is greater than B'.

(2) The site determination processing unit 64 compares the values of A' and B', and determines that the speech was made at another site if B' is greater than A'.

(3) When the sound pressures are the same, the site determination processing unit 64 determines "NA (unknown)".

FIG. 18 is a flow chart showing the above determination example. In the case of NA, generally speaking, the recognition result character string is also deleted because the base identification information E is unknown. However, the site determination processing unit 64 may leave the recognition result character string, or may switch whether to leave the recognition result character string determined as NA according to the site determination condition.

FIG. 18 is an example of a flow chart for explaining a method of determining a site based on the site determination condition 1-1.

As described above, the site determination processing unit 64 divides the voice into own site voice A and other site voice B according to a predetermined rule, and determines the magnitude relationship between the values of A' and B' (S101).

If A'>B', the base determination processing unit 64 determines that the speech was made at the own base (S102).

If A'<B', the site determination processing unit 64 determines that the speech was made at another site (S103).

The site determination processing unit 64 determines that it is unknown when A'=B' (S104).

<<Base Judgment Conditions 1-2>>
(1) The site determination processing unit 64 compares the values of A' and B', and determines that the statement was made at its own site if A' is greater than B'.

(2) The base determination processing unit 64 compares the values of A' and B', and if B' is equal to or greater than A', it determines that the speech was made at another base (if the sound pressure is base”).
According to the site determination condition 1-2, by comparing A' and B' and determining the louder voice as the site, it is possible to match the voice recognition result and the site identification information E with a high probability. In addition, since the voice of other sites is received via the remote communication system, the voice may be slightly reduced due to the noise canceling function of the web conferencing system. may be smaller. Therefore, if A' and B' have the same sound pressure according to site determination condition 1-2, misjudgment of the site can be reduced by determining that the site is another site.

FIG. 19 is an example of a flowchart for explaining a method of determining a site based on the site determination condition 1-2.

As described above, the site determination processing unit 64 divides the audio into own site audio A and other site audio B according to a predetermined rule, and determines whether A' is greater than B' (S111).

If A'>B', the base determination processing unit 64 determines that the speech was made at the own base (S112).

If A'≦B', the site determination processing unit 64 determines that the statement was made at another site (S113).

<<Base Judgment Condition 2>>
(1) If the value of B' is greater than or equal to the noise threshold X and the value of B' is greater than the value of A', the site determination processing unit 64 determines that the speech was made at another site.

(2) If the value of A' is equal to or greater than the noise threshold value X and A' is greater than the value of B', the site determination processing unit 64 determines that the speech was made at its own site.

(3) If both the values of A' and B' are less than the noise threshold value X, the site determination processing unit 64 determines that neither site has spoken (N/A (unknown)).

According to Condition 2, the site determination processing unit 64 first compares the magnitudes of A' and B' with the noise threshold value X, and then compares the magnitudes of A' and B' and selects the larger site as the site identification information. Return as E. As a result, it is possible to prevent the site from being erroneously identified due to the influence of environmental noise in an environment with environmental noise.

An appropriate value for the noise threshold X can be selected depending on the system, but when using the peak value of the decibel full scale as sound pressure information, use about -40 dBFS, and use the effective value (RMS value) of the decibel full scale as the sound pressure information. -50 dBFS is preferable.

FIG. 20 is an example of a flowchart for explaining a method of determining a site based on the site determination condition 2. FIG.

The site determination processing unit 64 divides the voice into own site voice A and other site voice B according to a predetermined rule. The site determination processing unit 64 determines whether the value of B' is equal to or greater than the noise threshold value X (S121). If the determination in step S121 is Yes, the base determination processing unit 64 determines whether or not the value of B' is greater than A' (S122).

If the determination in step S122 is Yes, the site determination processing unit 64 determines that there is a statement at another site (S123).

If the determination in step S121 or S122 is No, the base determination processing unit 64 determines whether or not the value of A' is equal to or greater than the noise threshold value X (S124).

If the determination in step S124 is Yes, the base determination processing unit 64 determines whether or not the value of A' is greater than B' (S125).

If the determination in step S125 is YES, the site determination processing unit 64 determines that the statement was made at its own site (S126).

If the determination in step S124 or S125 is No, the base determination processing unit 64 determines that the base is unknown (S127).

It should be noted that the order of determination between the other sites and the own site may be reversed.

<<Judgment example>>
An example of determination when base determination conditions 1 and 2 are applied to the sound pressure in FIG. 16 will be described.

In FIG. 17(a), when the site determination condition 1-1 is used and the peak value is used as the sound pressure information, the site identification information E is determined as the own site.

In FIG. 17(a), when site determination condition 1-2 is used and effective value (RSM value) is used as sound pressure information, site identification information E is determined to be the site itself.

In FIG. 17A, when the noise threshold is -40 dBFS, site determination condition 2 is used, and the peak value is used as the sound pressure information, the site identification information E is determined to be the site itself.

In FIG. 17A, when the noise threshold is -50 dBFS, site determination condition 2 is used, and effective value (RSM value) is used as sound pressure information, site identification information E is determined to be the site itself.

By using the site determination conditions 1 and 2, it is possible to determine, as the site identification information E, the own site that is presumed to have made a statement in the site determination.

In FIG. 17(b), when the site determination condition 1-1 is used and the peak value is used as the sound pressure information, the site identification information E is determined to be another site.

In FIG. 17(b), when site determination condition 1-2 is used and an effective value (RSM value) is used as sound pressure information, the site is determined to be the site itself as site identification information.

In FIG. 17(b), when the noise threshold is -40 dBFS, site determination condition 2 is used, and the peak value is used as the sound pressure information, the site identification information E is determined to be another site.

In FIG. 17(b), when the noise threshold is -50 dBFS, site determination condition 2 is used, and the effective value (RSM value) is used as the sound pressure information, the site identification information E is determined to be the site itself.

By using location determination condition 2, location determination results are less susceptible to noise.

In FIG. 17(c), when the site determination condition 1-1 is used and the peak value is used as the sound pressure information, the site identification information E is determined as the own site.

In FIG. 17(c), when site determination condition 1-2 is used and effective value (RSM value) is used as sound pressure information, site identification information E is determined to be the site itself.

In FIG. 17(c), when the noise threshold is -40 dBFS, site determination condition 2 is used, and the peak value is used as the sound pressure information, the site identification information E is determined to be the site itself.

In FIG. 17(c), when the noise threshold is -50 dBFS, site determination condition 2 is used, and the effective value (RSM value) is used as sound pressure information, the site identification information E is N/A (site identification information None).

By using the effective value (RMS value) as the sound pressure information, the site judgment result is less likely to be affected by sudden sounds.

<Association of Site Identification Information E, Speech Recognition Result D, and Synthesized Speech C>
In FIG. 12, IDs are used to associate the base identification information E, the speech recognition result D, and the synthesized speech C, but as a method of associating the same ID with the base identification information E, the speech recognition result D, and the synthesized speech C, Methods such as a queue method and a time stamp method are conceivable.

<< queue method >>
FIG. 21 is a diagram for explaining how the information processing system 50 associates the site identification information E, the speech recognition result D, and the synthesized speech C by the cue method. A queue is a data structure in which elements are arranged in the order in which they were received, and the elements that were put in first are taken out in order. In the queue method, the site identification information E, the voice recognition result D, and the synthesized voice C are input to the site determination result recording unit 58, the voice recognition result recording unit 57, and the voice recording unit 56, respectively. 50 sets the same ID to the site identification information E, the speech recognition result D, and the synthesized speech C. FIG.

The conference record confirmation screen 240 causes the segment display section 130-1, which is the first segment display section 130, to display the base identification information E-1, the speech recognition result D-1, and the synthesized speech C-1 in association with each other. The conference record confirmation screen 240 causes the segment display section 130-n, which is the n-th segment display section 130, to display the site identification information En, the speech recognition result Dn, and the synthesized speech Cn in association with each other. .

The cue method can display the base identification information E, the speech recognition result D, and the synthesized speech C in correspondence with each other to the user by a simple algorithm.

<< Timestamp method >>
FIG. 22 is a diagram for explaining how the information processing system 50 associates the site identification information E, the speech recognition result D, and the synthesized speech C by the time stamp method. The time stamp method is a method of adding the same time stamp to the base identification information E, the speech recognition result D, and the synthesized speech C as metadata. A time stamp is a method for reliably and easily confirming the identity of data based on the time stamp information F described in the time stamp such as time and hash value. The location identification information E and the time stamp for the synthesized speech C are added by the meeting device 60 . The information processing system 50 adds a time stamp to the speech recognition result D. FIG.

The information processing system 50 sets the same ID to the site identification information E, the speech recognition result D, and the synthesized speech C having the same time stamp information F. FIG. The conference record confirmation screen 240 displays the site identification information E, the speech recognition result D, and the synthesized speech C having the same time stamp information F in association with each other.

The conference record confirmation screen 240 displays base identification information E-1 and voice recognition result D-1, which are data having time stamp information F-1, in the segment display section 130-1, which is the first segment display section 130. Synthesized speech C-1 is associated and displayed. The meeting record confirmation screen 240 displays base identification information En, which is data having time stamp information Fn, and speech recognition result Dn in the segment display section 130-n, which is the n-th segment display section 130. Synthesized speech Cn is associated and displayed.

The time stamp method can disclose to the user the base identification information E, the speech recognition result D, and the synthesized speech C in association with each other using a simple algorithm. Furthermore, even if the order in which the site identification information E, the voice recognition result D, and the synthesized voice C enter the site determination result recording unit 58, the voice recognition result recording unit 57, and the voice recording unit 56 are switched due to data delay, etc., the The base identification information E, the speech recognition result D, and the synthesized speech C can be synchronized.

When generating the time stamp, it is preferable to generate it at the timing of acquiring own-site audio A or other-site audio B. FIG. However, the meeting device 60 may generate the synthesized speech C at the timing of generating the synthesized voice C, the timing of acquiring the own site sound pressure information A′ or the other site sound pressure information B′, or the timing of generating the site identification information E. The generation timing of the time stamp is not limited to these, as long as it is the timing at which the site identification information E, the speech recognition result D, and the synthesized speech C can be associated with each other.

<Operation procedure>
Next, operations and processes performed by the recording information creation system 100 based on the above configuration will be described.

<<Save Combined Image Movie>>
With reference to FIG. 23, processing for saving a combined image moving image will be described. FIG. 23 is an example of a sequence diagram showing a procedure for the information recording application 41 to record the panorama image, the speaker image, and the screen of the application. FIG. 23 illustrates processing of real-time speech recognition.

S1: The user activates the information recording application 41 on the terminal device 10 and connects the terminal device 10 (the information recording application 41) to the information processing system 50. FIG. If the access token has expired, the display control unit 13 displays the login screen. The user inputs authentication information (for example, user ID and password) for logging into the tenant into the information recording application 41 . The operation reception unit 12 of the information recording application 41 receives the input.

S2: The communication unit 11 of the information recording application 41 specifies authentication information and transmits a login request to the information processing system 50 .

S3: The communication unit 51 of the information processing system 50 receives the login request, and the authentication unit 52 authenticates the user based on the authentication information. Assume here that the authentication has succeeded. The communication unit 51 of the information processing system 50 transmits the access token 1 to the information recording application 41 . Although not shown in the drawing, the communication unit 51 attaches the access token 1 to communications with the information processing system 50 thereafter. The access token 1 is associated with the authority of the logged-in user.

S4: Similarly, the user logs into the storage service system 70 as well. This is because the recorded information is stored in the storage service system 70 . The user enters authentication information (eg, user ID, password) for logging into the storage service system 70 . The operation reception unit 12 of the information recording application 41 receives the input.

S5: The communication unit 11 of the information recording application 41 specifies authentication information and transmits a login request to the information processing system 50 .

S6: The communication unit 51 of the information processing system 50 receives the login request, and transfers the login request to the storage service system 70 since it is a login request to the storage service system 70 .

S7: The storage service system 70 authenticates the user based on the authentication information. Assume here that the authentication has succeeded. The storage service system 70 sends the access token 2 to the information processing system 50 .

S8: The communication unit 51 of the information processing system 50 receives the access token 2 and transmits it to the information recording application 41 . Although not shown in the figure, the communication unit 51 attaches the access token 2 to communications with the storage service system 70 thereafter. The access token 2 is associated with the authority of the logged-in user.

S21: Subsequently, the user operates the remote conference application 42 to start a remote conference. Here, it is assumed that the remote conference applications 42 of the local site 102 and the other site 101 have started the remote conference. The teleconference application 42 of the local site 102 transmits the image captured by the camera of the terminal device 10 and the sound collected by the microphone to the teleconference application 42 of the other site 101 . The teleconference application 42 of the other site 101 displays the received image on the display and outputs the received voice from the speaker. Similarly, the teleconference application 42 of the other site 101 transmits the image captured by the camera of the terminal device 10 and the sound collected by the microphone to the teleconference application 42 of the own site 102 . The teleconference application 42 of the own base 102 displays the received image on the display and outputs the received voice from the speaker. Each remote conference application 42 repeats this to realize a remote conference.

S22: The user makes settings related to recording on the recording setting screen 210 of the information recording application 41 shown in FIG. The operation reception unit 12 of the information recording application 41 receives the setting. Here, it is assumed that both the camera toggle button 211 and the PC screen toggle button 212 are on.

When a remote conference is reserved in advance, the user can display a list of remote conferences as shown in FIG. 15 and select the remote conference with which the combined image video is associated. Since the user has already logged into the information processing system 50, the information processing system 50 identifies remote conferences to which the logged-in user has viewing authority. Since the information processing system 50 transmits a list of identified remote conferences to the terminal device 10, the user selects a remote conference that is being held or will be held in the future. As a result, information related to the teleconference, such as the conference ID, is determined.

In addition, even if the user does not reserve a teleconference in advance, the user can create the conference when creating the combined image video. A case will be described below in which the information recording application 41 creates a meeting when creating a combined image moving image and acquires the meeting ID from the information processing system 50 .

S23: The user instructs the information recording application 41 to start recording (start recording now button 216). The operation reception unit 12 of the information recording application 41 receives the instruction. The display control unit 13 displays a screen during recording.

S24: Since the remote conference has not been selected (because the conference ID has not been determined), the communication section 11 of the information recording application 41 transmits a remote conference creation request to the information processing system 50 .

S25: The communication unit 51 of the information processing system 50 receives the remote conference creation request, the conference information acquisition unit 54 acquires a unique conference ID numbered by the conference management system 9, and the communication unit 51 receives the conference ID as information. Send to the recording application 41 .

S<b>26 : The meeting information acquisition unit 54 also transmits the storage destination of the combined image moving image (the URL of the storage service system 70 ) to the information recording application 41 via the communication unit 51 .

S27: When the communication unit 11 of the information recording application 41 receives the meeting ID and the storage destination of the recorded file, the video storage unit 17 determines that preparation for recording has been completed, and starts recording.

S28: The application screen acquisition unit 14 of the information recording application 41 requests the screen of the application selected by the user from the application (more specifically, the application screen acquisition unit 14 acquires the screen of the application via the OS). ). In FIG. 23, it is assumed that the application selected by the user is the teleconference application 42 .

S29: The video storage unit 17 of the information recording application 41 notifies the meeting device 60 to start recording via the device communication unit 16. At the time of notification, the moving image storage unit 17 may also notify that the camera toggle button 211 is on (request for panorama image and speaker image). The meeting device 60 transmits the panorama image and the speaker image to the information recording application 41 regardless of whether or not there is a request.

S30: When the device connection unit 67 of the meeting device 60 receives the start of recording, it assigns a unique recording ID and returns the recording ID to the information recording application 41 . Note that the recording ID may be numbered by the information recording application 41 or obtained from the information processing system 50 .

S31: The teleconference service system 90 repeatedly transmits the voice data and image data transmitted from other bases to the teleconference application.

S32: The audio acquisition unit 15 of the information recording application 41 acquires audio data output by the terminal device 10 (audio data received by the teleconference application 42).

S<b>33 : The device communication section 16 transmits the voice data acquired by the voice acquisition section 15 and the synthesis request to the meeting device 60 .

S34: The device connection section 67 of the meeting device 60 receives the audio data and the synthesis request. The sound collector 68 always collects surrounding sounds. The site determination processing unit 64 divides the other site audio B received by the device connection unit 67 and the own site audio A collected by the sound collection unit 68 according to a predetermined rule, and determines the site based on the sound pressure of each. do.

S35: Next, the voice synthesizing unit 65 synthesizes the local site voice A (surrounding voice data) collected by the sound collecting unit 68 and the other site voice B received by the device connecting unit 67 . Therefore, synthesized speech C is generated in a divided state. For example, the speech synthesizing unit 65 adds the local site's voice A and the other site's voice B together. Since the clear voice around the meeting device 60 is recorded, the text conversion accuracy of the voice especially around the meeting device 60 (conference room side) is improved.

This speech synthesis can also be performed by the terminal device 10 . However, by distributing the recording function to the terminal device 10 and the audio processing to the meeting device 60, the load on the terminal device 10 and the meeting device 60 can be reduced. The recording function may be distributed to the meeting device 60 and the audio processing may be distributed to the terminal device 10 .

S36 : The text conversion request unit 66 of the meeting device 60 transmits the voice recognition request (divided synthesized voice C) and the site identification information E to the information processing system 50 via the communication unit 61 .

S37: The communication unit 51 of the information processing system 50 receives the voice recognition request (divided synthetic voice C) and the location identification information E, and the voice recognition unit 55 transmits the voice recognition request (divided synthetic voice C). Send to the recognition service system and get the recognition result string.

S38: The information processing system 50 returns the recognition result character string, voice data, and base identification information E to the information recording application 41. FIG. Since the information processing system 50 returns these to the information recording application 41, the meeting device 60 attaches its own identification information in step S36. The information recording application 41 also sets the IP address of the terminal device 10 and the identification information obtained from the meeting device 60 in the information processing system 50 in advance. By doing so, the information processing system 50 can identify the terminal device 10 based on the identification information of the meeting device 60 .

S39: The site determination result recording unit 58, the voice recognition result recording unit 57, and the voice recording unit 56 of the information processing system 50, via the communication unit 51, combine the recognition result character string, the voice data, and the site identification information E into a combined image video. Save to the same destination as A conference ID is attached to these.

S40: Further, the panorama image creation unit 62 of the meeting device 60 creates a panorama image, and the speaker image creation unit 63 creates a speaker image.

S<b>41 : The device communication unit 16 of the information recording application 41 repeatedly acquires the panoramic image and the speaker image from the meeting device 60 . In addition, the device communication unit 16 repeatedly requests the meeting device 60 to acquire synthesized voice data. These acquisitions may be performed by the device communication section 16 requesting the meeting device 60 . Alternatively, the meeting device 60 may automatically transmit the panorama image and the speaker image upon receiving the fact that the camera toggle button 211 is on. The meeting device 60 that receives the voice data synthesis request may automatically transmit the synthesized voice data to the information recording application 41 .

S42: The display control unit 13 of the information recording application 41 displays the application screen, the panorama image, and the speaker image side by side on the recording screen 220 . In addition, the video storage unit 17 of the information recording application 41 combines the application screen acquired from the teleconference application 42, the panorama image, and the speaker image, and stores them as a combined image video. That is, the video storage unit 17 creates a combined image by combining the screen of the application that is repeatedly received, the panorama image, and the speaker image, and designates the combined image in the frame that constitutes the combined image moving image. Create image videos. Also, the video storage unit 17 stores audio data received from the meeting device 60 .

The information recording application 41 repeats the above steps S32 to S42.

S43: When the remote conference ends and recording is no longer necessary, the user instructs the information recording application 41 to end recording (for example, the recording end button 227). The operation reception unit 12 of the information recording application 41 receives the instruction.

S44: The device communication section 16 of the information recording application 41 notifies the meeting device 60 of the end of recording. The meeting device 60 continues to create panorama images and speaker images and to synthesize voices. However, the meeting device 60 may change whether processing is disabled, such as by changing the resolution or fps depending on whether recording is in progress.

S45: The moving image storage unit 17 of the information recording application 41 creates a combined image moving image with sound by combining audio data with the combined image moving image. If the panorama image, speaker image, and application screen are not saved at all, the audio data may be independent.

S46: The upload unit 20 of the information recording application 41 stores the combined image moving image in the storage destination of the combined image moving image via the communication unit 11 . In the recording information storage unit 5002, the combined image moving image is associated with the meeting ID and the recording ID. Uploaded is recorded in the combined image video.

Since the save destination is notified to the user, the user can share the combined image video with the participants by notifying the save destination by e-mail or the like. Even if the devices that create the combined image moving image, audio data, and text data are different, they can be collected and stored in one storage location so that users can easily view them later.

Note that the processing of steps S32 to S42 may not be performed in the order shown in FIG. 23, and the synthesis of the audio data and the storage of the combined image moving image may be performed in sequence.

<< Voice recognition after recording >>
Next, with reference to FIG. 24, a processing procedure for recognizing voice after the end of recording will be described. FIG. 24 is an example of a sequence diagram showing a procedure for the information recording application 41 to record the panorama image, the speaker image, and the screen of the application. FIG. 24 mainly explains differences from FIG. Steps S1-S35 may be the same as steps S1-S35 in FIG. However, there are no steps S36 to S39 in which the meeting device 60 requests speech recognition from the information processing system 50 .

S51: The device connection unit 67 of the meeting device 60 transmits the panorama image, speaker image, voice data (synthetic voice C), and base identification information E to the information recording application 41 . Therefore, at this point, there is no recognition result character string, but the voice data is segmented.

S52: The device communication unit 16 of the information recording application 41 repeatedly acquires the panorama image, the speaker image, the voice data, and the base identification information E from the meeting device 60. FIG. The display control unit 13 of the information recording application 41 displays the application screen, the panorama image, and the speaker image side by side on the recording screen 220 .

S53, S54: Recording end processing may be the same as steps S43 and S44 in FIG.

S55: When the user checks the check box 215 associated with "automatically transcribe after uploading a recording" on the recording setting screen 210, the audio data processing unit 18 sends information about conversion of audio data to text data. Request to processing system 50 . Specifically, the audio data processing unit 18 designates the URL of the storage destination and the base identification information E via the communication unit 11, and together with the conference ID and the recording ID, the audio data combined into the combined image video (divided to the information processing system 50 .

S56: The communication unit 51 of the information processing system 50 receives the voice data conversion request, and the voice recognition unit 55 uses the voice recognition service system 80 to convert the voice data into text data.

S57: The communication unit 51 acquires the recognition result character string from the speech recognition service system.

S58: The site determination result recording unit 58, the voice recognition result recording unit 57, and the voice recording unit 56 transmit the recognition result character string, voice data (segmented synthesized voice C), and site identification information E via the communication unit 51. Save to the same destination as the combined image movie.

S59: The moving image storage unit 17 of the information recording application 41 creates a combined image moving image with sound by combining audio data with the combined image moving image. If the panorama image, speaker image, and application screen are not saved at all, the audio data may be independent.

S60: The upload unit 20 of the information recording application 41 stores the combined image moving image in the storage destination of the combined image moving image via the communication unit 11 . In the recording information storage unit 5002, the combined image moving image is associated with the meeting ID and the recording ID. Uploaded is recorded in the combined image video.

In this way, even if the information recording application 41 performs voice recognition after the end of recording, the base can be determined.

<<Edit base identification information>>
FIG. 25 is an example of a sequence diagram for explaining the process of editing the site identification information E by the user.

S71: The user connects the terminal device 10 to the information processing system 50 and displays the conference list screen. The conference list screen displays a list of conferences for which the logged-in user has viewing authority.

S72: The user selects a meeting and inputs an operation to display the meeting record confirmation screen 240. FIG. The operation accepting unit 12 accepts an operation.

S73: The communication unit 11 of the terminal device 10 designates the conference ID and requests the information processing system 50 to display the conference record confirmation screen 240 .

S74, S75: The communication unit 51 of the information processing system 50 receives the request for the conference record confirmation screen 240, and acquires the recognition result character string specified by the conference ID, voice data, and site identification information E from the storage service system 70. do.

S76: The conference record confirmation unit 59 of the information processing system 50 generates the conference record confirmation screen 240 using the recognition result character string, the voice data, and the base identification information E. FIG. The conference record confirmation unit 59 aligns the recognition result character strings to the left and right sides of the conference record confirmation screen 240 in accordance with the base identification information E. FIG.

S77: The communication unit 51 of the information processing system 50 transmits the screen information of the conference record confirmation screen 240 to the terminal device 10. FIG. The communication unit 11 of the terminal device 10 receives the screen information of the conference record confirmation screen 240, and the display control unit 13 displays it.

S78: When the user changes, for example, the base identification information E, the operation accepting unit 12 accepts the change.

S79: The communication unit 11 of the terminal device 10 designates the conference ID and transmits the edited result to the information processing system 50 .

S80: The communication unit 51 of the information processing system 50 receives the editing result, and the site determination result recording unit 58 changes the site identification information E of the text data in the storage service system 70 according to the editing content.

<Main effects>
In this way, in the combined image moving image, the surrounding panorama image including the user, the speaker image, and the screen of the application displayed during the teleconference are displayed and recorded. Since speech recognition is performed on the synthesized speech C, the processing load on the speech recognition service system can be reduced as compared with separate speech recognition. In addition, since the site where the synthetic voice C was uttered is determined based on the sound pressure information, it is possible to record the site where the voice data was uttered.

<Other application examples>
Although the best mode for carrying out the present invention has been described above using examples, the present invention is by no means limited to such examples, and various modifications can be made without departing from the scope of the present invention. and substitutions can be added.

For example, in the present embodiment, only the voice of one's own site or the voice of another site is determined, but if the site ID is attached to the voice of the other site, the voice of which site among the other sites is recorded. can also

Further, in the present embodiment, the panorama image, the speaker image, and the screen of the application are combined and saved as a combined image video, but these may be saved as separate videos. In this case, the information recording application 41 arranges the panorama image, the speaker image, and the screen of the application on the reproduction screen when reproducing the moving image.

Also, the terminal device 10 and the meeting device 60 may be integrated. The meeting device 60 may be externally attached to the terminal device 10 . Also, the meeting device 60 may be one in which a omnidirectional camera, a microphone, and a speaker are connected by a cable.

Meeting devices 60 may also be arranged at other bases 101 . Other sites 101 separately use the meeting device 60 to create combined image moving images and text data. Also, a plurality of meeting devices 60 may be arranged at one site. In this case, a plurality of recorded information are created for each meeting device 60 .

Also, the arrangement of the panoramic image 203, the speaker image 204, and the application screen in the combined image moving image used in this embodiment is merely an example. The panorama image 203 may be placed at the bottom and the speaker image 204 may be placed at the top. Alternatively, the user may change the arrangement, or the panorama image 203 and the speaker image 204 may be individually switched between display and non-display during playback.

Also, the configuration example of FIG. 8 and the like is divided according to main functions in order to facilitate understanding of the processing by the terminal device 10, the meeting device 60, and the information processing system 50. FIG. The present invention is not limited by the division method or name of the unit of processing. The processing of the terminal device 10, the meeting device 60, and the information processing system 50 can also be divided into more processing units according to the content of the processing. Also, one processing unit can be divided to include more processing.

Moreover, the devices described in the examples are only representative of one of several computing environments for implementing the embodiments disclosed herein. In some embodiments, information handling system 50 includes multiple computing devices, such as a server cluster. Multiple computing devices are configured to communicate with each other over any type of communication link, including a network, shared memory, etc., to perform the processes disclosed herein.

Further, the information handling system 50 can be configured to share the disclosed processing steps, such as FIG. 23, in various combinations. For example, a process executed by a given unit may be executed by multiple information processing devices included in the information processing system 50 . Moreover, the information processing system 50 may be integrated into one server device, or may be divided into a plurality of devices.

Each function of the embodiments described above may be implemented by one or more processing circuits. Here, "processing circuitry" as used herein refers to processors programmed by software to perform the functions, such as processors implemented by electronic circuitry, or processors designed to perform the functions described above. It includes devices such as ASICs (Application Specific Integrated Circuits), DSPs (Digital Signal Processors), FPGAs (Field Programmable Gate Arrays), and conventional circuit modules.

10 Terminal Device 50 Information Processing System 60 Meeting Device 100 Recording Information Creation System

Special Table 2014-206896

Claims (12)

A voice processing system in which a terminal device and a device equipped with a microphone communicate,
a speech synthesizing unit that synthesizes a first speech externally received by the terminal device and a second speech collected by the device to generate synthesized speech;
a text conversion request unit that requests an external conversion of the synthesized speech synthesized by the speech synthesis unit into text data;
A speech processing system characterized by comprising: the terminal device receives the first audio from a terminal device at another base via a network;
Before the speech synthesis unit synthesizes the first speech and the second speech, the first speech and the second speech are synthesized based on the sound pressures of the first speech and the second speech. 2. The voice processing system according to claim 1, further comprising a base judgment processing unit for judging a base where each voice is uttered. The base determination processing unit divides the first sound and the second sound according to a predetermined rule,
determining a base based on the sound pressure of the divided first audio and the second audio;
Corresponding the synthetic voice obtained by synthesizing the first voice and the second voice after determining the site, the recognition result character string obtained by converting the synthesized voice, and the site identification information determined by the site determination processing unit. 3. The voice processing system according to claim 2, wherein the voice processing system is stored with an attachment. The site determination processing unit determines a site where the sound pressure of the first voice and the sound pressure of the second voice, whichever is greater, is uttered by synthesizing the first voice and the second voice. 4. The speech processing system according to claim 2 or 3, wherein it is determined that said synthesized speech that has been created is a base where said speech was made. When the sound pressure of the first sound is equal to or greater than a threshold and the sound pressure of the first sound is greater than the sound pressure of the second sound, the base determination processing unit determines that the synthesized sound is different from the sound pressure of the second sound. Judging that it was said at the base,
when the sound pressure of the second voice is equal to or greater than a threshold and the sound pressure of the second voice is greater than the sound pressure of the first voice, determining that the synthesized voice is spoken at its own site;
4. The method according to claim 2, wherein when the sound pressure of the first voice and the sound pressure of the second voice are less than a threshold, it is determined that the site where the synthesized voice was uttered is unknown. audio processing system. The synthesized speech, the recognition result character string obtained by converting the synthesized speech, and the site identification information determined by the site determination processing unit are associated and stored on a network,
Having an information processing system that communicates with the terminal device,
The information processing system is
a meeting record confirmation unit that provides a terminal device with a screen displaying the recognition result character string converted from the synthesized voice and the site identification information determined by the site determination processing unit in association with each other;
3. The speech processing system according to claim 2, wherein the conference record confirmation unit changes the arrangement of the recognition result character string on the screen according to the site identification information. The conference record confirmation unit associates a button for reproducing the synthesized speech with the recognition result character string and the base identification information, and arranges it for each of the divided synthetic speeches,
7. The speech processing system according to claim 6, wherein when said button is pressed, said terminal device reproduces said synthesized speech corresponding to said button. The terminal device
an operation receiving unit that receives editing of the base identification information;
8. The voice processing system according to claim 6, further comprising a communication unit that transmits the edited site identification information to the information processing system. The terminal device creates record information in which a screen acquired from an application running on the terminal device and a surrounding image acquired by the device are combined,
9. The speech processing system according to any one of claims 1 to 8, wherein text data obtained by converting said synthesized speech by speech recognition is stored in association with said recorded information. A time stamp is added to the synthesized speech, the recognition result character string obtained by converting the synthesized speech, and the site identification information determined by the site determination processing unit,
4. The speech processing system according to claim 3, wherein the synthesized speech, the recognition result character string, and the site identification information are stored in association with each other based on the time stamp. A device equipped with a microphone capable of communicating with a terminal device,
a speech synthesizing unit that synthesizes a first speech externally received by the terminal device and a second speech collected by the device to generate synthesized speech;
a text conversion request unit that requests an external conversion of the synthesized speech synthesized by the speech synthesis unit into text data;
A device comprising: A voice processing method performed by a voice processing system in which a terminal device and a device equipped with a microphone communicate,
a step in which a speech synthesizing unit synthesizes a first speech externally received by the terminal device and a second speech collected by the device to generate synthesized speech;
a step in which the text conversion request unit requests an external conversion of the synthesized speech synthesized by the speech synthesis unit into text data;
A speech processing method characterized by comprising:

Images