JP2022113375A - Information processing method and monitoring system - Google Patents

Abstract

To provide an information processing method, a monitoring system, and the like that make a conference or the like via a network smoothly proceed when a defect occurs in voice.SOLUTION: An information processing method for monitoring voice data in a conference system that provides a conference by a plurality of participant terminal devices includes the steps of: acquiring, from the conference system, the voice data transmitted from the plurality of participant terminal devices to the conference system; detecting a defect in the voice data; and outputting, to the conference system, transcript data including text, which is a voice recognition result of the voice data when a defect in the voice data is detected.SELECTED DRAWING: Figure 4

Description

The present invention relates to an information processing method, a monitoring system, and the like.

2. Description of the Related Art Conventionally, there has been known a system that realizes a conference or the like among a plurality of participants using a network such as the Internet. In a conference or the like over a network, the volume or sound quality may be degraded due to network delay or the like. In this case, it becomes difficult to hear the voice, which may hinder the progress of the conference.

Patent Literature 1 discloses a technique for controlling the bandwidth of transmitted and received information in a network conference system. Japanese Patent Laid-Open No. 2002-200001 discloses a method of controlling packet readout according to data reproduction time in a system that provides an electronic conference via a network.

Japanese Patent Application Laid-Open No. 2003-152793 JP-A-2008-227968

The techniques of Patent Literature 1 and Patent Literature 2 are techniques for suppressing the occurrence of audio defects by controlling the bandwidth and packet reading. However, even if these methods are used, there is a possibility that failure occurrence cannot be suppressed depending on the degree of network delay. In the conventional method, there is no disclosure of how to deal with the problem of sound.

According to some aspects of the present disclosure, it is possible to provide an information processing method, a monitoring system, and the like that smoothly proceed with a conference or the like via a network when a problem occurs in audio.
can provide

One aspect of the present disclosure is an information processing method for monitoring audio data in a conference system that provides a conference by a plurality of participant terminal devices, comprising: a step of acquiring voice data from the conference system; a step of detecting a defect in the voice data; and, when a defect in the voice data is detected, transcription data including text that is a voice recognition result of the voice data. to the conference system.

Another aspect of the present disclosure is a monitoring system for monitoring audio data in a conference system providing a conference by a plurality of participant terminals, wherein audio data transmitted from the plurality of participant terminals to the conference system a voice data acquisition unit that acquires from the conference system, a defect detection unit that performs processing for detecting defects in the voice data, and a voice recognition result of the voice data when a defect is detected in the voice data. and a transcript data output unit that outputs transcript data including a certain text to the conference system.

A configuration example of a communication system including a monitoring system. A hardware configuration example of a conference system. Hardware configuration example of the monitoring system. A functional block diagram of the conference system. Functional block diagram of the monitoring system. A hardware configuration example of a participant terminal device. A functional block diagram of a participant terminal device. FIG. 4 is a sequence diagram for explaining the processing of the embodiment; 5 is a flowchart for explaining defect detection processing; An example of a display screen in a conference system using voice. An example of a display screen in a conference system using chat. Another configuration example of a communication system including a monitoring system.

Hereinafter, this embodiment will be described with reference to the drawings. In the drawings, the same or equivalent elements are denoted by the same reference numerals, and overlapping descriptions are omitted. In addition, this embodiment described below does not unduly limit the content described in the claims. Moreover, not all the configurations described in the present embodiment are essential constituent elements of the present disclosure.

1. System Configuration Example 1.1 Overall Configuration FIG. 1 is a diagram showing the configuration of a conference communication system 10 including a monitoring system 200 according to this embodiment. As shown in FIG. 1, the communication system 10 includes a conference system 100, a monitoring system 200, and a plurality of participant terminals 300. FIG. The configuration of the communication system 10 is not limited to that shown in FIG. 1, and modifications such as adding other configurations are possible. For example, although FIG. 1 shows an example in which there are two participant terminal devices 300, the number of participant terminal devices 300 may be three or more.

A communication system 10 is a system used for a conference via a network. Note that the term "meeting" as used here broadly refers to a situation in which communication by a plurality of users is required. That is, the conference in this embodiment is not limited to a narrowly defined conference where a plurality of users exchange opinions, deliberate, and make decisions, and may include forms such as seminars and lectures. A seminar is a study session on a predetermined theme. A lecture is a place where the audience listens to a lecturer's presentation. In addition, the conference of the present embodiment can include various situations in which a given participant speaks and other participants may hear the speech.

The participant terminal device 300 transmits participant data to the conference system 100 . The participant data in this embodiment includes various information transmitted from the participant terminal device 300 to the conference system 100 . For example, the participant data includes pointer data representing operation results of a pointing device 392, which will be described later, key input data input using a keyboard 393, audio data recorded by a microphone 394, and captured image data captured by a camera 395. include.

The participant data may also include screen data displayed on the display 391, which will be described later with reference to FIG. 5, file data uploaded by the user, and the like. Participant data may also include identification information about the participant terminal device 300 . The identification information here includes terminal identification information that uniquely identifies the participant terminal device 300 and user identification information that identifies the user who uses the participant terminal device 300 . Note that the user identification information may be a user ID that uniquely identifies a user, or a user name that is allowed to overlap with other users.

The conference system 100 distributes conference data to a plurality of participant terminal devices 300. FIG. The conference data here includes various information transmitted from the conference system 100 to the participant terminal device 300 . For example, conference data is data created based on participant data transmitted from a plurality of participant terminal devices 300, and includes conference audio data obtained by multiplexing a plurality of audio data, captured image data, and screen data. It includes conference screen data and the like created based on this.

Conference system 100 is, for example, a server system. The conference system 100 may be a single server or a set of multiple servers. For example, the conference system 100 may include a database server that stores identification information and the like regarding a plurality of participant terminal devices 300, and an application server that performs conference data creation processing and distribution processing. The server here may be a physical server or a virtual server. For example, the database server and the application server may be separate physical servers, and the conference system 100 may consist of two physical servers. Alternatively, the database server and application server may each be a virtual server. In this case, one virtual server may be constructed on one physical server, or may be distributed among a plurality of physical servers. Also, multiple virtual servers may be built on the same physical server. As described above, the functional configuration and physical configuration of the conference system 100 can be modified in various ways.

The conference system 100 also has a function of providing a conference service using voice data. As a specific example of a conference service using audio data, a video conference service that can use both audio data and image data will be described below. The conference system 100 may also have a function of providing a chat service. In this case, one server may have two functions, or each function may be implemented by a different server. For example, conferencing system 100 may include an application server for providing video conferencing services, an application server for providing chat services, and a database server shared by the two services. However, the specific configuration is not limited to this, and various modifications are possible. Note that the conference data in this embodiment may include both data used for the video conference service and data used for the chat service.

The monitoring system 200 is a system that monitors audio data in the conference provided by the conference system 100. FIG. The monitoring system 200 is, for example, a server system, and may be a single server or a set of multiple servers. As with the conference system 100 described above, the functional configuration and physical configuration of the monitoring system 200 can be modified in various ways. The monitoring system 200 monitors audio data failures in the conference system 100, and outputs transcript data when a failure is detected. Specific processing will be described later.

The participant terminal device 300 is a device used by a user who is a participant in the conference, and is, for example, a PC (Personal Computer). However, the participant terminal device 300 may be a mobile terminal device such as a tablet terminal or a smart phone.

1.2 Conference System and Monitoring System FIG. 2A is a hardware configuration diagram of the conference system 100. As shown in FIG. Conferencing system 100 includes processor 140 , memory 150 and communication interface 160 . However, the configuration of the conference system 100 is not limited to that shown in FIG. 2, some components may be omitted, other components may be added, and various modifications are possible. 2B and FIGS. 3 to 6, which will be described later, are the same in that the specific configuration is not limited to the contents of the drawings.

The processor 140 can use various processors such as a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), and a DSP (Digital Signal Processor). Processor 140 may also include peripheral circuit devices in addition to CPU, GPU, and DSP. The peripheral circuit device may be an IC (Integrated Circuit), and may include resistors, capacitors, and the like.

The memory 150 may be a semiconductor memory such as an SRAM (Static Random Access Memory), a DRAM (Dynamic Random Access Memory), a ROM (Read Only Memory), a flash memory, a register, or an HDD ( It may be a magnetic storage device such as a Hard Disk Drive, or an optical storage device such as an optical disk device.

The communication interface 160 is an interface for communicating via a network, and includes, for example, an antenna, an RF (radio frequency) circuit, and a baseband circuit. Communication interface 160 may operate under the control of processor 140 or may include a processor for communication control different from processor 140 . The communication interface 160 is an interface for performing communication according to TCP/IP (Transmission Control Protocol/Internet Protocol), for example. However, the specific communication method can be modified in various ways.

FIG. 2B is a hardware configuration diagram of the conference system 100. As shown in FIG. Conferencing system 100 includes processor 240 , memory 250 and communication interface 260 .

Various processors such as a CPU, GPU, and DSP can be used as the processor 240 . The memory 250 may be a semiconductor memory such as SRAM, DRAM, ROM, flash memory, etc., a register, a magnetic storage device, or an optical storage device. . A communication interface 260 is an interface for communicating via a network, and includes, for example, an antenna, an RF circuit, and a baseband circuit.

FIG. 3 is a functional block diagram of the conference system 100. As shown in FIG. The conference system 100 includes a processing section 110 , a storage section 120 and a communication section 130 . The processing unit 110 includes a participant data acquisition unit 111, a first conference data creation unit 112, a first conference data distribution unit 113, a second conference data creation unit 114, a second conference data distribution unit 115, and a control unit 116. .

Processing unit 110 corresponds to processor 140 in FIG. 2A. Storage unit 120 corresponds to memory 150 in FIG. 2A. The communication unit 130 corresponds to the communication interface 160 of FIG. 2A.

The storage unit 120 stores computer-readable instructions, and the functions of the processing unit 110 are realized as processes by the processing unit 110 executing the instructions. Specifically, the processor 140 operates in accordance with the instructions stored in the memory 150, so that the participant data acquisition unit 111, the first conference data creation unit 112, the first conference data distribution unit 113, and the Processing in each of the second conference data creation unit 114, the second conference data delivery unit 115, and the control unit 116 is executed. The instruction here may be an instruction set that constitutes a program, or an instruction that instructs the hardware circuit of the processor 140 to operate.

The participant data acquisition unit 111 performs processing for acquiring participant data from a plurality of participant terminal devices 300 via the communication unit 130 . Participant data acquisition section 111 outputs the acquired participant data to first conference data creation section 112 and second conference data creation section 114 .

When the participant terminal device 300 is connected for the first time, terminal identification information for uniquely identifying the participant terminal device 300 is given. By using the terminal identification information, the participant data acquisition unit 111 can associate the participant data with the participant terminal device 300 that is the transmission source of the participant data. For example, when voice data is acquired from a plurality of participant terminal devices 300, the participant data acquisition unit 111 can identify from which participant terminal device 300 each voice data is transmitted. . Also, the participant data acquisition unit 111 may accept input of a user name, for example, when the participant terminal device 300 is connected for the first time. The storage unit 120 stores terminal identification information and user identification information such as a user name in association with each other.

The first conference data creation unit 112 creates conference data in the video conference service based on the participant data acquired by the participant data acquisition unit 111 . Participant data here includes voice data acquired by the microphone 394, for example. The participant data may also include captured image data captured by the camera 395, screen data displayed on the display unit 340, and the like. Also, the conference data here includes, for example, conference audio data obtained by multiplexing audio data from a plurality of participant terminal devices 300 . The conference data may also include conference screen data generated by arranging image data and screen data from a plurality of participant terminal devices 300 according to a given rule. First conference data creation unit 112 outputs the created conference data to first conference data distribution unit 113 .

The conference screen data is a file created using a markup language or the like, and specific screen generation may be executed in the participant terminal device 300 . In other words, the conference screen data in this embodiment is data for displaying the screen used in the conference on the display unit 340 of the participant terminal device 300, and is not limited to the data of the screen itself. This point also applies to the conference screen data generated by the second conference data generation unit 114 .

The first conference data distribution unit 113 performs processing for transmitting conference data for video conference service to a plurality of participant terminal devices 300 via the communication unit 130 . In the conference system 100, the first conference data creation unit 112 and the first conference data delivery unit 113 correspond to a video conference system that provides video conference services. Hereinafter, the video conference system will also be referred to as the first system.

The second conference data creation unit 114 creates conference data in the chat service based on the participant data acquired by the participant data acquisition unit 111 . The participant data here includes key input data input using the keyboard 393, for example. For example, in the chat screen of FIG. 10, which will be described later, when text is entered in the text posting area Re5 and the post button is pressed, the communication unit 330 of the participant terminal device 300 sends the text to the post. It is transmitted to the conference system 100 as party data. The chat screen here is a screen that displays text data posted from the participant terminal device 300 . The conference data in the second conference data creation unit 114 includes conference screen data and the like for displaying the chat screen. Second conference data creation unit 114 outputs the created conference data to second conference data delivery unit 115 .

The second conference data distribution unit 115 performs processing for transmitting conference data for chat service to a plurality of participant terminal devices 300 via the communication unit 130 . In the conference system 100, the second conference data creation unit 114 and the second conference data distribution unit 115 correspond to a chat system that provides chat services. Hereinafter, the chat system is also referred to as the second system.

The control unit 116 controls each unit included in the conference system 100 . For example, the control unit 116 performs read/write control of the storage unit 120 and communication control of the communication unit 130 . Also, the control unit 116 may control each unit included in the processing unit 110 .

The storage unit 120 stores various types of information such as the above-described participant data, conference data, and identification information. The storage unit 120 may also store a program that operates on the participant terminal device 300 . The program here is, for example, a web application program which will be described later. The communication unit 130 performs various types of communication such as receiving participant data and distributing conference data.

FIG. 4 is a functional block diagram of the monitoring system 200. As shown in FIG. The monitoring system 200 includes a processing section 210 , a storage section 220 and a communication section 230 . The processing unit 210 includes a voice data acquisition unit 211 , a voice recognition result acquisition unit 212 , a defect detection unit 213 , a written data output unit 214 and a control unit 215 .

Processing unit 210 corresponds to processor 240 in FIG. 2B. Storage unit 220 corresponds to memory 250 in FIG. 2B. Communication unit 230 corresponds to communication interface 260 in FIG. 2B.

The memory 250 stores computer-readable instructions, and when the processor 240 executes the instructions, the functions of the units included in the processing unit 210 are realized as processes. Specifically, the processor 240 operates according to the instructions stored in the memory 250, so that the speech data acquisition unit 211, the speech recognition result acquisition unit 212, the defect detection unit 213, the written data output unit 214, and the control unit 215 is executed.

The audio data acquisition unit 211 acquires from the conference system 100 audio data transmitted from the plurality of participant terminal devices 300 to the conference system 100 . For example, monitoring system 200 may participate in a conference as a guest user, as described below with reference to FIG. In other words, the monitoring system 200 may be connected to the conference system 100 as the participant terminals 300 . In this case, the audio data acquisition unit 211 acquires conference data including audio data from the conference system 100 . The voice data acquisition unit 211 outputs the acquired voice data to the voice recognition result acquisition unit 212 and the defect detection unit 213 .

The speech recognition result acquisition unit 212 acquires a speech recognition result that is the result of performing speech recognition processing on speech data. The speech recognition result acquisition unit 212 may output the speech data to an external speech recognition server via the communication unit 230 and acquire the speech recognition result from the speech recognition server. In speech recognition processing, acoustic analysis is first performed to extract features from speech data. The results of the acoustic analysis are processed to identify phonemes with similar features using an acoustic model. Furthermore, speech recognition results are obtained by converting phonemes into words and sentences using a pronunciation dictionary and language model. A speech recognition result is data representing a conversion result obtained by converting speech data into text. In addition, in the speech recognition processing of this embodiment, since a well-known method can be widely applied, further detailed description will be omitted. Also, the voice recognition process is not limited to being performed outside the monitoring system 200 . For example, the speech recognition result acquisition unit 212 may perform speech recognition processing to convert speech data into text.

The defect detection unit 213 performs defect detection processing for detecting defects in the audio data. The problem here includes at least one of volume reduction and sound quality deterioration. Details of the defect detection process will be described later.

The transcription data output unit 214 outputs transcription data including text, which is the speech recognition result of the audio data, to the conference system 100 when a problem is detected in the audio data. For example, when the monitoring system 200 functions as the participant terminal device 300 , the draft data output unit 214 creates participant data including the draft data and transmits the participant data to the conference system 100 . Note that the transcription data here may be the text data itself, which is the speech recognition result, or may be information in which some kind of metadata is added to the text data. Metadata includes, for example, identification information about the participant terminal device 300 that is the transmission source of the audio data.

The control unit 215 controls each unit included in the monitoring system 200 . For example, the control unit 215 performs read/write control of the storage unit 220, communication control of the communication unit 230, control of each unit included in the processing unit 210, and the like.

The storage unit 220 stores various types of information such as voice data and written data. The communication unit 230 transmits and receives data to and from the conference system 100 . When the monitoring system 200 functions as the participant terminal device 300, the communication unit 230 transmits participant data, receives conference data, and the like.

Note that the technique of the present embodiment may be applied to an information processing method including steps executed by each unit of the monitoring system 200 . The information processing method is, for example, an information processing method for monitoring audio data in a conference system 100 that provides a conference by a plurality of participant terminal devices 300 .

1.3 Participant Terminal Device FIG. 5 is a hardware configuration diagram of the participant terminal device 300. As shown in FIG. Participant terminal device 300 includes processor 360, memory 370, communication interface 380, display 391, pointing device 392, keyboard 393, microphone 394, camera 395, and the like.

Various processors such as a CPU, GPU, and DSP can be used as the processor 360 . The memory 370 may be a semiconductor memory such as SRAM, DRAM, ROM, flash memory, etc., a register, a magnetic storage device, or an optical storage device. . A communication interface 380 is an interface for communicating via a network, and includes, for example, an antenna, an RF circuit, and a baseband circuit.

The display 391 is for displaying various display screens, and can be realized by, for example, a liquid crystal display or an organic EL display.

A pointing device 392 is an operation interface for moving a pointer displayed on the display 391 . The pointing device 392 can be implemented by various devices such as a mouse, pen tablet, touch pad, trackball, and the like.

The keyboard 393 is an operation interface that has a plurality of keys and outputs corresponding signals when the keys are operated. Various modifications can be made to the specific shape of the keyboard, key arrangement, connection method, and the like.

A microphone 394 is an interface that receives voice and outputs voice information. Note that the participant terminal device 300 may include a speaker (not shown) that outputs various sounds based on signals from the processor.

The camera 395 is arranged at a given position of the participant terminal device 300, and outputs a captured image of, for example, the vicinity of the user's face. The camera 395 includes, for example, a lens unit into which light from a subject is incident, and an imaging device that forms a subject image via the lens unit and outputs a captured image signal.

FIG. 6 is a functional block diagram of the participant terminal device 300. As shown in FIG. The participant terminal device 300 includes a processing section 310 , a storage section 320 , a communication section 330 , a display section 340 and a user input reception section 350 . The processing unit 310 includes a participant data transmission unit 311 , a conference data presentation unit 312 and a control unit 313 .

Processing unit 310 corresponds to processor 360 in FIG. Storage unit 320 corresponds to memory 370 in FIG. Communication unit 330 corresponds to communication interface 380 in FIG. Display unit 340 corresponds to display 391 in FIG. User input reception unit 350 corresponds to at least one of pointing device 392, keyboard 393, microphone 394, and camera 395 in FIG.

The memory 370 stores computer-readable instructions, and when the processor 360 executes the instructions, the functions of the units included in the processing unit 310 are realized as processes. For example, the storage unit 120 of the conference system stores a web application program. The participant terminal device 300 receives the web application program via the communication unit 330 and stores the received web application program in the storage unit 320 . The processing unit 310 implements the functions of each unit of the processing unit 310 by operating according to the web application program stored in the storage unit 320 .

The participant data transmission unit 311 performs processing of creating participant data and transmitting the participant data to the conference system 100 via the communication unit 330 . Examples of the participant data are as described above, and may be data received by the user input receiving unit 350, data displayed on the display unit 340, or an upload file selected by the user. may be

The conference data presenting unit 312 performs processing for presenting the conference data received by the communication unit 330 from the conference system 100 . For example, the conference data presentation unit 312 acquires the conference data transmitted by the first conference data delivery unit 113 . The conference data presentation unit 312 performs processing for displaying conference screen data of the conference data on the display unit 340 and processing for outputting conference audio data to a speaker. This provides video conferencing services.

Also, the conference data presentation unit 312 acquires the conference data transmitted by the second conference data delivery unit 115 . The conference data is chat screen data that is conference screen data for displaying a chat screen. The conference data presentation unit 312 performs processing for displaying chat screen data on the display unit 340 . This provides a chat service. Note that the conference data presentation unit 312 may present the received conference data as it is, or may present the result of performing some processing.

The control unit 313 controls each unit included in the participant terminal device 300 . For example, the control unit 313 performs read/write control of the storage unit 320 , communication control of the communication unit 330 , display control of the display unit 340 , and control of each unit of the user input reception unit 350 . Also, the control unit 313 may control each unit included in the processing unit 310 .

The storage unit 320 stores various types of information such as the above-described participant data and conference data. The communication unit 330 performs various types of communication such as sending participant data and receiving conference data. The display unit 340 displays conference screen data. The user input reception unit 350 receives input from the user. User input is applicable to input using various devices as described above, and may include input using voice, gestures, and the like. Sound is detected based on the output of microphone 394 . Gestures are detected based on the output of camera 395 .

2. Flow of Processing FIG. 7 is a sequence diagram illustrating the flow of processing in this embodiment. First, the user performs an operation to create a conference using the participant terminal device 300 . The user here is, for example, the host of the conference, and the participant terminal device 300 is a device used by the host. For example, the organizer performs an operation to create setting data used for the conference, such as the start time of the conference, the participants, and the authority granted to each participant. Based on the operation, in step S101, the participant terminal device 300 transmits data for creating a conference including the setting data, for example, to the conference system 100. FIG.

In step S102, the conference system 100 performs processing for providing a conference to the participant terminal devices 300. FIG. Specifically, the conference system 100 sends data for participating in the conference, such as a conference ID, a conference URL (uniform resource locator), a password, etc., to the participant terminal device 300 designated as a conference participant. to send. Each participant terminal device 300 transmits a conference participation request to the conference system 100 using the data. When the conference system 100 approves the participation request, the conference system 100 provides the conference by receiving the participation data from the participant terminal device 300 and by starting to distribute the conference data to the participant terminal device 300 . Although detailed description is omitted in FIG. 7, as can be seen from the above description, the process of step S101 is performed by a specific participant terminal device 300 such as the organizer, and the process of step S102 is performed by the participant. It is executed with a plurality of participant terminal devices 300 as targets.

In step S103, the monitoring system 200 performs processing for participating as a guest in the conference to be monitored. For example, the conference system 100 may perform processing to invite the monitoring system 200 as a conference participant. Specifically, the conference system 100 transmits data for participating in the conference including the conference ID and the like to the monitoring system 200 . The monitoring system 200 executes the process of step S103 by requesting participation in the designated conference using the data. Alternatively, the user who is the organizer may decide whether or not to monitor the audio data. For example, when the organizer performs an operation to monitor voice data, the conference system 100 executes processing for inviting the monitoring system 200 to the conference. Alternatively, the conference system 100 may determine whether the organizer is a user authorized to use the monitoring system 200 . If it is determined that the host has the authority, the conference system 100 performs processing to invite the monitoring system 200 as a conference participant.

In step S104, the conference system 100 performs approval processing for the participation request in step S103. As a result, the monitoring system 200 can transmit participant data to the conference system 100 and receive conference data from the conference system 100 in the same manner as the plurality of participant terminal devices 300 .

By the processing of steps S101 to S104, a conference is provided by the conference system 100, and a plurality of participant terminal devices 300 and the monitoring system 200 enter a state of participating in the conference.

During the conference, the plurality of participant terminal devices 300 each transmit participant data to the conference system 100 at given timings. The conference system 100 distributes conference data generated based on participant data to a plurality of participant terminal devices 300 .

As described above, the conference system 100 acquires audio data from a plurality of participant terminal devices 300, and includes a first system that provides a conference using at least audio, and text data from the plurality of participant terminal devices 300. and a second system that obtains as party data and provides a conference by chat. The first system is a system that corresponds to the first conference data creation unit 112 and the first conference data distribution unit 113 and provides video conference services. The second system is a system that corresponds to the second conference data creation unit 114 and the second conference data distribution unit 115 and provides a chat service.

FIG. 9 is an example of a display screen used in a video conference system, and FIG. 10 is an example of a display screen used in a chat system. 9 and 10 show screens displayed on the display unit 340 of the participant terminal device 300 based on the conference screen data from the conference system 100. FIG.

As shown in FIG. 9, the display screen in the video conference service includes a participant display area Re1, an image display area Re2, an operation button display area Re3, and the like. The participant display area Re1 is an area for displaying user identification information representing conference participants. In the example of FIG. 9, the participants of the conference are participants A and B who are users using the participant terminal device 300 and pseudo users corresponding to the monitoring system 200 . "Guest Recorder" in FIG. 9 represents a pseudo user. The image display area Re2 is an area in which specific contents of the uploaded file, an image around each participant's face captured by the camera 395, and the like are displayed. The operation button display area Re3 is an area where objects for performing on/off operations of the microphone 394 and the camera 395, volume operations, and the like are displayed.

As shown in FIG. 10, the display screen in the chat service includes a text display area Re4 and a text posting area Re5. The text display area Re4 is an area for displaying the text data posted by the participants in chronological order in association with the posted participants. The text posting area Re5 includes an area for inputting text to be posted and an area for displaying a post button.

As shown in FIGS. 9 and 10, by providing a plurality of conferences with different methods, the conference can proceed more smoothly.

Returning to FIG. 7, the description continues. After the start of the conference in which the plurality of participant terminal devices 300 and the monitoring system 200 participate, in step S105, a given participant terminal device 300 among the plurality of participant terminal devices 300 receives participant data including voice data. is transmitted to the conference system 100 . The process of step S105 is repeatedly executed, for example, during a meeting.

In step S<b>106 , the conference system 100 transmits the acquired audio data to the monitoring system 200 . In step S<b>106 , the conference system 100 may transmit conference data common to the plurality of participant terminals 300 to the monitoring system 200 . For example, the monitoring system 200 creates conference audio data and transmits the conference audio data to the plurality of participant terminal devices 300 and the monitoring system 200 . Alternatively, the conference system 100 may separately create and transmit conference data for the monitoring system 200 . For example, the conference system 100 extracts voice data from the collected participant data, associates the extracted voice data with identification information about the participant terminal device 300, which is the source of the voice data, and monitors the monitoring system. 200. For example, the monitoring system 200 receives information in which identification information relating to the first participant terminal device is associated with audio data from the first participant terminal device among the plurality of participant terminal devices 300 .

Thus, in the step of acquiring audio data from the conference system 100 , the monitoring system 200 may communicate with the conference system 100 as the participant terminal device 300 . The monitoring system 200 receives audio data from the conference system 100 according to the communication method used when the conference system 100 distributes data to the plurality of participant terminal devices 300 .

In this way, the conference system 100 can handle the monitoring system 200 as one of the participant terminal devices 300. FIG. For example, the conference system 100 may have the same configuration as a conventional system that collects participant data and distributes conference data, and the audio monitoring function according to the technique of the present embodiment is different from the conference system 100. It becomes possible to realize it as a different system. In this case, there is an advantage that the system change of the conference system 100 itself is unnecessary when constructing the system according to the present embodiment. Further, in the monitoring system 200, the same configuration as the participant terminal device 300 can be used for the interface for transmitting and receiving data with the conference system 100. FIG. For example, some functions of the monitoring system 200 may be realized by the processor 240 operating according to a web application program like the participant terminal device 300. In this way, input/output to/from the conference system 100 can be realized using a prescribed protocol such as HTTPS (Hypertext Transfer Protocol Secure), so implementation of the monitoring system 200 is facilitated.

The voice recognition result acquisition unit 212 of the monitoring system 200 acquires the voice recognition result of the voice data acquired in step S106. FIG. 7 illustrates an example in which speech recognition processing is performed by an external speech recognition server. In step S107, the communication unit 230 of the monitoring system 200 transmits the voice data to the voice recognition server.

In step S108, the speech recognition server performs speech recognition processing. As described above, the speech recognition processing in step S108 can widely apply known methods. In step S<b>109 , the speech recognition server transmits the speech recognition result to the monitoring system 200 .

In step S110, the monitoring system 200 may acquire the speech recognition result of the speech data and store the speech recognition result as minutes data. In this way, it becomes possible to automatically create minutes data. Since the speech recognition result is used not only as a countermeasure when a problem occurs, but also as meeting minutes data, as will be described in S112 and later, the data can be used effectively. Storing speech recognition results as meeting minutes data facilitates management and utilization of meeting-related data. Further, as can be seen by comparing with the processing after S112, the processing of step S110 may be executed regardless of the sound defect detection result. In this way, many scenes in the conference are subject to creation of minutes data, so that it is possible to create minutes data that is useful for understanding the overall content of the conference.

Also, in step S111, the fault detection unit 213 of the monitoring system 200 performs fault detection processing for detecting a fault in the voice data acquired in step S106. The problem of sound here means that the volume (sound pressure) is reduced by a predetermined value or more compared to a given reference sound volume, and that the sound quality is less than a given reference sound quality to the extent that a predetermined condition is satisfied. It represents at least one of Note that the process of step S111 can be executed at any timing after the voice data is obtained in step S106, and the relationship between steps S107 to S110 is not limited to that illustrated in FIG.

FIG. 8 is a flowchart for explaining the defect detection processing in step S111. When this process is started, first, in step S201, the defect detection unit 213 detects the sound volume of the audio data during the detection period that is the target of the defect detection process. The volume here is the amplitude value of audio data, and is numerical data in units of dB, for example. The defect detection unit 213 may obtain the average volume of the audio data during the detection period, or may obtain the maximum volume or the minimum volume. Note that the detection period here can be set arbitrarily, and may be several seconds to several tens of seconds or longer. Alternatively, a period during which the same person continues to speak may be detected, and the detection period may be set dynamically using the detected period.

In step S202, the defect detection unit 213 determines whether the detected volume value is equal to or less than a given volume threshold. If it is determined that the volume is equal to or less than the volume threshold, the fault detection unit 213 determines that there is a fault in step S203. The volume threshold here may be a given fixed value, or may be dynamically set using the average volume of voice data acquired during the meeting.

If it is determined in step S202 that the volume value is greater than the volume threshold, the defect detection unit 213 determines the sound quality. For example, in step S204, the defect detection unit 213 performs sound source separation processing for separating the voice data into voice and noise. For example, in a three-dimensional spectrogram of time, frequency, and signal component intensity, there is known a method of separating sound sources by using nonlinear filtering, focusing on the fact that signals from a plurality of sound sources overlap little. In recent years, there is also a machine learning method that generates a trained model for performing sound source separation processing using a data set that associates speech data with noise superimposed and data from which the speech part is extracted. Widely known. The defect detection unit 213 may perform the sound source separation processing in step S204 by acquiring a trained model and inputting voice data to the trained model.

In step S205, the defect detection unit 213 calculates a sound quality index value based on the separated audio signal and noise signal. The index value here is, for example, the S/N ratio, which is the ratio of the signal level of voice to the signal level of noise.

In step S206, the defect detection unit 213 determines whether the calculated S/N ratio is equal to or less than a given SN threshold. When the S/N ratio is equal to or less than the SN threshold, the process proceeds to step S203, and the defect detection unit 213 determines that there is a defect. The SN threshold here may be a given fixed value, or may be dynamically set using an average value of S/N ratios obtained during the conference. If the S/N ratio is greater than the SN threshold in step S206, the defect detection unit 213 determines that there is no defect in step S207.

Note that FIG. 8 is an example of processing, and the defect detection processing of the present embodiment is not limited to this. For example, if noise related to the microphone 394 of the participant terminal device 300, noise related to the communication between the participant terminal device 300 and the conference system 100, etc. are assumed as the cause of the problem in the voice data, typical examples of problems caused by these factors are shown below. data can be estimated in advance. That is, the defect detection unit 213 may hold normal data and one or more abnormal data in advance. The defect detection unit 213 may determine the presence or absence of a defect according to whether the acquired voice data is similar to normal data or abnormal data. In addition, although FIG. 8 illustrates an example in which it is determined that there is no problem when both the volume and sound quality satisfy the conditions, the defect detection process may be performed based on only the sound volume or only the sound quality. good. In addition, the fault detection of this embodiment can be extended to other processes related to volume and sound quality.

Returning to FIG. 7, the description continues. When a problem in the audio data is detected by the problem detection process, the monitoring system 200 executes a process to prevent the progress of the conference from being stopped due to the problem. Specifically, first, in step S112, the written data output unit 214 creates participant data including written data that is the result of speech recognition of the voice data. The transcription data here is, for example, the voice recognition result of the voice data determined to be defective. However, the transcript data may include speech recognition results of at least one of audio data acquired before the audio data in which the defect was detected and audio data acquired after the audio data in which the defect was detected. . Moreover, it is not essential that all of the voice data in which defects are detected be included in the draft data, and part of it may be omitted. In step S<b>113 , the draft data output unit 214 transmits the draft data to the conference system 100 .

As described above, the monitoring system 200 may be connected to the conference system 100 as a participant terminal device 300. FIG. As shown in steps S112 and S113, the monitoring system 200 may transmit the draft data to the conference system 100 according to the communication method used when the plurality of participant terminal devices 300 transmit data to the conference system 100. good.

In this way, it becomes possible to treat the monitoring system 200 as one of the participant terminal devices 300 . Therefore, as described above, the technique of this embodiment can be easily implemented without changing the conference system 100 itself. Moreover, since the input/output interface between the monitoring system 200 and the conference system 100 can be implemented by the same configuration as the participant terminal device 300, the monitoring system 200 can be easily implemented.

In step S114, the conference system 100 creates conference data including draft data. Note that in the step of outputting the draft data shown in step S113, the monitoring system 200 may output the draft data to the second system. That is, the conference data created in step S114 is chat screen data that is used in a chat system, and may be created by the second conference data creating section 114 . By using the chat system in this way, it is possible to present the contents of the voice data relating to the trouble to the user in a manner that makes it easy to visually recognize the chronological order and the contents.

In step S<b>115 , the second conference data distribution unit 115 of the conference system 100 distributes the chat screen data to the plurality of participant terminal devices 300 . In step S116, the plurality of participant terminal devices 300 display a chat screen on which the written data is displayed on the display unit 340 based on the received chat screen data.

As described above, in the step of acquiring the voice data from the conference system 100 shown in step S106, the monitoring system 200 associates the identification information of the participant terminal device 300, which is the source of the voice data, with the voice data. may be obtained. Then, in the step of outputting the draft data to the conference system 100 shown in step S113, the monitoring system 200 associates the identification information indicating the transmission source of the audio data in which the problem is detected with the draft data, and outputs the data to the conference system 100. may

For example, when there are participant A and participant B as shown in FIG. In this case, the written data output unit 214 outputs the written data in association with the participant B. FIG. In this way, it is possible to detect which participant terminal device 300 has a problem in the audio data, and to present the detection result to the user in an easy-to-understand manner.

At this time, in the step of outputting the chat screen data to the plurality of participant terminal devices 300 shown in step S115, the second system of the conference system 100 identifies the user represented by the identification information associated with the voice data as the poster. , and may output chat screen data that displays the written data as the posted content. For example, the monitoring system 200 may transmit to the conference system 100 disguised data as if it were the participant terminal device 300 of the participant B itself. Alternatively, the second conference data creating unit 114 of the conference system 100 may correct the data posted from the monitoring system 200 as if the participant terminal device 300 of the participant B posted the data.

In the example of FIG. 10, although the actual contributor is the monitoring system 200, participant B is displayed in the area representing the contributor, and text corresponding to the written data is displayed in the area representing the posted content. In this way, it becomes possible for the user viewing the chat screen to easily understand which participant's voice has a problem. At this time, clearly indicating that the posting was made by the monitoring system 200 makes it easier to distinguish the data from the data actually posted by the participant B. FIG. For example, in FIG. 10, a message such as "BOT" or the like is displayed as follows: "Due to the interruption of participant B's voice, we will start writing down."

As described above with reference to FIGS. 7 to 10, according to the method of the present embodiment, when a problem is detected in voice data, related voice recognition results are output to the conference system 100 as draft data. Therefore, the conference system 100 can appropriately detect audio defects. For example, the conferencing system 100 distributes the draft data to the participant terminal device 300, so that the participant can be notified of the audio problem. Since the utterer of the voice data knows the problem of the voice data by the delivery of the written data, it is possible to suppress the continuation of the voice data and to identify the cause of the problem and solve it. In addition, participants other than the speaker can understand the content of the speech at the time of occurrence of the problem to some extent by referring to the written data, thereby preventing the progress of the conference from being stopped. If there is a problem with the voice data, it may be difficult for a human user to recognize the contents of the voice data due to factors such as voice interruption, low volume, and high noise. However, in speech recognition processing, signal processing such as noise reduction processing and interpolation processing is possible, so it is considered possible to obtain speech recognition results that enable the general understanding of the content of speech.

Further, as shown in FIG. 7, if no problem is detected in the audio data by the problem detection process of step S111, the monitoring system 200 skips the processes after step S112. That is, the monitoring system 200 does not output to the conference system 100 the transcription data including the text that is the speech recognition result of the audio data when a problem is detected in the audio data. In this way, it is possible to suppress the output of information with low necessity when there is no problem. From a different point of view, if the written data is output, it is possible to clarify that the cause is a defect in the audio data.

3. Modified Example 3.1 Other Configurations of Communication System An example in which the conference system 100 and the monitoring system 200 are separately provided has been described above. By separating the conference system 100 and the monitoring system 200 and connecting the two systems via a network, for example, if an error occurs in one of them, the error will be propagated to the other. can be suppressed. However, the configuration of the communication system 10 of this embodiment is not limited to this.

FIG. 11 is a diagram showing another configuration of the communication system 10. As shown in FIG. As shown in FIG. 11, the communication system 10 may include an integrated server 400 including the functions of the conference system 100 and the functions of the monitoring system 200, and a plurality of participant terminals 300. FIG. For example, the hardware configuration of server 400 is similar to that of FIG. 2A and includes a processor, memory, and communication interface. Participant data acquiring unit 111, first conference data creating unit 112, first conference data distributing unit 113, second conference data creating unit 114, second conference data distributing unit 114, and second conference data distributing unit 114 Functions of the unit 115, the control unit 116, the voice data acquisition unit 211, the voice recognition result acquisition unit 212, the defect detection unit 213, the written data output unit 214, and the control unit 215 are realized. Either one of the control unit 116 and the control unit 215 may be omitted.

In the configuration of FIG. 11, the monitoring system 200 in the server 400 may function as the virtual participant terminal device 300 and connect to the conference system 100 in the same device. Alternatively, the monitoring system 200 may acquire voice data and output written data without participating in the conference as the participant terminal device 300 . For example, audio data acquired by the conference system 100 may be stored in a given area of the memory, and the monitoring system 200 may acquire the audio data by reading the area of the memory. In this case, there is no need for communication via the network, so there is an advantage that delays and errors in communication between the conference system 100 and the monitoring system 200 need not be considered. The same applies to transmission of written data from the monitoring system 200 to the conference system 100. The monitoring system 200 may function as a virtual participant terminal device 300, or input/output may be performed via memory. good.

3.2 Identification Information Also, an example in which identification information is associated when the monitoring system 200 outputs the draft data has been described above. For example, as shown in FIG. 10, the identification information regarding the sender of the voice data such as "participant B" is associated with the written data. However, identification information is not essential in the method of this embodiment.

For example, the monitoring system 200 may output the draft data without associating it with the identification information. The conference system 100 distributes the draft data to a plurality of participant terminal devices 300 without specifying the speaker. Even in this case, the participants of the conference can understand the problem of the voice by posting the written data.

Also, when a plurality of participants speak at the same time in a conference, it becomes difficult to hear them. Therefore, it is assumed that there are many cases in which a single participant speaks at a given timing, except for occasional situations. Therefore, even if identification information is not associated, each participant can check whether the problem is related to their own voice data by referring to the timing when the draft data was posted and the specific utterance content. It is considered possible to determine whether or not Therefore, it is possible for the utterer to recognize defects in his/her own voice data, and for other participants to roughly grasp the content of the utterance.

Although the present embodiment has been described in detail as above, those skilled in the art will easily understand that many modifications that do not substantially deviate from the novel matters and effects of the present embodiment are possible. . Accordingly, all such modifications are intended to be included within the scope of this disclosure. For example, a term described at least once in the specification or drawings together with a different broader or synonymous term can be replaced with the different term anywhere in the specification or drawings. All combinations of this embodiment and modifications are also included in the scope of the present disclosure. Also, the configurations and operations of the conference system, monitoring system, participant terminal devices, etc. are not limited to those described in the present embodiment, and various modifications are possible.

DESCRIPTION OF SYMBOLS 10... Communication system, 100... Conference system, 110... Processing part, 111... Participant data acquisition part, 112... First meeting data creation part, 113... First meeting data delivery part, 114... Second meeting data creation part, 115 second conference data distribution unit 116 control unit 120 storage unit 130 communication unit 140 processor 150 memory 160 communication interface 200 monitoring system 210 processing unit 211 audio Data acquisition unit 212 Speech recognition result acquisition unit 213 Defect detection unit 214 Draft data output unit 215 Control unit 220 Storage unit 230 Communication unit 240 Processor 250 Memory 260 Communication interface 300 Participant terminal device 310 Processing unit 311 Participant data transmission unit 312 Conference data presentation unit 313 Control unit 320 Storage unit 330 Communication unit 340 Display unit 350 User input reception unit 360 Processor 370 Memory 380 Communication interface 391 Display 392 Pointing device 393 Keyboard 394 Microphone 395 Camera 400 Server Re1 Participant display Areas Re2... Image display area Re3... Operation button display area Re4... Text display area Re5... Text posting area

Claims (8)

An information processing method for monitoring voice data in a conference system providing a conference by a plurality of participant terminal devices,
obtaining from the conference system audio data transmitted from the plurality of participant terminal devices to the conference system;
detecting defects in the audio data;
a step of outputting transcript data including text, which is a speech recognition result of the speech data, to the conference system when a defect in the speech data is detected;
Information processing methods, including In claim 1,
The conference system is
a first system that acquires the audio data from the plurality of participant terminal devices and provides a conference using at least audio;
a second system that acquires text data from the plurality of participant terminal devices and provides a chat conference,
The information processing method of outputting the draft data to the second system in the step of outputting the draft data. In claim 1 or 2,
In the step of acquiring the audio data from the conference system,
Acquiring, of the plurality of participant terminal devices, identification information relating to a first participant terminal device, which is a transmission source of the audio data, in association with the audio data;
In the step of outputting the draft data to the conference system,
An information processing method for associating the identification information indicating the transmission source of the voice data in which the defect is detected with the draft data and outputting the data to the conference system. In claim 2,
The second system of the conference system further comprising a step of outputting chat screen data to the plurality of participant terminals;
In the step of acquiring the audio data from the conference system,
Acquiring, of the plurality of participant terminal devices, identification information relating to a first participant terminal device, which is a transmission source of the audio data, in association with the audio data;
In the step of outputting the draft data to the conference system,
associating the identification information representing the transmission source of the audio data in which the defect is detected with the draft data and outputting the data to the conference system;
In the step of outputting the chat screen data,
An information processing method for outputting the chat screen data in which the user represented by the identification information associated with the voice data is displayed as a poster, and the written data is displayed as posted content. In any one of claims 1 to 4,
In the step of acquiring the audio data from the conference system,
communicating with the conferencing system as one of the plurality of participant terminals;
An information processing method for receiving the voice data from the conference system according to a communication method used when the conference system distributes data to the plurality of participant terminal devices. In claim 5,
In the step of outputting the draft data to the conference system,
An information processing method for transmitting the written data to the conference system according to a communication method used when the plurality of participant terminal devices transmit data to the conference system. In any one of claims 1 to 6,
An information processing method comprising the steps of acquiring the speech recognition result of the speech data and storing the speech recognition result as meeting minutes data regardless of the detection result of a defect in the speech data. A monitoring system for monitoring audio data in a conference system providing a conference by a plurality of participant terminal devices,
an audio data acquisition unit configured to acquire, from the conference system, audio data transmitted from the plurality of participant terminal devices to the conference system;
a defect detection unit that performs processing for detecting defects in the audio data;
a transcription data output unit that performs a process of outputting transcription data including text, which is a speech recognition result of the audio data, to the conference system when a defect in the audio data is detected;
Surveillance system including.

Images