TW202318252A - Caption service system for remote speech recognition - Google Patents

Abstract

The present invention provides a caption service system for remote speech recognition, which provides caption service for the hearing impaired. This system includes a speaker and a live broadcast equipment at A, a listener-typist and a computer at B, a hearing impaired and a live screen at C, and an automatic speech recognition (ASR) caption server at D. Connect the live broadcast equipment, the computer, the live screen and the ASR caption server with a network. The speaker's audio is sent to the automatic speech recognition (ASR) caption server to be converted into text, which is corrected by the listener-typist, and then the text caption is sent to the live screen of the hearing impaired together with the speaker's video and audio, so that the hearing impaired can see the text caption spoken by the speaker.

Description

Subtitle Service System for Remote Speech Recognition

The present invention relates to a subtitle service system for remote voice recognition, in particular to a subtitle service system that provides remote voice recognition for the hearing-impaired via a subtitle server and a typist.

Due to the outbreak of the COVID-19 epidemic, remote live broadcasting and teaching has become a widely adopted trend. However, the current general remote live broadcast and teaching do not have subtitles, which makes it impossible for hearing-impaired students to attend classes.

In general classrooms, hearing-impaired students also have problems in class, because there are no monitors that directly display subtitles of what the teacher is teaching. In various lectures and conferences, the hearing impaired cannot participate because there is no monitor to directly display subtitles.

Therefore, it is a great boon for the hearing-impaired to set subtitles that can display what the teacher or speaker said.

At present, some conferences use audiovisual staff to type what the speaker said on the spot with a computer and display them as subtitles on the computer screen, so that the hearing-impaired can understand the situation at the scene. However, the listener spends a lot of energy listening to the content of the speaker. Once the working hours are too long, there may be missing sentences and typos. Therefore, a more complete remote listener solution must be provided.

The purpose of the present invention is to propose a subtitle service system for remote speech recognition, for The subtitle service of remote voice recognition is provided for the hearing-impaired, and the content of the present invention is described as follows.

This system includes speakers and live broadcast equipment at A, listeners and computers at B, hearing-impaired people and live broadcast images at C, automatic speech recognition (ASR) subtitle server at D, and Internet Connect the live broadcast equipment, computer, live screen and ASR subtitle server.

Automatic Speech Recognition (ASR) subtitle server includes: Instant Messaging Protocol (RTMP) for receiving live streaming from A through the network; open source speech recognition toolkit for speech recognition and signal processing; network The server is responsible for providing the interface of the webpage, which is transmitted to the live broadcast device, computer and live broadcast screen through the HTTP protocol; the recording module is used for playback by the listener.

The voice of the speaker is sent to the Automatic Speech Recognition (ASR) subtitle server to convert it into text, which is corrected by the auditory typist. See text subtitles of what the speaker said.

1: speaker

2: Live broadcast equipment

3: listener

4: computer

5: hearing impaired

6: Live screen

61: subtitle area

7: Automatic Speech Recognition (ASR) Subtitle Server

8: Network

9: Instant Messaging Protocol (RTMP)

10: Open source speech recognition toolkit Kaldi ASR

11:Web server

12:Recording module

13:OBS Live Streaming Software

14: ASR upload interface

15: Audio streaming

16: Audio record

17: Video and sound captured by live equipment

18: Subtitle content captured from browser

FIG. 1 is a schematic diagram of the basic structure of a subtitle service system for remote speech recognition according to the present invention.

FIG. 2 is a schematic diagram of the content of the Automatic Speech Recognition (ASR) subtitle server of the present invention.

Fig. 3 is a schematic diagram of the content of the live broadcast device of the present invention.

FIG. 4 is a schematic diagram of the subtitle generation process of the Automatic Speech Recognition (ASR) subtitle server of the present invention.

Fig. 5 is a schematic diagram of the operation of the listener in different places according to the present invention.

Fig. 6 is a schematic diagram of combining and outputting the speaker's live video and subtitles according to the present invention.

FIG. 1 illustrates the basic structure of the subtitle service system for remote speech recognition of the present invention. The speaker 1 and the live broadcast device 2 are at site A, the listener 3 and his computer 4 are at site B, the hearing-impaired person 5 and the live broadcast screen 6 are at site C, the automatic speech recognition (ASR) subtitle server 7 is at site D, and four sites The network 8 is connected, and the network 8 may be a local area network or a global Internet. If site A, site B, and site C are in the same site, it means that speaker 1, auditory typist 3, and hearing-impaired person 5 are in the same classroom or meeting room.

FIG. 2 illustrates what is included in the Automatic Speech Recognition (ASR) subtitle server 7 of the present invention. Instant Messaging Protocol (Real-Time Messaging Protocol, RTMP) 9 is a protocol widely used in live streaming. The ASR subtitle machine server 7 equipped with this protocol can receive the live streaming from A through the network 8 . If you do not use RTMP, you can also use HTTP Live Streaming (abbreviated as HLS), which is an HTTP-based streaming media network transmission protocol proposed by Apple, and can also receive live streams from A through the network 8 flow. But the method of the present invention is not limited to RTMP or HLS.

The speech recognition of the ASR subtitle server 7 uses an open source speech recognition toolkit Kaldi ASR 10 for speech recognition and signal processing, which can be obtained free of charge under the Apache License v2.0.

The ASR subtitle server 7 needs to set up a network server 11, an interface responsible for providing web pages, which is transmitted to the client (generally referring to a web browser) through the HTTP protocol. The client refers to the live broadcast device 2, the computer 4, the live screen 6, etc. .

The ASR subtitle server 7 has a recording module 12 for the function of "playback" by the listener.

Please refer to FIG. 3 , illustrating the content of the live broadcast device 2 of the present invention. In A, speaker 1's live broadcast device 2 records the image and sound of speaker 1 and divides it into two channels, of which the first channel is input Open Broadcaster Software (OBS) live broadcast software 13, this software is a free and open source cross-platform streaming media and video recording program developed by the OBS Project, and is commonly used by live broadcasters. The backend of OBS Live Streaming Software 13 can be directly pushed to various platforms such as YouTube, Facebook or Twitch.

The second channel only provides the voice of the speaker 1, and the voice can be packaged through the simple ASR upload interface 14 and streamed to the ASR subtitle server 7 through RTMP 9 (or HLS).

Please refer to FIG. 4 , which illustrates the subtitle generation process of the Automatic Speech Recognition (ASR) subtitle server 7 of the present invention. When the streaming packet is sent to the RTMP 9 (or HLS) receiving module of the ASR subtitle server 7, the streaming packet will be unpacked into an audio source stream 15, and sent to the Kaldi ASR 10 and the recording module 12 respectively. The recording module 12 will record the sound source into the sound source record 16 according to time. After the Kaldi ASR 10 module receives the audio source stream 15, it will gradually convert it into text, and each text will contain a "tag", as shown in Figure 4. "Tag" will describe the position of this text corresponding to the second of the audio recording, and the recording duration. These words and tags will be displayed on the webpage of the web server 11, and transmitted to the live broadcast device 2, the computer 4, and the live broadcast screen 6 via the network 8.

See also Fig. 5, illustrate the operation of the listener 3 of the present invention in different places. The listener 3 in a different place turns on YouTube, Facebook or Twitch to receive the live video and sound from the speaker 1 in A. The listener 3 logs into the web server 11 page of the ASR subtitle server 7 through web browsing, and listens to the words and sounds of the lecturer 1 on the page.

Listener 3 is set to have readable and writable authority on ASR subtitle server 7, therefore can revise the literal that Kaldi ASR 10 produces on network server 11 pages. Each section of text has a tag attribute on the web server 11 page. For example, the listener 3 clicks twice on the C paragraph text, and the web server 11 page will request the N ₃ of the audio source record 16 according to the tag instruction. seconds, and the audio duration is Z seconds for playback. In this way, the listener 3 can confirm what the speaker 1 has said, and then modify the text.

Please refer to FIG. 6 , which illustrates that the speaker 1 of the present invention A combines the live image 6 and subtitles for output. The live broadcast device 2 of the speaker 1 can browse and log in to the web server 11 page of the ASR subtitle server 7 through the webpage, but only has the read permission. In other words, only the text translated by the ASR subtitle server 7 and the text modified by the listener 3 can be viewed on the live broadcast device 2 of the speaker 1 .

The OBS live broadcast software 13 has the function of superimposing pictures. The lecturer 1 selects the subtitle content from the web server 11 page of the ASR subtitle server 7 on the live broadcast device 2 and intercepts it, and then the video and sound 17 recorded by the live broadcast device 2 and from the The subtitle content 18 captured by the browser is superimposed, and the live screen 6 containing the subtitle generated by the ASR subtitle server 7 is output via the OBS live broadcast software 13 . Then push to platforms such as YouTube, Facebook or Twitch by OBS live broadcast software 13, make the hearing-impaired person 5 of C place see the subtitle content in the subtitle area 61 on the live broadcast picture 6.

The spirit and scope of the present invention are determined by the scope of claims below, and are not limited to the above-mentioned embodiments.

1: speaker

2: Live broadcast equipment

3: listener

4: computer

5: hearing impaired

6: Live screen

61: subtitle area

7: Automatic Speech Recognition (ASR) Subtitle Server

8: Network

Claims (10)

A subtitle service system for remote speech recognition, comprising: A speaker and a live broadcast device are at a site A, a listener and a computer are at a site B, a hearing-impaired person and a live broadcast screen are at a site C, and an automatic speech recognition (ASR) subtitle server is at a site D , the live broadcast equipment, the computer, the live screen and the ASR subtitle server are connected through a network; the voice of the speaker is sent to the automatic speech recognition (ASR) subtitle server and converted into text, which is corrected by the listener, Then send the text subtitles together with the speaker's image and sound to the live broadcast of the hearing-impaired person, so that the hearing-impaired person can see the text subtitles spoken by the speaker. For example, the subtitle service system for remote speech recognition in item 1 of the patent scope, wherein the Automatic Speech Recognition (ASR) subtitle server includes: A real-time message protocol (RTMP) is used to receive a live streaming from the place A via the network; An open source speech recognition toolkit for speech recognition and signal processing; A web server is responsible for providing the interface of the webpage, which is transmitted to the live broadcast device, the computer and the live broadcast screen through the HTTP protocol; A recording module for playback by the listener. For example, the subtitle service system for remote speech recognition in item 2 of the patent scope, in which the live broadcast device intercepts the video and audio of the speaker and divides it into two channels: The first way is to input the video and sound of the speaker into an OBS live broadcast software, and the back end of the OBS live broadcast software can directly push the video and sound of the speaker to various platforms such as YouTube, Facebook or Twitch, etc.; The second channel only provides the speaker's voice, which can be uploaded through an ASR interface Form a stream packet and send it to the ASR subtitle server through the RTMP. For example, the subtitle service system for remote speech recognition in the third item of patent scope, wherein the subtitle generation process of the ASR subtitle server is as follows: When the streaming packet is sent to the RTMP receiving module of the ASR subtitle server, the streaming packet will be unpacked into a sound stream, and sent to the open source speech recognition toolkit and the recording module respectively; The recording module records the audio source stream into an audio source record according to time as time progresses; After receiving the audio source stream, the open source speech recognition toolkit will gradually convert it into text, and each text will contain a tag, which describes the position of the text corresponding to the second recorded by the audio source, and the recording time is how much; the text and the label are displayed on the web server page, and transmitted to the live broadcast device, the computer and the live broadcast screen via the network. For example, the subtitle service system for remote speech recognition in item 4 of the patent scope, wherein the listener opens YouTube, Facebook or Twitch platform to receive images and sounds from the speaker; The web server page of the ASR subtitle server, read and listen to the text and sound of the speaker from this page; the listener is set to have readable and writable permissions on the ASR subtitle server, so it can be used on the ASR subtitle server The text generated by the open source speech recognition toolkit is modified on the web server page. For example, the subtitle service system for remote speech recognition in claim 5 of the patent scope, wherein the speaker logs in the web server page of the ASR subtitle server through web browsing from the live broadcast device, and reads the text; the speaker Combine the live video and the text and output it to the OBS live software for superimposition, and then the OBS live software will push it to YouTube, Facebook or Twitch, etc. The platform enables the hearing-impaired person to see the text content in a subtitle area on the live broadcast screen. Such as the subtitle service system for remote speech recognition in item 1 of the scope of the patent application, wherein the network is a local area network or a global Internet. For example, in the subtitle service system for remote speech recognition in item 2 of the scope of the patent application, the instant message protocol (RTMP) can be replaced by Apple's HTTP Live Streaming (HLS), or replaced by any network transmission protocol with similar functions. For example, the subtitle service system for remote speech recognition in item 2 of the patent application scope, in which the open source speech recognition tool is Kaldi ASR, which can be obtained free of charge under the Apache License v2.0. For example, the subtitle service system for remote voice recognition in item 1 of the scope of the patent application, wherein if the location A, the location B, and the location C are in the same location, it means that the speaker, the typing personnel, and the hearing-impaired person are in the same location. classroom or conference room.

Images