KR102523814B1 - Electronic apparatus that outputs subtitle on screen where video is played based on voice recognition and operating method thereof - Google Patents

Abstract

Disclosed are an electronic device and an operating method for outputting a caption on a screen on which a video is reproduced based on voice recognition. The present invention provides an electronic device and an operating method for outputting subtitles on a screen on which a video is played based on voice recognition, so that a user can conveniently view subtitles at the playback time he/she wants to watch when watching a video. can support you

Description

Electronic device for outputting subtitles on the screen where video is played based on voice recognition and its operating method

The present invention relates to an electronic device that outputs a caption on a screen on which an image is reproduced based on voice recognition and an operating method thereof.

Recently, as the corona era is prolonged, the ratio of watching video content indoors is increasing. Accordingly, the amount of real-time TV viewing has increased, and the number of VOD views has also increased.

In this way, as the ratio of viewing video content indoors increases, the need to support more efficient functions for viewers in relation to video content is also increasing.

In this regard, in the existing technology, when a viewer watches video content, a function of outputting subtitles on the screen on which the video content is played based on voice recognition is not performed, so that the viewer can view the playback time he or she wants to watch. It wasn't easy to see the subtitles in .

If, when a viewer watches video content, it is possible to output subtitles on the screen where the video content is played based on voice recognition, so that the viewer can view the subtitles at the playback point they want to watch. If the technology is introduced, viewers will feel more convenient when viewing video content.

Therefore, it is necessary to research a caption service technology capable of outputting captions on a screen on which a video is played based on voice recognition.

The present invention provides an electronic device and an operating method for outputting subtitles on a screen on which a video is played based on voice recognition, so that a user can conveniently view subtitles at the playback time he/she wants to watch when watching a video. We want to support you so that

According to an embodiment of the present invention, an electronic device that outputs a caption on a screen on which an image is reproduced based on voice recognition reproduces the first image data when a play command for the first image data is received by a user, A plurality of divided image data is generated by dividing the first image data by a first playback time interval set in advance, and information about a playback time point of each of the plurality of divided image data in the total playback time of the first image data. A segmentation unit that checks subtitles for generating a plurality of subtitles corresponding to each of the plurality of segmented image data by sequentially applying the plurality of segmented image data as inputs to a pre-built voice recognition model to perform voice recognition. a generator, a caption table generator for generating a caption table in which the plurality of subtitles and information about reproduction time of the divided video data corresponding to each of the plurality of subtitles are recorded in correspondence with each other; and the first video data is reproduced. In the meantime, if a command to display a caption for the first image data is received by the user at the first playback time, the subtitle table is displayed at each reproduction time recorded in the subtitle table from the first playback time. and a caption display unit which floats and displays recorded captions according to each playback time at a first preset point in a screen area where an image is displayed.

In addition, in an operating method of an electronic device that outputs a subtitle on a screen on which an image is reproduced based on voice recognition according to an embodiment of the present invention, when a play command for the first image data is received by the user, the first image data is played. While reproducing data, a plurality of divided image data is generated by dividing the first image data by a first playback time interval set in advance, and each of the plurality of divided image data is generated during the entire reproduction time of the first image data. Checking information about playback time; Sequentially applying the plurality of divided image data as inputs to a pre-built voice recognition model to perform voice recognition, thereby providing a plurality of subtitles corresponding to each of the plurality of divided image data. generating subtitles, generating a subtitle table in which the plurality of subtitles and information about reproduction time of the divided video data corresponding to each of the plurality of subtitles are recorded in correspondence with each other, and while the first video data is being reproduced , When a command to display a caption for the first video data is received by the user at the first reproduction time, the caption table is recorded at each reproduction time recorded in the caption table from the first reproduction time. and displaying a subtitle according to each reproduction time point by floating it at a first preset point in a screen area where an image is displayed.

The present invention provides an electronic device and an operating method for outputting subtitles on a screen on which a video is played based on voice recognition, so that a user can conveniently view subtitles at the playback time he/she wants to watch when watching a video. can support you

1 is a diagram showing the structure of an electronic device that outputs a caption on a screen on which an image is reproduced based on voice recognition according to an embodiment of the present invention.
2 and 3 are diagrams for explaining an operation of an electronic device that outputs a caption on a screen on which an image is reproduced based on voice recognition according to an embodiment of the present invention.
4 is a flowchart illustrating an operating method of an electronic device that outputs a caption on a screen on which a video is reproduced based on voice recognition according to an embodiment of the present invention.

Hereinafter, embodiments according to the present invention will be described in detail with reference to the accompanying drawings. This description is not intended to limit the present invention to specific embodiments, but should be understood to include all modifications, equivalents, and substitutes included in the spirit and scope of the present invention. While describing each drawing, similar reference numerals have been used for similar components, and unless otherwise defined, all terms used in this specification, including technical or scientific terms, are common knowledge in the art to which the present invention belongs. has the same meaning as commonly understood by the person who has it.

In this document, when a certain component is said to "include", it means that it may further include other components without excluding other components unless otherwise stated. In addition, in various embodiments of the present invention, each component, functional block, or means may be composed of one or more sub-components, and the electrical, electronic, and mechanical functions performed by each component are electronic It may be implemented with various known elements or mechanical elements such as circuits, integrated circuits, ASICs (Application Specific Integrated Circuits), and may be implemented separately or two or more may be integrated into one.

On the other hand, the blocks of the accompanying block diagram or the steps of the flowchart are computer program instructions that perform designated functions by being loaded into a processor or memory of a device capable of data processing, such as a general-purpose computer, a special purpose computer, a portable notebook computer, and a network computer. can be interpreted as meaning Since these computer program instructions may be stored in a memory included in a computer device or in a computer readable memory, the functions described in blocks of a block diagram or steps of a flowchart are produced as a product containing instruction means for performing them. It could be. Further, each block or each step may represent a module, segment or portion of code that includes one or more executable instructions for executing specified logical function(s). Also, it should be noted that in some alternative embodiments, functions mentioned in blocks or steps may be executed out of a predetermined order. For example, two blocks or steps shown in succession may be performed substantially simultaneously or in reverse order, and in some cases, some blocks or steps may be omitted.

1 is a diagram showing the structure of an electronic device that outputs a caption on a screen on which an image is reproduced based on voice recognition according to an embodiment of the present invention.

Referring to FIG. 1 , an electronic device 110 according to an embodiment of the present invention includes a division unit 111, a caption generator 112, a caption table generator 113, and a caption display unit 114.

The division unit 111 reproduces the first image data when a reproduction command for the first image data is received by the user, and divides the first image data by a first playback time interval set in advance to divide the first image data into a plurality of divisions. Image data is generated, and information about a reproduction time point of each of the plurality of divided image data is checked in the entire reproduction time of the first image data.

The caption generator 112 sequentially applies the plurality of divided image data as inputs to a pre-built voice recognition model to perform voice recognition, thereby generating a plurality of captions corresponding to each of the plurality of divided image data. .

The caption table generating unit 113 generates a caption table in which the plurality of captions and information about playback time of the divided video data corresponding to each of the plurality of captions are recorded in correspondence with each other.

While the first image data is being reproduced, the caption display unit 114, when receiving a command to display a caption for the first image data from the user at the first reproduction time, records the caption in the caption table from the first reproduction time. At each playback time, the subtitle according to each playback time recorded in the caption table is floated and displayed at a preset first point in the screen area where the video is displayed.

Hereinafter, with reference to FIGS. 1 and 2, operations of the division unit 111, the caption generation unit 112, the caption table generation unit 113, and the caption display unit 114 will be described in detail by way of example. .

First, when a user's command to reproduce the first image data is received by the electronic device 110, the division unit 111 reproduces the first image data for a preset first reproduction time. By dividing by intervals, a plurality of divided image data may be generated.

If the preset first playback time interval is '1 minute' and the total playback time of the first image data is '5 minutes', the dividing unit 111 converts the first image data to '1 minute'. By splitting at intervals, a plurality of split image data having a playback time of '1 minute' can be created as 'segmented image data 1, split image data 2, split image data 3, split image data 4, and split image data 5'. can

Thereafter, the segmentation unit 111 divides 'segmented image data 1, segmented image data 2, segmented image data 3, segmented image data 4, and segmented image data 5' in the entire reproduction time of '5 minutes' of the first image data. Information about each playback time, 'playback time 1 (0 seconds), playback time 2 (1 minute), playback time 3 (2 minutes), playback time 4 (3 minutes), playback time 5 (4 minutes)' You can check.

Then, the subtitle generator 112 sequentially applies 'segmented image data 1, segmented image data 2, segmented image data 3, segmented image data 4, and segmented image data 5' as inputs to the pre-built speech recognition model. By performing voice recognition, a plurality of subtitles corresponding to 'segmented image data 1, segmented image data 2, segmented image data 3, segmented image data 4, and segmented image data 5' are selected as 'caption 1, subtitle 2, subtitle 3, It can be created with subtitle 4 and subtitle 5'.

After that, the subtitle table generating unit 113 generates divided images corresponding to 'caption 1, subtitle 2, subtitle 3, subtitle 4, and subtitle 5' and 'caption 1, subtitle 2, subtitle 3, subtitle 4, and subtitle 5', respectively. Data 'Split video data 1, Split video data 2, Split video data 3, Split video data 4, Split video data 5' playback time 'playback time 1 (0 sec), playback time 2 (1 minute), playback A subtitle table recorded by matching information on the time point 3 (2 minutes), playback time 4 (3 minutes), and playback time 5 (4 minutes)' can be generated as shown in Table 1 below.

multiple subtitles Playback point of split video data subtitles 1 Playback time of split video data 1 1 (0 sec) subtitles 2 Playback time of split video data 2 2 (1 minute) subtitles 3 Playback time of split video data 3 3 (2 minutes) subtitles 4 Playback time of split video data 4 4 (3 minutes) subtitles 5 Playback time of split video data 5 5 (4 minutes)

At this time, while the first video data is being reproduced by the electronic device 110, the electronic device 110 receives a subtitle display command for the first video data from the user at the playback time of '1 minute and 30 seconds'. Let's assume that

Then, the caption display unit 114 displays each reproduction time recorded in the caption table as shown in Table 1 from '1 minute and 30 seconds', 'playback time 3 (2 minutes), play time 4 (3 minutes), play time 5 (4 minutes)', 'Subtitle 3, Subtitle 4, and Subtitle 5', which are subtitles according to each playback time recorded in the subtitle table as shown in Table 1, are displayed in the screen area where the video is displayed. It can be displayed by plotting at 1 point.

Specifically, as shown in FIG. 2 , when the playback time of the first video data is 'playback time 3 (2 minutes)', the caption display unit 114 displays 'caption 3' in the screen area where the video is displayed. It can be displayed by floating at the first point 211 in 210, and when the playback time of the first video data is 'playback time 4 (3 minutes)', 'caption 4' is displayed on the screen It can be displayed by floating at the first point 211 in the area 210, and when the playback time of the first video data becomes 'playback time 5 (4 minutes)', 'caption 5' is displayed when the video is displayed. It can be displayed by floating at the first point 211 in the screen area 210 .

At this time, according to an embodiment of the present invention, the caption generator 112, for each of the plurality of divided image data, uses a frequency band other than the first frequency band preset as a frequency band corresponding to human voice. After all sound components have been removed, each segmented image data from which all sound components in frequency bands other than the first frequency band have been removed is sequentially applied as an input to the voice recognition model to perform voice recognition, thereby generating the plurality of subtitles. can create

For example, assume that the first frequency band preset as a frequency band corresponding to human voice is '50 Hz to 4000 Hz'.

At this time, according to the above-described example, the caption generator 112 responds to human voice for 'segmented video data 1, segmented video data 2, segmented video data 3, segmented video data 4, and segmented video data 5', respectively. It is possible to remove all sound components of frequency bands other than the preset first frequency band '50Hz ~ 4000Hz'.

Thereafter, the subtitle generator 112 sequentially applies each segmented image data from which all sound components of frequency bands other than the first frequency band '50Hz to 4000Hz' have been removed to the speech recognition model as inputs for speech recognition. , it is possible to generate the plurality of subtitles as 'caption 1, subtitle 2, subtitle 3, subtitle 4, and subtitle 5'.

According to one embodiment of the present invention, the electronic device 110 may further include a translated caption generator 115, a translated caption recording processor 116, and a translated caption addition display unit 117.

When the translated caption generator 115 receives an additional display command instructing to additionally display a caption composed of a first foreign language from the user at a second playback time point while the first image data is being reproduced, the translated caption generator 115 displays the caption Referring to the table, a plurality of translated subtitles are generated by translating the plurality of subtitles into the first foreign language through a pre-built translation engine.

When the plurality of translated captions are generated, the translated caption recording processor 116 additionally records a translated caption corresponding to each of the plurality of captions on the caption table for each of the plurality of captions.

The translated subtitle addition display unit 117 displays a video of subtitles and translated subtitles according to each reproduction time recorded in the subtitle table whenever each reproduction time recorded in the subtitle table is reached from the second reproduction time. It is displayed by floating at the first point and the preset second point in the screen area.

Hereinafter, with reference to FIGS. 1 and 3, operations of the translated caption generator 115, the translated caption recording processor 116, and the translated caption addition display unit 117 will be described in detail, for example.

First, since the first video data is video data dubbed in 'Korean', a plurality of subtitles generated by the subtitle generator 112 are referred to as 'Korean' subtitles, and the first foreign language is 'English'. While the first video data is being reproduced, at the playback time of '2 minutes and 30 seconds', an additional display command instructing the user to additionally display subtitles composed of 'English' is sent to the electronic device 110. Suppose it is received by

Then, the translation subtitle generation unit 115 refers to the subtitle table shown in Table 1 above, and translates 'subtitle 1, subtitle 2, subtitle 3, subtitle 4, and subtitle 5' into 'English' through a pre-built translation engine. By doing so, a plurality of translated subtitles can be generated as 'translated subtitle 1, translated subtitle 2, translated subtitle 3, translated subtitle 4, and translated subtitle 5'.

In this way, when the plurality of translated subtitles are generated by the translated subtitle generator 115, the translated subtitle record processing unit 116 displays 'subtitle 1, subtitle 2, subtitle 3, subtitle 4, subtitle on the subtitle table as shown in Table 1 above. 5', respectively, 'translation subtitle 1, translation subtitle 2, translation subtitle 3, translation subtitle 4, translation subtitle 5' corresponding to 'subtitle 1, subtitle 2, subtitle 3, subtitle 4, and subtitle 5' respectively. It can be recorded as shown in Table 2 below by additional correspondence.

multiple subtitles multiple translated subtitles Playback point of split video data subtitles 1 translation subtitles 1 Playback time of split video data 1 1 (0 sec) subtitles 2 translation subtitles 2 Playback time of split video data 2 2 (1 minute) subtitles 3 translation subtitles 3 Playback time of split video data 3 3 (2 minutes) subtitles 4 translation subtitles 4 Playback time of split video data 4 4 (3 minutes) subtitles 5 translation subtitles 5 Playback time of split video data 5 5 (4 minutes)

After that, the translated caption addition display unit 117 displays each playback time recorded in the subtitle table as shown in Table 2 from the playback time of '2 minutes and 30 seconds', 'playback time 4 (3 minutes), playback time 5 (4 minutes) minutes)', the subtitles 'subtitle 4, subtitle 5' and translation subtitles 'translation subtitle 4, translation subtitle 5' according to each playback time recorded in the subtitle table as shown in Table 2 above are displayed. It may be displayed by floating at the first point and a preset second point in the displayed screen area.

In more detail, as shown in FIG. Subtitle 4' can be displayed by floating at the first point 211 and the second point 311 in the screen area 210 where the video is displayed, and the playback time of the first video data is 'playback time 5'. (4 minutes)', 'subtitles 5' and 'translated subtitles 5' are plotted and displayed at the first point 211 and the second point 311 respectively in the screen area 210 where the video is displayed can do.

According to one embodiment of the present invention, the electronic device 110 may further include a voice synthesis unit 118, a mixing unit 119, a combination unit 120, and a dubbing playback unit 121.

While the first video data is being reproduced, the voice synthesis unit 118 receives a dubbing reproduction command requesting dubbing and reproduction of the first video data in the first foreign language from the user at a third playback time, A plurality of synthesized voices corresponding to the plurality of translated captions are generated by applying each of the plurality of translated captions as an input to a pre-constructed voice synthesis model and performing voice synthesis.

The mixing unit 119 removes a sound component of a first frequency band preset as a frequency band corresponding to human voice from each of the plurality of divided image data, and then converts the plurality of synthesized voices into the first frequency band. A plurality of mixed image data is generated by mixing each of the divided image data from which the sound component of the band has been removed.

The combining unit 120 generates first dubbing video data by combining the plurality of mixing video data into one video data according to the playback time order.

The dubbing reproducing unit 121 reproduces the first dubbing image data from the third reproducing time point.

Hereinafter, operations of the voice synthesis unit 118, the mixing unit 119, the combining unit 120, and the dubbing playback unit 121 will be described in detail by way of example.

First, it is assumed that subtitles and translated subtitles are generated as shown in Table 2, and while the first video data is being reproduced in the electronic device 110, the user selects the first video data at a playback time of '3 minutes and 30 seconds'. Assume that the electronic device 110 receives a dubbing play command requesting dubbing and playing video data in 'English'.

Then, the speech synthesis unit 118 performs speech synthesis by applying 'translated subtitles 1, translated subtitles 2, translated subtitles 3, translated subtitles 4, and translated subtitles 5' as inputs to a pre-built speech synthesis model, ' Translated subtitles 1, translated subtitles 2, translated subtitles 3, translated subtitles 4, and translated subtitles 5' are converted into 'synthesized speech 1, synthetic speech 2, synthetic speech 3, synthetic speech 4, and synthetic speech 5'. can create

Then, the mixing unit 119 preliminarily determines that each of 'segmented image data 1, segmented image data 2, segmented image data 3, segmented image data 4, and segmented image data 5' is a frequency band corresponding to human voice. A sound component of the set first frequency band may be removed.

If the first frequency band preset as a frequency band corresponding to a human voice is '50Hz to 4000Hz', the mixing unit 119 performs 'split image data 1, split image data 2, split image data 3, For each of the divided image data 4 and the divided image data 5', a sound component of '50 Hz to 4000 Hz', which is a first frequency band preset as a frequency band corresponding to human voice, may be removed.

Thereafter, the mixing unit 119 divides 'synthesized voice 1, synthesized voice 2, synthesized voice 3, synthesized voice 4, and synthesized voice 5' from which sound components of the first frequency band '50Hz to 4000Hz' have been removed. By mixing with video data, a plurality of mixing video data can be generated as 'mixing video data 1, mixing video data 2, mixing video data 3, mixing video data 4, and mixing video data 5'.

In detail, the mixing unit 119 may generate 'mixed video data 1' by mixing 'synthesized voice 1' and 'segmented image data 1' from which the sound component of the human voice has been removed. 2' and 'Split Video Data 2' from which the human voice component has been removed can be mixed together to create 'Mixed Video Data 2', and 'Synthetic Voice 3' and the human voice component removed 'Mixed video data 3' can be created by mixing 'split video data 3', and 'synthetic voice 4' and 'split video data 4' from which the sound components for human voices have been removed are mixed together to create 'mixed video data 3'. 'Mixed video data 4' may be generated, and 'mixed video data 5' may be generated by mixing the 'synthesized voice 5' and the 'divided image data 5' from which the sound component of the human voice is removed.

In this way, when 'mixing video data 1, mixing video data 2, mixing video data 3, mixing video data 4, and mixing video data 5' are generated by the mixing unit 119, the combiner 120 generates 'mixing video data 1'. , mixing video data 2, mixing video data 3, mixing video data 4, and mixing video data 5' are combined into one image data in order of playback time, thereby generating first dubbing video data.

Then, the dubbing reproducing unit 121 may reproduce the first dubbing image data from a playback time of '3 minutes and 30 seconds'.

Through this, the user can enjoy the first dubbing video data dubbed in 'English' from '3 minutes and 30 seconds'.

According to an embodiment of the present invention, the electronic device 110 may further include a confirmation unit 122, an extraction unit 123, a search result display unit 124, and a reproduction unit 125.

While the first image data is being reproduced, the confirmation unit 122 is configured to search for a reproduction time matching the first keyword in the first image data by the user while the user inputs the first keyword. When an indicated search command is received, it is checked whether a caption including the first keyword exists among the plurality of captions with reference to the caption table.

The extraction unit 123 extracts a subtitle including the first keyword from among the plurality of subtitles, and if it is determined that at least one first subtitle exists, a subtitle corresponding to the at least one first subtitle is obtained from the subtitle table. Information on at least one fourth reproduction time point is extracted.

The search result display unit 124 designates the information on the at least one fourth reproduction time point as a search result matching the first keyword and displays it at a preset third point in the screen area where the video is displayed.

In a state in which information on the at least one fourth reproduction time point is displayed at the third point, the playback unit 125 controls a fifth playback time point, which is any one of the at least one fourth playback time point, by the user. When a selective reproduction command is received, the first image data is reproduced from the fifth reproduction time point.

Hereinafter, operations of the confirmation unit 122, the extraction unit 123, the search result display unit 124, and the reproduction unit 125 will be described in detail, for example.

First, while the first image data is being reproduced, a first keyword is input by the user, and a search command instructs the user to search for a reproduction time point matching the first keyword in the first image data. Assume that this is received by the electronic device 110.

Then, the verification unit 122 refers to the caption table as shown in Table 1, and among the plurality of captions 'caption 1, caption 2, caption 3, caption 4, and caption 5' includes the first keyword. You can check whether it exists or not.

If, through the verification unit 122, 'caption 1, subtitle 3, and subtitle 5' exist as subtitles including the first keyword among 'caption 1, subtitle 2, subtitle 3, subtitle 4, and subtitle 5' If confirmed, the extractor 123 selects 'playback time 1 (0 sec), playback time 3 (2 minutes)', which are playback times corresponding to 'caption 1, subtitle 3, and subtitle 5' from the subtitle table as shown in Table 1 above. ), information on playback time 5 (4 minutes)' can be extracted.

Thereafter, the search result display unit 124 designates information on 'playback time 1 (0 sec), playback time 3 (2 minutes), and playback time 5 (4 minutes)' as a search result matching the first keyword. Accordingly, the image may be displayed at a preset third point in the screen area where the image is displayed.

In this way, in a state in which information on 'playback time 1 (0 second), play time 3 (2 minutes), and play time 5 (4 minutes)' is displayed at the third point by the search result display unit 124, the A selection playback command for 'playback time 3 (2 minutes)', which is any one playback time among 'playback time 1 (0 seconds), playback time 3 (2 minutes), and playback time 5 (4 minutes)', is issued by the user. When received by the electronic device 110, the playback unit 125 can reproduce the first image data from 'playback time 3 (2 minutes)'.

Through this, the user can enjoy the first image data from 'playback point 3 (2 minutes)'.

4 is a flowchart illustrating an operating method of an electronic device that outputs a caption on a screen on which a video is reproduced based on voice recognition according to an embodiment of the present invention.

In step S410, when a play command for the first image data is received by the user, the first image data is reproduced, and the first image data is divided by a first play time interval set in advance, thereby forming a plurality of divided images. Data is generated, and information about a reproduction time point of each of the plurality of divided image data is checked in the entire reproduction time of the first image data.

In step S420, a plurality of subtitles corresponding to each of the plurality of divided image data are generated by sequentially applying the plurality of divided image data as inputs to a pre-built voice recognition model to perform voice recognition.

In step S430, a caption table is created in which the plurality of captions and information about reproduction time of the divided video data corresponding to each of the plurality of captions are recorded in correspondence with each other.

In step S440, if a command to display a caption for the first image data is received from the user at the first reproduction time while the first image data is being reproduced, it is recorded in the caption table from the first reproduction time. Whenever there is a playback time, the subtitle according to each playback time recorded in the caption table is floated and displayed at a first preset point in the screen area where the video is displayed.

At this time, according to one embodiment of the present invention, in step S420, for each of the plurality of divided image data, sound components of a frequency band other than the first frequency band preset to be a frequency band corresponding to human voice After all of the subtitles are removed, the plurality of subtitles are generated by sequentially applying each segmented image data from which all of the sound components of the frequency band other than the first frequency band have been removed as inputs to the speech recognition model to perform speech recognition. can

Further, according to an embodiment of the present invention, the operating method of the electronic device instructs the user to additionally display subtitles in a first foreign language at a second playback time while the first image data is being reproduced. generating a plurality of translated subtitles by translating the plurality of subtitles into the first foreign language through a pre-built translation engine with reference to the subtitle table when an additional display command to display the subtitle is received; If generated, additionally correspondingly and recording translated subtitles corresponding to each of the plurality of subtitles on the subtitle table, and each of the subtitles recorded in the subtitle table from the second reproduction time point. Whenever a play point comes, plotting and displaying subtitles and translated subtitles according to each play point recorded in the subtitle table at the first point and a preset second point in the screen area where the video is displayed, respectively. can include more.

At this time, according to an embodiment of the present invention, the operating method of the electronic device may include dubbing the first image data into the first foreign language by the user at a third playback point while the first image data is being reproduced. When a dubbing playback command requesting reproduction is received, a plurality of synthesized voices corresponding to the plurality of translated subtitles are generated by applying each of the plurality of translated subtitles as an input to a pre-constructed speech synthesis model and performing speech synthesis. Generating, for each of the plurality of divided image data, after removing a sound component of a first frequency band preset to be a frequency band corresponding to human voice, the plurality of synthesized voices of the first frequency band Generating a plurality of mixing image data by mixing each divided image data from which sound components have been removed, generating first dubbing image data by combining the plurality of mixed image data into one image data in order of reproduction time and reproducing the first dubbing video data from the third reproduction time point.

In addition, according to an embodiment of the present invention, the operating method of the electronic device may include, while the first image data is being reproduced, the user inputs a first keyword, and the user inputs the first keyword in the first image data. When a search command instructing to search for a playback time matching a first keyword is received, checking whether a subtitle including the first keyword among the plurality of subtitles exists by referring to the subtitle table , If it is determined that at least one first caption among the plurality of captions including the first keyword exists, at least one fourth caption corresponding to the at least one first caption is obtained from the caption table. extracting information on playback time; specifying the at least one fourth playback time information as a search result matching the first keyword and displaying the information at a preset third point in a screen area where an image is displayed; and in a state in which the information on the at least one fourth reproduction time point is displayed at the third point, a selection playback command for a fifth playback time point, which is any one of the at least one fourth playback time point, is received by the user. , the method may further include reproducing the first image data from the fifth reproducing time point.

In the above, the operating method of the electronic device according to an embodiment of the present invention has been described with reference to FIG. 4 . Here, since the operating method of the electronic device according to an embodiment of the present invention may correspond to the configuration of the operation of the electronic device 110 described with reference to FIGS. 1 to 3, a detailed description thereof will be omitted. do.

A method of operating an electronic device that outputs a caption on a screen on which an image is reproduced based on voice recognition according to an embodiment of the present invention may be implemented as a computer program stored in a storage medium for execution through a combination with a computer.

In addition, a method of operating an electronic device that outputs subtitles on a screen on which an image is reproduced based on voice recognition according to an embodiment of the present invention is implemented in the form of computer program commands to be executed through combination with a computer, so that the computer can read them. can be recorded on media. The computer readable medium may include program instructions, data files, data structures, etc. alone or in combination. Program instructions recorded on the medium may be those specially designed and configured for the present invention or those known and usable to those skilled in computer software. Examples of computer-readable recording media include magnetic media such as hard disks, floppy disks and magnetic tapes, optical media such as CD-ROMs and DVDs, and magnetic media such as floptical disks. - includes hardware devices specially configured to store and execute program instructions, such as magneto-optical media, and ROM, RAM, flash memory, and the like. Examples of program instructions include high-level language codes that can be executed by a computer using an interpreter, as well as machine language codes such as those produced by a compiler.

As described above, the present invention has been described by specific details such as specific components and limited embodiments and drawings, but these are provided to help a more general understanding of the present invention, and the present invention is not limited to the above embodiments. , Those skilled in the art in the field to which the present invention belongs can make various modifications and variations from these descriptions.

Therefore, the spirit of the present invention should not be limited to the described embodiments, and it will be said that not only the claims to be described later, but also all modifications equivalent or equivalent to these claims belong to the scope of the present invention. .

110: Electronic device that outputs subtitles on the screen on which video is played based on voice recognition
111: division unit 112: subtitle generation unit
113: subtitle table generation unit 114: subtitle display unit
115: translation subtitle generation unit 116: translation subtitle record processing unit
117: Translation subtitle addition display unit 118: Voice synthesis unit
119: mixing unit 120: coupling unit
121: dubbing playback unit 122: confirmation unit
123: extraction unit 124: search result display unit
125: regeneration unit

Claims (12)

An electronic device that outputs subtitles on a screen on which a video is reproduced based on voice recognition, comprising:
When a playback command for the first image data is received by the user, the first image data is reproduced, and a plurality of divided image data is generated by dividing the first image data by a first playback time interval set in advance; a segmentation unit that checks information about a reproduction point of each of the plurality of divided image data within the total reproduction time of the first image data;
a caption generating unit generating a plurality of subtitles corresponding to each of the plurality of divided image data by sequentially applying the plurality of divided image data as inputs to a pre-built voice recognition model and performing voice recognition;
a caption table generating unit generating a caption table in which the plurality of captions and information about playback time of the divided video data corresponding to each of the plurality of captions are recorded in correspondence with each other;
While the first video data is being reproduced, if a command to display a caption for the first video data is received by the user at the first reproduction time, each reproduction time recorded in the caption table from the first reproduction time a caption display unit that floats and displays captions according to each playback time recorded in the caption table at a first preset point in a screen area where an image is displayed;
While the first video data is being reproduced, if an additional display command instructing to additionally display a subtitle composed of a first foreign language is received from the user at a second reproduction time point, the plurality of subtitles are displayed with reference to the subtitle table. a translation subtitle generation unit that generates a plurality of translated subtitles by translating subtitles into the first foreign language through a pre-built translation engine;
a translated caption recording processor configured to, when the plurality of translated captions are generated, additionally map and record translated captions corresponding to each of the plurality of captions to each of the plurality of captions on the caption table;
Every time from the second reproduction time to each reproduction time recorded in the subtitle table, subtitles and translated subtitles according to each reproduction time recorded in the subtitle table are displayed at the first point in the screen area where the video is displayed. and a translation subtitle addition display unit which floats and displays at a preset second point, respectively;
While the first video data is being played, if a dubbing playback command requesting dubbing and playing the first video data in the first foreign language is received from the user at a third playback time, each of the plurality of translated subtitles is displayed. a voice synthesis unit generating a plurality of synthesized voices corresponding to the plurality of translated subtitles by applying as an input to a pre-built voice synthesis model and performing voice synthesis;
For each of the plurality of divided image data, after removing a sound component of a first frequency band preset as a frequency band corresponding to human voice, the sound component of the first frequency band is removed from the plurality of synthesized voices. a mixing unit generating a plurality of mixed image data by mixing each divided image data;
a combiner configured to generate first dubbing video data by combining the plurality of mixing video data into one video data in order of reproduction time; and
A dubbing reproducing unit to reproduce the first dubbing video data from the third reproduction time point.
An electronic device comprising a. According to claim 1,
The subtitle generator
For each of the plurality of split image data, after removing all sound components in frequency bands other than the first frequency band preset as a frequency band corresponding to human voice, sound in frequency bands other than the first frequency band The electronic device characterized in that the electronic device generates the plurality of subtitles by sequentially applying the divided image data from which all components have been removed as inputs to the speech recognition model to perform speech recognition. delete delete According to claim 1,
While the first image data is being reproduced, while a first keyword is input by the user, a search command for instructing to search for a reproduction point matching the first keyword in the first image data is received by the user. a confirmation unit for checking whether a caption including the first keyword exists among the plurality of captions by referring to the caption table;
If it is determined that at least one first subtitle among the plurality of subtitles including the first keyword exists, at least one fourth play corresponding to the at least one first subtitle from the subtitle table. an extraction unit for extracting information about a viewpoint;
a search result display unit that designates the information on the at least one fourth playback time point as a search result matching the first keyword and displays it at a preset third point in a screen area where an image is displayed; and
In a state in which the information on the at least one fourth reproduction time point is displayed at the third point, when a selection reproduction command for a fifth reproduction time point, which is any one of the at least one fourth reproduction time point, is received by the user , a playback unit that reproduces the first image data from the fifth playback time point
An electronic device further comprising a. A method of operating an electronic device that outputs subtitles on a screen on which a video is reproduced based on voice recognition, the method comprising:
When a playback command for the first image data is received by the user, the first image data is reproduced, and a plurality of divided image data is generated by dividing the first image data by a first playback time interval set in advance; checking information about a playback time point of each of the plurality of divided image data within a total playback time of the first video data;
generating a plurality of subtitles corresponding to each of the plurality of divided image data by sequentially applying the plurality of divided image data as inputs to a pre-built voice recognition model and performing voice recognition;
generating a caption table in which the plurality of captions and information about playback time of divided video data corresponding to each of the plurality of captions are recorded in correspondence with each other;
While the first video data is being reproduced, if a command to display a caption for the first video data is received by the user at the first reproduction time, each reproduction time recorded in the caption table from the first reproduction time displaying a floating subtitle according to each playback time recorded in the subtitle table at a first preset point in a screen area where an image is displayed whenever a subtitle is displayed;
While the first video data is being reproduced, if an additional display command instructing to additionally display a subtitle composed of a first foreign language is received from the user at a second reproduction time point, the plurality of subtitles are displayed with reference to the subtitle table. generating a plurality of translated subtitles by translating subtitles into the first foreign language through a pre-built translation engine;
when the plurality of translated subtitles are generated, additionally correspondingly recording a translated subtitle corresponding to each of the plurality of subtitles to each of the plurality of subtitles on the subtitle table;
Every time from the second reproduction time to each reproduction time recorded in the subtitle table, subtitles and translated subtitles according to each reproduction time recorded in the subtitle table are displayed at the first point in the screen area where the video is displayed. Floating and displaying at the second and preset points, respectively;
While the first video data is being played, if a dubbing playback command requesting dubbing and playing the first video data in the first foreign language is received from the user at a third playback time, each of the plurality of translated subtitles is displayed. generating a plurality of synthesized speeches corresponding to the plurality of translated subtitles by applying as an input to a pre-built speech synthesis model to perform speech synthesis;
For each of the plurality of divided image data, after removing a sound component of a first frequency band preset as a frequency band corresponding to human voice, the sound component of the first frequency band is removed from the plurality of synthesized voices. generating a plurality of mixed image data by mixing each divided image data;
generating first dubbing video data by combining the plurality of mixing video data into one video data in order of reproduction time; and
Reproducing the first dubbing video data from the third reproduction time point
A method of operating an electronic device comprising a. According to claim 6,
The step of generating the plurality of subtitles is
For each of the plurality of split image data, after removing all sound components in frequency bands other than the first frequency band preset as a frequency band corresponding to human voice, sound in frequency bands other than the first frequency band The operating method of the electronic device, characterized in that generating the plurality of subtitles by sequentially applying the divided image data from which all components have been removed as inputs to the speech recognition model to perform speech recognition. delete delete According to claim 6,
While the first image data is being reproduced, while a first keyword is input by the user, a search command for instructing to search for a reproduction point matching the first keyword in the first image data is received by the user. checking whether a caption including the first keyword exists among the plurality of captions by referring to the caption table;
If it is determined that at least one first subtitle among the plurality of subtitles including the first keyword exists, at least one fourth play corresponding to the at least one first subtitle from the subtitle table. extracting information about a viewpoint;
designating information on the at least one fourth playback point as a search result matching the first keyword and displaying the information at a preset third point in a screen area where an image is displayed; and
In a state in which the information on the at least one fourth reproduction time point is displayed at the third point, when a selection reproduction command for a fifth reproduction time point, which is any one of the at least one fourth reproduction time point, is received by the user , Reproducing the first image data from the fifth playback time point
Method of operating an electronic device further comprising a. A computer-readable recording medium recording a computer program for executing the method of any one of claims 6, 7, or 10 through a combination with a computer. A computer program stored in a storage medium for executing the method of any one of claims 6, 7 or 10 through a combination with a computer.

Images