KR102080315B1 - Method for providing vedio service and service server using the same - Google Patents

Abstract

The present application relates to a video service providing method and a service server using the same. The video service providing method according to the present invention is based on a characteristic change of a sound included in a video, and the unit section is divided into a plurality of unit sections. step; A script string generation step of generating a script string corresponding to the speech by recognizing the speech included in the unit section; A caption string generation step of recognizing a caption image included in the unit section and generating a caption string corresponding to the caption image; And generating a keyword corresponding to the unit section by applying natural language processing to the script string and the subtitle string.

Description

Method for providing video service and service server using same {Method for providing vedio service and service server using the same}

The present application relates to a video service providing method and a service server using the same, and to a video service providing method capable of automatically generating a keyword for each unit section by separating a video based on a semantic basis and a service server using the same.

Recently, with the development of Internet technology, a video service for providing a video through the Internet is widely used. When a user wants to watch a video through the Internet, it is necessary to search for a desired video among numerous videos provided on the Internet, and various video search methods for effective video search have been proposed.

Recently, however, a user is interested in a part of a video rather than the entire video, and an increasing number of users want to watch only that part. For example, a user who wants to watch a football relay may want to watch only a scene where a specific player scores a goal, rather than watching the entire football relay program. However, since a general video search method uses the entire soccer relay as the target of the search, it is difficult to search for some scenes and the like of the video desired by the user.

Korean Patent Registration No. 10-0721409

The present application is to provide a video service providing method and a service server using the same that can automatically generate a keyword for each unit section by separating the video on a semantic basis.

The present application is to provide a video service providing method and a service server using the same that can divide the video into a plurality of unit sections based on the change in the characteristics of the voice in the video.

The present application is to provide a video service providing method and a service server using the same, which can automatically generate a keyword according to the contents of the unit section by applying the voice recognition and subtitle recognition to each unit section separated the video.

The present application, by applying natural language processing using machine learning, to provide a video service providing method that can automatically generate a keyword according to the content of each unit section of the video and a service server using the same.

The video service providing method according to an embodiment of the present invention relates to a video service providing method in which a service server provides a video to a terminal device, wherein the video is provided in a plurality of units based on a change in the characteristics of the voice included in the video. A unit section separation step of separating the section; A script string generation step of generating a script string corresponding to the speech by recognizing the speech included in the unit section; A caption string generation step of recognizing a caption image included in the unit section and generating a caption string corresponding to the caption image; And generating a keyword corresponding to the unit section by applying natural language processing to the script string and the subtitle string.

According to an embodiment of the present invention, a service server includes: a unit section separator configured to separate the video into a plurality of unit sections based on a change in a characteristic of a voice included in the video; A script string generator for recognizing a voice included in the unit section and generating a script string corresponding to the voice; A caption string generator for recognizing a caption image included in the unit section and generating a caption string corresponding to the caption image; And a keyword generator for generating a keyword corresponding to the unit section by applying natural language processing to the script string and the subtitle string.

Service server according to another embodiment of the present invention, a processor; And a memory coupled to the processor, wherein the memory includes one or more modules configured to be executed by the processor, wherein the one or more modules are based on a change in a characteristic of a voice included in a video. A video is divided into a plurality of unit sections, a voice included in the unit section is recognized, a script string corresponding to the voice is generated, a caption image included in the unit section is recognized, and a corresponding subtitle image is generated. And generating a subtitle string and applying natural language processing to the script string and the subtitle string to generate a keyword corresponding to the unit section.

In addition, the solution of the said subject does not enumerate all the characteristics of this invention. Various features of the present invention and the advantages and effects thereof may be understood in more detail with reference to the following specific embodiments.

According to the video service providing method and the service server using the same according to an embodiment of the present invention, since the video is separated based on the change in the characteristics of the voice in the video, it is possible to separate the video without losing the context or meaning.

According to a video service providing method and a service server using the same according to an embodiment of the present invention, the contents included in a unit section are extracted by applying voice recognition and caption recognition, and then, the keyword for each unit section is used. Since it is set, it is possible to set keywords according to the contents of the unit section.

According to a video service providing method and a service server using the same according to an embodiment of the present invention, a user may search for a specific scene included in a video based on content, and generate a summary video based on a specific theme or content. It is possible.

However, the video service providing method and the effect that can be achieved by the service server using the same according to the embodiments of the present invention are not limited to those mentioned above, and other effects not mentioned are not described in the following description. It will be clearly understood by those skilled in the art.

1 is a schematic diagram showing a video service providing system according to an exemplary embodiment of the present invention.
2 and 3 are block diagrams illustrating a service server according to an exemplary embodiment of the present invention.
4 is a schematic diagram illustrating separation of unit sections of a video according to an embodiment of the present invention.
5 is a schematic diagram illustrating generation of a script string and a caption string according to an embodiment of the present invention.
6 is a schematic diagram illustrating detection of a caption image according to an embodiment of the present invention.
7 is a flowchart illustrating a video service providing method according to another exemplary embodiment of the present invention.

Hereinafter, embodiments of the present disclosure will be described in detail with reference to the accompanying drawings, and the same or similar components will be given the same reference numerals regardless of the reference numerals, and redundant description thereof will be omitted. The suffixes "module" and "unit" for components used in the following description are given or mixed in consideration of ease of specification, and do not have distinct meanings or roles. In other words, the term 'part' used in the present invention refers to a hardware component such as software, FPGA or ASIC, and 'part' plays a role. But wealth is not limited to software or hardware. The 'unit' may be configured to be in an addressable storage medium and may be configured to play one or more processors. Thus, as an example, a 'part' may include components such as software components, object-oriented software components, class components, and task components, processes, functions, properties, procedures, Subroutines, segments of program code, drivers, firmware, microcode, circuits, data, databases, data structures, tables, arrays, and variables. The functionality provided within the components and 'parts' may be combined into a smaller number of components and 'parts' or further separated into additional components and 'parts'.

In addition, in the following description of the embodiments disclosed herein, when it is determined that the detailed description of the related known technology may obscure the gist of the embodiments disclosed herein, the detailed description thereof will be omitted. In addition, the accompanying drawings are only for easily understanding the embodiments disclosed in the present specification, the technical idea disclosed in the specification by the accompanying drawings are not limited, and all changes included in the spirit and scope of the present invention. It should be understood to include equivalents and substitutes.

1 is a schematic diagram showing a video service providing system according to an exemplary embodiment of the present invention.

Referring to FIG. 1, the video service providing system according to an exemplary embodiment of the present invention may include a terminal device 1 and a service server 100.

Hereinafter, a video service providing system according to an exemplary embodiment of the present invention will be described with reference to FIG. 1.

The terminal device 1 may communicate with the service server 100 through a network, and may receive a video service provided by the service server 100. The terminal device 1 may include a display unit, a speaker, or the like for providing visual or audio content to a user, such as a video, and may include an input unit for receiving a user's input, a memory and at least one program stored therein. It may include.

The terminal device 1 may be a fixed device such as a mobile terminal or a desktop such as a smartphone or a tablet PC, and according to an embodiment, a mobile phone, a smart phone, a laptop computer, a digital broadcasting terminal, Personal digital assistants (PDAs), portable multimedia players (PMPs), slate PCs, tablet PCs, ultrabooks, wearable devices (e.g., smartwatches) , A glass type terminal (smart glass), a head mounted display (HMD), etc. may correspond to the terminal device 1.

The network connecting the terminal device 1 and the service server 100 may include a wired network and a wireless network, and specifically, a local area network (LAN) and a metropolitan area network (MAN). And various networks such as a wide area network (WAN). The network may also include a known World Wide Web (WWW). However, the network according to the present invention is not limited to the networks listed above, and may include a known wireless data network, a known telephone network, a known wired or wireless television network, and the like.

The service server 100 may provide a video service to the terminal device 1 through a network. The service server 100 may store a plurality of video contents that can be provided to the terminal device 1, and provide the video to the terminal device 1 at the request of the terminal device 1. For example, the service server 100 may provide to stream content such as a video in real time or to download the corresponding content.

When providing the video service, the service server 100 may further include meta information about the video. That is, by setting meta information about the video itself, it is possible to provide the user with additional information such as a character, a plot, and a genre of the video. By using this, it is also possible to provide a user with a video search or recommendation service.

Here, the service server 100 according to an exemplary embodiment of the present invention may set meta information about contents included in the video, in addition to setting meta information about the video itself. That is, the service server 100 may divide the video into semantic based unit sections and set keywords for each unit section, so that the user may search only the desired section among the entire videos. In addition, it is also possible to provide a user with a summary video in which the entire video is shortened by collecting unit sections having the same keyword.

2 is a block diagram illustrating a service server according to an exemplary embodiment of the present invention.

Referring to FIG. 2, the service server 100 according to an exemplary embodiment of the present invention may include a unit section separator 110, a script string generator 120, a subtitle string generator 130, and a keyword generator 140. The searcher 150 may include a summary video generator 160.

Hereinafter, the service server 100 according to an embodiment of the present invention will be described with reference to FIG. 2.

The unit section separator 110 may divide the video into a plurality of unit sections. That is, the unit section separator 110 may load a target video, and may divide the video into target units based on a change in the characteristics of the voice included in the loaded video. Here, the change in the characteristic of the voice may be a change in volume or sound quality, and in some embodiments, may include a change in height and tone of the sound.

In detail, the unit section separator 110 may track the volume in the video in order to confirm the change in the characteristic of the voice. For example, the volume may be maintained within a certain range for a certain period in a video, and then suddenly out of a certain range, the volume may be rapidly increased or decreased. In this case, the unit section separating unit 110 may track a volume in the video and detect a point in the video where a change in volume occurs. That is, the unit section separator 110 may detect a sudden rising point or a falling point of the volume by using a change amount of the volume.

Here, the change amount of the volume may be calculated based on the average value of the volume during a certain period in the video, or the maximum or minimum value of the volume appearing in the corresponding section. That is, the unit section separation unit 110 may calculate how much the measured volume is compared with a reference such as an average value, and the rising point and the decreasing point are the points where the change in volume is increased above a certain threshold. Can be set with In this case, a threshold for setting the rising point and the falling point may be set differently, and the threshold may be set differently for each video.

The unit section separating unit 110 may divide the video into a plurality of unit sections based on the corresponding rising point or the falling point, and through this, detects a home run scene by using the batter's home run cry in baseball, or news Can detect where the anchor is talking and taking a break to move on to the next news.

In addition, according to an exemplary embodiment, the unit section separator 110 may grasp the sound quality in the video in order to confirm the characteristic change of the voice included in the video. For example, the unit section separator 110 may detect a portion in which the sound quality of the video is clean and suddenly contains a lot of noise, and may separate the unit section based on the detected portion. That is, the unit section separation unit 110 may detect a change in sound quality that occurs when an anchor speaks to the announcer in the field while talking in the news, and then separate the video based on this.

In addition, in the case where a plurality of speakers exist in the video, the unit section separator 110 may classify each speaker by using a tone and then separate the unit sections for each speaker. In addition, the unit section separator 110 may detect a characteristic change of the voice in various ways and separate the unit section accordingly.

On the other hand, in the case of a news video, the anchor can read the script (script) at a constant speed, and when one paragraph ends, you can hang up for a while and continue reading the next paragraph. That is, each paragraph read by the speaker in the video can be distinguished based on the amount of change in the speaker's volume. Since the content of the same topic is generally included in the same paragraph, dividing the video based on this can separate the video on the basis of meaning. Also, in the case of a moving picture or the like without a script, it is advantageous to classify moving pictures based on the change amount of the speaker's volume in order to maintain the context in which the speaker speaks. Therefore, the unit section separator 110 may divide the video into a plurality of unit sections based on the volume change of the voice included in the video.

For example, as shown in FIG. 4, if the volume change amount of the anchor in the news video V is used, the entire video can be divided into a section A where the anchor utters and a stop section B where the utterance stopped. have. Here, since the video is separated based on the change amount of the volume of the anchor, it can be seen that a plurality of screen changes may occur within one unit section.

On the other hand, since the section (A) ignited by the anchor corresponds to each unit section, each unit section can be separated by setting the stop section (B) where the utterance is stopped as a cutting point. Here, the stop section B may be set to a section in which the volume is reduced below the set value, and the volume reduced below the set value is maintained for more than the reference time. The length of the stop section B may be set differently for each video.

Therefore, the unit section separator 110 may determine the stop section B included in the video by using the characteristic change of the voice, and may divide the unit section into a plurality of unit sections.

The script string generator 120 may generate a script string corresponding to the voice by recognizing the voice included in the unit section. After dividing the video into a plurality of unit sections, it is necessary to recognize the content included in each unit section. To this end, the script string generation unit 120 may recognize the voice spoken by the speaker, convert it into a character, and combine the converted characters to generate a script string.

According to an embodiment, a separate voice recognition device may be provided in the service server 100, and the script string generation unit 120 may convert the voice into text using the voice recognition device. For example, the voice included in the unit section may be expressed as a voice pattern which is an electrical signal, and a standard voice pattern corresponding to each character may be stored in the voice model database. In this case, the voice recognition apparatus may compare the input voice patterns with standard voice patterns stored in the voice model database, and extract a standard voice pattern corresponding to each voice pattern. Thereafter, the extracted standard speech pattern may be converted into a corresponding character, and a script string may be generated by combining the converted characters. That is, as shown in FIG. 5, the script string generation unit 120 may recognize the voice spoken by the speaker in the video and generate the script string S1.

However, the method of converting the voice to the text by the script string generating unit 120 is not limited thereto, and the script string generating unit 120 may convert the voice included in the video into the text in various ways.

The caption string generation unit 130 may recognize a caption image included in a unit section and generate a caption string corresponding to the caption image. The video may include a caption image to emphasize contents spoken by the speaker or contents to be delivered in the video. For example, as shown in Fig. 5, even in the case of a news video, a caption image C may be included to summarize and convey main contents of the news.

As described above, since the contents of the video are summarized and displayed on the caption image, it is necessary to recognize the characters included in the caption image in order to confirm the contents of each unit section. However, since the caption image is recognized as a shape instead of a character, it is necessary to apply a character recognition algorithm or the like to recognize the character included in the caption image.

According to an exemplary embodiment, a separate character recognition device may be provided in the service server 100, and the caption string generation unit 130 may convert the caption image into text using the character recognition device. For example, by scanning a caption image included in a unit section, the distribution of pixel values for the caption image can be expressed as a shape pattern, which is an electrical signal, and a standard shape pattern corresponding to each character is stored in the character model database. There may be. In this case, the character recognition apparatus may compare the input shape pattern with a standard shape pattern stored in the character model database, and extract a standard shape pattern corresponding to each shape pattern. Subsequently, subtitle strings may be generated by converting each character into a character corresponding to the extracted standard shape pattern. That is, as shown in FIG. 5, the shape included in the caption image C in the video frame f may be converted into a character and extracted as a caption string S2.

On the other hand, in order for the caption string generation unit 130 to extract the caption string from the caption image, it is necessary to determine whether the caption image exists within the unit section and the position of the caption image in the video frame. That is, the character recognition may be performed only on the video frame including the caption image, and the character recognition may be performed only on the region in which the caption image is located in the video frame, thereby enabling more efficient character recognition. In addition, this may prevent a problem such as converting a character other than the caption image included in the video frame. Therefore, before generating the caption string, the caption string generation unit 130 may first detect a video frame including the caption image in the unit section and specify the position of the caption image included in the video frame.

In detail, the subtitle string generation unit 130 may set a plurality of landmarks in each video frame included in the unit section. That is, as shown in FIG. 6, the landmarks L may be uniformly positioned in the video frame, and color or luminance may be measured at each landmark L. FIG. In detail, the color, luminance, and the like of each pixel may be input from the pixel corresponding to the position of the landmark L. FIG.

Subsequently, when the color or the luminance measured in the landmark corresponds to the reference color or the reference luminance corresponding to the caption image, it may be determined that the caption image is located in the corresponding video frame. As shown in FIG. 6, the caption image C may be displayed to cover the original image D, and the caption image C may be set to have a reference color and a reference luminance. Here, the reference color and the reference luminance of the caption image (C) are set to have a characteristic color or luminance different from that of the original image (D), so that the caption character string generation unit 130 uses the caption or color to display the caption. It is possible to distinguish images.

In addition, the caption string generation unit 130 may extract landmarks in which a reference color or reference luminance corresponding to the caption image is measured, from among a plurality of landmarks uniformly distributed on the video frame. The position or size of the caption image can be specified. That is, the set coordinates in the video frame of each landmark may be set in advance, and the caption character string generation unit 130 uses the set coordinates of the landmarks for which the caption image is detected, and the position and size of the caption image. Can be specified. In this case, the subtitle string generation unit 130 may control to perform character recognition only within the specified subtitle image area. That is, since the area for performing the character recognition can be specified among the entire video frames, more efficient character recognition is possible.

Meanwhile, the subtitle string generation unit 130 may receive feature information such as a reference color, a reference luminance, a position or a size in a video frame, etc. of a subtitle image used in each video from a video producer, and may use it when extracting a subtitle image. have. For example, when receiving feature information on the position and size of the caption image, the landmark can be set within the area where the caption image is to be located, without setting the landmark uniformly throughout the video frame.

The keyword generator 140 may generate a keyword corresponding to a unit section by applying natural language processing to the script string and the caption string. That is, after checking the contents of the unit section, the user may automatically set a semantic based keyword for each unit section instead of setting keywords or annotations. Here, various methods may be applied to natural language processing applied to script strings and subtitle strings, and machine learning such as word2vec and late diallet allocation (LDA) may be applied according to embodiments.

According to an embodiment, the keyword generator 140 may implement a word embedding word2vec model using word2vec, and set words extracted from a caption string or a script string as an input word for the word2vec model. Associated words corresponding to the input word may be extracted. Thereafter, the extracted related words may be set as keywords for a corresponding unit section.

For example, when the video provided by the service server 100 is a news video, the word2vec model may be implemented by word embedding news articles for the last five years using word2vec. In the case of Word2vec, each word is embedded in a vector space to express a word as a vector. The words related to each other are arranged adjacent to each other in the space. That is, the higher the frequency of each word appearing adjacent to each other in the plurality of samples trained by the word2vec model, the more closely it may be displayed in the vector space. For example, if "nuclear" is frequently mentioned in relation to "North Korea" in an existing news article used in the sample, the vectors corresponding to "North Korea" and "Nuclear" may be embedded adjacent to each other, and they are associated with each other. Can be determined to be present.

However, since the script string includes a plurality of words, the keywords may be too large to set all of the related words extracted corresponding to the respective words included in the script string as keywords. In order to prevent this, the keyword generation unit 140 may set only a related word having a high similarity as a keyword by comparing the related word and the input word.

Specifically, the keyword generation unit 140 calculates the similarity between the input word vector corresponding to the input word input to the word2vec model and the related word vector corresponding to the related word, and extracts only the related word having high similarity as a keyword. Can be set.

Through word embedding, each word can be distributed in a vectorized space, and words set as similar or related to each other in the learned sample are located at adjacent positions in the vector space. Therefore, by calculating the similarity between the input word vector and the related word vectors, it is possible to grasp the relationship between the input word and the related words. Here, the similarity between the vectors can be calculated using cosine similarity, but is not limited thereto, and any similarity can be applied as long as the similarity between the vectors can be calculated.

The keyword generation unit 140 may extract an associated word vector having a similarity degree or more with a limit value greater than or equal to the input vector, and may set an associated word corresponding to the extracted associated word vector as a keyword. That is, only related words corresponding to the related word vector having a similarity or more than a limit value may be set as keywords. In addition, according to an exemplary embodiment, a predetermined number of related word vectors may be extracted according to a high similarity with the input vector, and the related words corresponding to the extracted predetermined number of related word vectors may be set as keywords. It is also possible. For example, 10 related word vectors with the highest similarity may be extracted, and the extracted 10 related words may be set as keywords.

In addition, an embodiment in which the keyword generator 140 sets keywords by using real-time search word information is also possible. The real-time search term information may be information about search terms that have increased in search volume in real time among search terms used in a search service provided by a portal site. Since each of the search terms included in the real-time search term information is related to a topic that is currently an issue, the keyword generator 140 may preferentially set a word related to the real-time search term as a keyword. The real-time search term information may be received by the service server 100 from the outside and provided to the keyword generator 140.

In detail, the keyword generator 140 may extract an associated word corresponding to a search word included in the real-time search word information from the related words extracted from the word2vec model, and may add a weight to the extracted related word when calculating the similarity. . That is, even when the similarity is relatively low, the related word corresponding to the real-time search word information may be set as a keyword by weight. In this case, the weights provided to the related words corresponding to the search terms may be differentially assigned according to the real-time search ranking of the search terms. For example, weights may be set differently for search words corresponding to the first place and the fifth place.

When real-time search term information is used in the keyword setting, the keyword generation unit 140 may set different keywords each time for each unit section. That is, the keyword may be set to reflect the interest or demand of the user, and thus, the user may easily search for the unit section related to the content that is the issue.

In some embodiments, the keyword generation unit 140 may set a keyword by using Latent Dirichlet Allocation (LDA). That is, a main word corresponding to a unit section may be extracted by applying a script string and a subtitle string to the machine learning model learned by LDA, and then the extracted main word may be set as a keyword of the unit section.

LDA is a topic model and corresponds to an unsupervised learning algorithm that can classify which topics exist in each document by using a plurality of document sets. When modeling using LDA, you can get words that correspond to specific topics and the topics that are contained in a particular document.

For example, when the video provided by the service server 100 is a news video, a machine learning model may be implemented by learning news articles for the last five years using LDA. In this case, a main word representing the topics included in each article and a set of words corresponding to each topic word may be extracted. For example, an article on the denuclearization of the Korean Peninsula may be classified as containing the subjects of "North Korea", "Politics", and "Denuclearization", and "North and South", "North Korean Nuclear", " Words such as "summit meeting" can be set to be included in the subject. Therefore, if a script string and a subtitle string extracted from a unit section of a news video are inputted into the machine learning model, the words included in the input script string and the subtitle string correspond to the subject words. It can be seen that the corresponding contents of the news video are included in the news video. Thereafter, the keyword generator 140 may set the main word extracted through the machine learning model as a keyword for the corresponding unit section.

In addition, according to an embodiment, the keyword generator 140 may generate keywords for the entire video. In detail, a natural language processing may be applied to a keyword set in each unit section included in the video to generate a keyword corresponding to the video. The natural language processing technique may include word learning, machine learning such as Latent Dirichlet Allocation (LDA), and the like. That is, it is advantageous for the user to set a keyword for the content of the entire video, and the keyword generator 140 may also generate a keyword for the video. In this case, in order to reflect the contents of the video, keywords of the corresponding video may be generated by using keywords for each unit section.

The searcher 150 may search for a unit section corresponding to the keyword input from the user, and provide the searched unit section to the user. Since a keyword is set in each unit section, the search unit 150 may search for a unit section including specific content and provide the same to the user. In addition, since the search unit 150 may search for each unit section separated from the video, the search unit 150 may provide only a unit section desired by the user. That is, according to the search unit 150, it is possible to greatly improve the user convenience when providing a video service.

The summary video generator 160 may extract a unit section corresponding to the reference keyword for the same video, and combine the extracted unit sections to generate a summary video of the corresponding video. Here, the reference keyword may be preset by an administrator or the like, or may be input by a user.

For example, in the case of a soccer relay video, if the reference keyword is set to "goal", "scoring", etc., only a shot scene from the unit section can be extracted to generate a summary of the scene shot, and the reference keyword is the name of a specific player. If set to, the player can extract only the unit section where the player touches the ball and generate a highlight summary video for the player. In addition, in the case of a news video, it is also possible to generate a summary video of the economic field by setting the reference keyword as "economy", or to generate a single summary video by collecting news on a specific issue such as "virtual currency". In other words, a summary video can be easily generated and provided to the user without the need for a separate editing operation for the video.

On the other hand, the service server 100 according to an embodiment of the present invention, as shown in Figure 3, may include a physical configuration such as the processor 10, the memory 40, the memory 40 Within may be included one or more modules configured to be executed by the processor 10. Specifically, the one or more modules may include a unit section separation module, a script string generation module, a subtitle string generation module, a keyword generation module, a search module, and a summary video generation module.

The processor 10 may execute various software programs and an instruction set stored in the memory 40 to perform various functions and to process data. The peripheral interface unit 30 may connect an input / output peripheral device of the computer device to the processor 10 and the memory 40, and the memory controller 20 may include the memory 10 of the processor 10 or a component of the computer device. In the case of accessing to, the function of controlling the memory access may be performed. According to an embodiment, the processor 10, the memory controller 20, and the peripheral interface unit 30 may be implemented on a single chip or may be implemented as separate chips.

The memory 40 may include fast random access memory, one or more magnetic disk storage devices, nonvolatile memory such as a flash memory device, and the like. In addition, the memory 40 may further include a storage device located away from the processor 10 or a network attached storage device accessed through a communication network such as the Internet.

On the other hand, as shown in Figure 3, the service server 100 according to an embodiment of the present invention, including the operating system in the memory 40, a unit section separation module, a script string generation module, a subtitle corresponding to an application program It may include a string generation module, a keyword generation module, a search module and a summary video generation module. Here, each module may be stored in the memory 40 as a set of instructions for performing the above-described function.

Therefore, the service server 100 according to an embodiment of the present invention, the processor 10 may access the memory 40 to execute instructions corresponding to each module. However, the unit section separation module, script string generation module, subtitle string generation module, keyword generation module, search module, and summary video generation module may include the above-described unit section separation unit, script string generation unit, subtitle string generation unit, and keyword generation unit. , The searcher and the summary video generator, respectively, will not be described here.

7 is a flowchart illustrating a video service providing method according to an embodiment of the present invention.

7, the video service providing method according to an embodiment of the present invention, the unit section separation step (S10), script string generation step (S20), subtitle string generation step (S30), keyword generation step (S40) It may include a search step (S50) and a summary video generation step (S60). Here, the video service providing method according to an embodiment of the present invention may be executed by a service server.

Hereinafter, a video service providing method according to an exemplary embodiment of the present invention will be described with reference to FIG. 7.

In the unit section separating step (S10), the unit section may be divided into a plurality of unit sections based on the change in the characteristic of the voice included in the video. Here, the change in the characteristic of the voice may include a change in volume or sound quality. In detail, a stop section in which the utterance of the speaker in the video is stopped may be extracted by using a change in the characteristic of the voice, and the video may be separated by setting the stop section as a cutting point. For example, the stop section may be set to a section in which the volume is reduced below the set value, and the volume reduced below the set value is maintained for more than the reference time. That is, in consideration of the context, the video may be set as a section until the speaker in the video stops talking. For this purpose, the volume change may be used when separating the unit section.

In the script string generation step (S20), the voice included in the unit section may be recognized to generate a script string corresponding to the voice. After dividing the video into a plurality of unit sections, it is necessary to recognize the content included in each unit section. To this end, the speaker may recognize the spoken voice, convert it into a character, and combine the converted characters into a script string.

In some embodiments, a voice recognition device may be provided, and the voice may be converted into a text using the voice recognition device. For example, the voice included in the unit section may be expressed as a voice pattern which is an electrical signal, and a standard voice pattern corresponding to each character may be stored in the voice model database. In this case, the voice recognition apparatus may compare the input voice patterns with standard voice patterns stored in the voice model database, and extract a standard voice pattern corresponding to each voice pattern. Thereafter, the extracted standard speech pattern may be converted into a corresponding character, and a script string may be generated by combining the converted characters.

In the subtitle string generation step (S30), a subtitle string corresponding to the subtitle image may be generated by recognizing a subtitle image included in a unit section. Since the caption image may be displayed by summarizing the contents of the video, it is necessary to recognize a character included in the caption image. However, since the caption image is recognized as a shape instead of a character, it is necessary to apply a character recognition algorithm or the like to recognize the character included in the caption image. Here, the subtitle string generation step S30 may be executed simultaneously with the script string generation step S20, but is not limited thereto.

According to an embodiment, a separate text recognition device may be provided, and the caption image may be converted into text using the text recognition device. For example, by scanning a caption image included in a unit section, the distribution of pixel values for the caption image can be expressed as a shape pattern, which is an electrical signal, and a standard shape pattern corresponding to each character is stored in the character model database. There may be. In this case, the character recognition apparatus may compare the input shape pattern with a standard shape pattern stored in the character model database, and extract a standard shape pattern corresponding to each shape pattern. Subsequently, subtitle strings may be generated by converting each character into a character corresponding to the extracted standard shape pattern.

On the other hand, in order to extract the caption character string from the caption image, it is necessary to determine whether the caption image exists within the unit section and the position of the caption image in the video frame. That is, before generating the caption string, the video frame including the caption image in the unit section may be first detected, and the position of the caption image included in the video frame may be specified. In detail, in the subtitle character string generation step S30, a plurality of landmarks may be set in a video frame included in a unit section, and the caption image may be detected by measuring color or luminance in the landmark. In addition, the position of the caption image may be specified by uniformly distributing the landmarks on the video frame and extracting landmarks in which the reference color or the reference luminance corresponding to the caption image is measured.

In the keyword generation step S40, natural language processing may be applied to the script string and the subtitle string to generate a keyword corresponding to the unit section. That is, after checking the contents of the unit section, the user may automatically set a semantic based keyword for each unit section instead of setting keywords or annotations. Here, various methods may be applied to natural language processing applied to script strings and subtitle strings, and machine learning such as word2vec and late diallet allocation (LDA) may be applied according to embodiments.

According to an embodiment, in the keyword generation step (S40), a word embedding word2vec model may be implemented using word2vec, and words extracted from a caption string or a script string are set as input words for the word2vec model. Associated words corresponding to the input word may be extracted. Thereafter, the extracted related words may be set as keywords for a corresponding unit section.

Here, the keyword generation step (S40) may be limited to set only the related words with a high similarity by comparing the related words and the input words. In detail, the similarity between the input word vector corresponding to the input word input to the word2vec model and the related word vector corresponding to the related word may be calculated to extract only the related word having a high similarity and set it as a keyword.

Each word may be distributed in a vectorized space through word embedding, and words set as similar or related to each other in the learned sample may be located at adjacent positions on the vector space. Therefore, by calculating the similarity between the input word vector and the related word vectors, it is possible to grasp the relationship between the input word and the related words. Here, the similarity between the vectors may be calculated using cosine similarity.

In detail, an associated word vector having a similarity degree to or more than a limit value with an input vector may be extracted, and an associated word corresponding to the extracted associated word vector may be set as a keyword. That is, only related words corresponding to the related word vector having a similarity or more than a limit value may be set as keywords. In addition, according to an exemplary embodiment, a predetermined number of related word vectors may be extracted according to a high similarity with the input vector, and the related words corresponding to the extracted predetermined number of related word vectors may be set as keywords. It is also possible. For example, 10 related word vectors with the highest similarity may be extracted, and the extracted 10 related words may be set as keywords.

In addition, in the keyword generation step S40, an embodiment of setting a keyword using real-time search word information is also possible. For example, among the related words extracted from the word2vec model, the related words corresponding to the search words included in the real-time search word information may be extracted, and weights may be added to the extracted related words when calculating the similarity. That is, even when the similarity is relatively low, the related word corresponding to the real-time search word information may be set as a keyword by weight. In this case, the weights provided to the related words corresponding to the search terms may be differentially assigned according to the real-time search ranking of the search terms.

In some exemplary embodiments, a keyword may be set using late diallet allocation (LDA) in the keyword generation step (S40). That is, a main word corresponding to a unit section may be extracted by applying a script string and a subtitle string to the machine learning model learned by LDA, and then the extracted main word may be set as a keyword of the unit section. However, since the contents of setting the keywords using the machine learning model learned using the LDA have been described above, detailed descriptions thereof will be omitted.

In addition, according to an embodiment, it is also possible to generate keywords for the entire video in the keyword generation step (S40). That is, natural language processing may be applied to a keyword set in each unit section included in the video to generate a keyword corresponding to the video. The natural language processing technique may include word learning, machine learning such as Latent Dirichlet Allocation (LDA), and the like.

In the search step (S50), it is possible to search for a unit section corresponding to the keyword input from the user, and provide the searched unit section to the user. Since a keyword is set in each unit section, a unit section including a specific content can be searched and provided to the user. In addition, since it is possible to search for each unit section separated from the video, it is possible to provide only the unit section desired by the user. That is, it is possible to greatly improve user convenience when providing a video service.

In the summary video generation step (S60), a unit section corresponding to the reference keyword is extracted for the same video, and the extracted unit sections may be combined to generate a summary video of the corresponding video. Here, the reference keyword may be preset by an administrator or the like, or may be input by a user. In other words, a summary video can be easily generated and provided to the user without the need for a separate editing operation for the video.

The present invention described above can be embodied as computer readable codes on a medium on which a program is recorded. The computer readable medium may be to continuously store a computer executable program or to temporarily store a program for execution or download. In addition, the medium may be a variety of recording means or storage means in the form of a single or several hardware combined, not limited to a medium directly connected to any computer system, it may be distributed on the network. Examples of media include magnetic media such as hard disks, floppy disks and magnetic tape, optical recording media such as CD-ROMs and DVDs, magneto-optical media such as floptical disks, And ROM, RAM, flash memory, and the like, configured to store program instructions. In addition, examples of another medium may include a recording medium or a storage medium managed by an app store that distributes an application, a site that supplies or distributes various software, a server, or the like. Accordingly, the above detailed description should not be construed as limiting in all aspects and should be considered as illustrative. The scope of the present invention should be determined by reasonable interpretation of the appended claims, and all changes within the equivalent scope of the present invention are included in the scope of the present invention.

The present invention is not limited by the above-described embodiment and the accompanying drawings. It will be apparent to those skilled in the art that the present invention may be substituted, modified, and changed in accordance with the present invention without departing from the spirit of the present invention.

1: terminal device 100: service server
10: processor 20: memory control unit
30: peripheral interface 40: memory
110: unit section separator 120: script string generation unit
130: subtitle string generator 140: keyword generator
150: search unit 160: summary video generation unit

Claims (18)

In the video service providing method for providing a video to the terminal device, the service server,
A unit section separation step of separating the video into a plurality of unit sections based on a change in a characteristic of a voice included in the video;
A script string generation step of generating a script string corresponding to the speech by recognizing the speech included in the unit section;
A caption string generation step of recognizing a caption image included in the unit section and generating a caption string corresponding to the caption image; And
And a keyword generation step of generating a semantic-based keyword corresponding to each unit section by applying natural language processing to the script string and the subtitle string.
The unit section separation step is
Video service, characterized in that for extracting a stop section in which the utterance of the speaker in the video is stopped using the change in the characteristics of the voice, and separating the video by setting the stop section as a cutting point (cutting point) How to Provide.
delete The method of claim 1, wherein the separating unit section
If the volume of the voice is less than the set value, and the volume of the voice less than the set value is maintained for more than a reference time, the video service providing method characterized in that it is determined as the stop section.
The method of claim 1, wherein the script string generation step
Using a voice recognition device, converting the voice pattern extracted from the voice to a corresponding character, and combining the converted characters to generate the script string, characterized in that for generating a script string.
The method of claim 1, wherein the subtitle string generation step
Using a character recognition device, converting the shape pattern extracted from the caption image to a corresponding character, and combining the converted characters to generate the subtitle character string.
The method of claim 1, wherein the subtitle string generation step
And a plurality of landmarks are set in the moving picture frame included in the unit section, and the caption image is detected using the color or the luminance measured by the landmark.
The method of claim 6, wherein the subtitle string generation step
Among the plurality of landmarks uniformly distributed on the video frame, the landmarks in which the reference color or the reference luminance corresponding to the caption image are measured are extracted, and the position of the caption image is specified using the extracted landmarks. Video service providing method, characterized in that.
The method of claim 1, wherein the keyword generation step
A video service, comprising: extracting related words by inputting input words extracted from the subtitle string or script string into a word embedding word2vec model using word2vec, and setting the related word as the keyword How to Provide.
In the video service providing method for providing a video to the terminal device, the service server,
A unit section separation step of separating the video into a plurality of unit sections based on a change in a characteristic of a voice included in the video;
A script string generation step of generating a script string corresponding to the speech by recognizing the speech included in the unit section;
A caption string generation step of recognizing a caption image included in the unit section and generating a caption string corresponding to the caption image; And
And a keyword generation step of generating a keyword corresponding to the unit section by applying natural language processing to the script string and the subtitle string.
The keyword generation step
input corresponding words extracted from the subtitle string or script string into a word embedding word2vec model using word2vec to extract corresponding association words, and set the association word as the keyword,
The keyword generation step
Calculating a similarity between an input word vector corresponding to the input word input to the word2vec model and an associated word vector corresponding to the related word;
Extracting an associative word vector having a similarity or more than a limit value or a predetermined number of associative word vectors selected according to the order of high similarity; And
And setting a related word corresponding to the extracted related word vector as the keyword.
The method of claim 9, wherein calculating the similarity level
And extracting a related word corresponding to a search word included in real-time search word information, and adding the weighted value to the extracted related word when calculating the similarity.
The method of claim 10, wherein generating the keyword
According to the real-time search ranking of the search terms, the video service providing method characterized in that the weight is provided to the associated word corresponding to the search by differential.
The method of claim 1, wherein generating the keyword
Providing a video service comprising extracting a main word corresponding to the unit section by applying the script string and a subtitle string to a machine learning model trained by using LDA (Latent Dirichlet Allocation), and setting the main word as the keyword. Way.
The method of claim 1, wherein generating the keyword
And a keyword corresponding to the video by applying natural language processing to the keyword corresponding to the unit section.
The method of claim 1,
And searching for a unit section corresponding to a keyword input from a user, and providing the searched unit section to a user.
The method of claim 1,
And a summary video generating step of extracting a unit section corresponding to a reference keyword and generating a summary video by combining the extracted unit sections with respect to the same video.
16. A computer program stored in a medium in combination with hardware to execute the video service providing method of any one of claims 1 and 3.
A unit section separator configured to separate the video into a plurality of unit sections based on a change in a characteristic of a voice included in the video;
A script string generation unit for recognizing a voice included in the unit section and generating a script string corresponding to the voice;
A caption string generator for recognizing a caption image included in the unit section and generating a caption string corresponding to the caption image; And
By applying a natural language processing (Natural Language Processing) to the script string and subtitle string, comprising a keyword generation unit for generating a semantic-based keyword corresponding to the unit section for each unit section,
The unit section separation unit
A service server, characterized in that for extracting a stop section in which the utterance of the speaker in the video is stopped using the change in the characteristics of the voice, and separating the video by setting the stop section as a cutting point (cutting point) .
A processor; And
Including a memory coupled to the processor,
The memory includes one or more modules configured to be executed by the processor,
The one or more modules,
The video is divided into a plurality of unit sections based on the change in the characteristics of the voice included in the video.
Recognizing a voice included in the unit section, generates a script string corresponding to the voice,
Recognizing a caption image included in the unit section, generates a caption string corresponding to the caption image,
By applying Natural Language Processing to the script string and the subtitle string, a semantic based keyword corresponding to the unit section is generated for each unit section,
A service server including a command for extracting a stop section in which a utterance of a speaker is stopped in the video using the change in the characteristic of the voice, and setting the stop section as a cutting point to separate the video. .

Images