JP6824332B2 - Video service provision method and service server using this - Google Patents

Description

The present application relates to a video service providing method and a service server using the same, and relates to a video service providing method capable of separating moving images into unit intervals based on meaning and automatically generating keywords for each unit interval, and a service server using the same.

Recently, with the development of Internet technology, video services that provide videos via the Internet have been widely used. When a user wants to watch a video via the Internet, he / she needs to search for the desired video from a large number of videos provided on the Internet, and various video search methods for effective video search, etc. Has been presented.

However, recently, there are increasing cases where users are interested in a part of a video rather than the whole video and try to watch only that part. For example, a user who wants to watch a soccer broadcast may want to watch only a scene in which a specific player scores a goal, rather than watching the entire soccer broadcast program. However, in the general video search method, since the entire soccer broadcast is targeted for the search, it is difficult to search for a part of the scene of the video desired by the user.

Korean Registered Patent No. 10-0721409

The present application provides a video service providing method capable of separating moving images into unit intervals based on meaning and automatically generating keywords for each unit interval, and a service server using the same.

The present application provides a video service providing method capable of separating a moving image into a plurality of unit intervals based on a change in audio characteristics in the moving image, and a service server using the same.

The present application provides a video service providing method capable of automatically generating a keyword according to the content of a unit interval by applying voice recognition and subtitle recognition to each unit section in which a moving image is separated, and a service server using the same.

The present application provides a video service providing method capable of automatically generating keywords according to the contents of each unit interval of a moving image by applying natural language processing using machine learning, and a service server using the same.

The video service providing method according to one embodiment of the present invention relates to a video service providing method in which a service server provides a moving image to a terminal device, and the moving image is divided into a plurality of unit sections based on a change in the characteristics of audio included in the moving image. The unit section separation step to be separated, the script character string generation step of recognizing the sound included in the unit section and generating the script character string corresponding to the sound, the subtitle image included in the unit section, and the above A subtitle character string generation step for generating a subtitle character string corresponding to a subtitle image, and natural language processing (Natural Language Processing) are applied to the script character string and the subtitle character string to generate a keyword corresponding to the unit interval. Includes keyword generation steps.

The service server according to the embodiment of the present invention recognizes a unit interval separation unit that separates the moving image into a plurality of unit sections and the sound included in the unit section based on a change in the characteristics of the sound included in the moving image. , A script character string generator that generates a script character string corresponding to the sound, a subtitle character string generator that recognizes a subtitle image included in the unit interval and generates a subtitle character string corresponding to the subtitle image, and It includes a keyword generation unit that applies natural language processing to the script character string and the subtitle character string to generate a keyword corresponding to the unit interval.

A service server according to another embodiment of the present invention includes a processor and a memory coupled to the processor, the memory including one or more modules configured to be executed by the processor. , The one or more modules separate the moving image into a plurality of unit sections based on a change in the characteristics of the sound included in the moving image, recognize the sound included in the unit section, and correspond to the sound. A script character string is generated, a subtitle image included in the unit section is recognized, a subtitle character string corresponding to the subtitle image is generated, and natural language processing is applied to the script character string and the subtitle character string. Includes a command word that generates a keyword corresponding to the unit interval.

In addition, the matters described in the column of means for solving the problem do not list all the features of the present invention. The various features of the present invention and the corresponding advantages and effects can be understood in more detail with reference to the specific embodiments below.

According to the video service providing method according to the embodiment of the present invention and the service server using the same, the moving image is separated based on the change in the characteristics of the sound in the moving image, so that the moving image can be separated without damaging the context or meaning. it can.

According to the video service providing method according to the embodiment of the present invention and the service server using the same, voice recognition and subtitle recognition are applied to extract the contents included in the unit interval, and then each unit is used. Since the keyword for the section is set, the keyword can be set according to the content of the unit interval.

According to the video service providing method according to the embodiment of the present invention and the service server using the same, the user can search for a specific scene included in the video based on the content, and based on a specific subject or content. A summary video can be generated.

Further, the video service providing method according to the embodiment of the present invention and the effects that can be achieved by the service server using the same are not limited to those mentioned above, and other effects not mentioned are the techniques to which the present invention belongs according to the following description. It is clearly understandable to those who have ordinary knowledge in the field.

It is the schematic which shows the moving image service providing system by one Embodiment of this invention. It is a block diagram which shows the service server by one Embodiment of this invention. It is a block diagram which shows the service server by one Embodiment of this invention. It is the schematic which shows the separation of the unit section of a moving image by one Embodiment of this invention. It is the schematic which shows the generation of the script character string and the subtitle character string by one Embodiment of this invention. It is the schematic which shows the detection of the caption image by one Embodiment of this invention. It is a flowchart which shows the moving image service providing method by another embodiment of this invention.

Hereinafter, embodiments disclosed in the present specification will be described in detail with reference to the accompanying drawings, but the same or similar components will be given the same reference number regardless of the drawings, and duplicate description thereof will be omitted. I will do it. The suffixes "module" and "part" to the components used in the following description are given or mixed only for the ease of writing the specification, meaning that they are distinguished from each other by themselves. Or it does not have a role. That is, the term "part" as used in the present invention means a hardware component such as software, FPGA or ASIC, and the "part" plays a role. However, "department" is not limited to software or hardware. The "part" may be configured to be in an addressable storage medium or to regenerate one or more processors. Thus, as an example, a "part" is a component such as a software component, an object-oriented software component, a class component, a task component, and a process, function, attribute, procedure, subroutine, program code segment, driver, Includes firmware, microcode, circuits, data, databases, data structures, tables, arrays and variables. The functionality provided within the components and "parts" may be combined by a smaller number of components and "parts" or further separated into additional components and "parts".

In addition, in the description of the embodiments disclosed in the present specification, when it is determined that the specific description of the related publicly known technology may obscure the gist of the embodiments disclosed in the present specification. Will omit the detailed description thereof. In addition, the accompanying drawings are merely intended to facilitate the understanding of the embodiments disclosed herein, and the attached drawings limit the technical ideas disclosed herein. It must be understood as including all modifications, equivalents or alternatives contained within the ideas and technical scope of the present invention.

FIG. 1 is a schematic view showing a moving image service providing system according to an embodiment of the present invention.

Referring to FIG. 1, the moving image service providing system according to the embodiment of the present invention can include the terminal device 1 and the service server 100.

Hereinafter, the moving image service providing system according to the embodiment of the present invention will be described with reference to FIG.

The terminal device 1 can communicate with the service server 100 via the network, and can receive the video service provided by the service server 100. The terminal device 1 can include a display unit, a speaker, and the like for visually or audibly providing contents such as moving images to the user, an input unit that receives input from the user, and a memory in which at least one program is stored. And can include processors.

The terminal device 1 may be a mobile terminal such as a smartphone or tablet PC or a fixed device such as a desktop, and depending on the embodiment, a mobile phone, a smartphone (Smartphone), a laptop computer (laptop computer), or a digital broadcast. Terminals, PDA (personal digital assistants), PMP (portable multimedia player), slate PC (slate PC), tablet PC (tablet PC), ultrabook (ultrabook), wearable device (warewatch), wearable device (warewatch) ), Smart glasses, head mount display (HMD: head mounted computer), and the like can correspond to the terminal device 1.

The network connecting the terminal device 1 and the service server 100 can include a wired network and a wireless network, and specifically, a local area network (LAN: Local Area Network), a metropolitan area network (MAN:). It can include various networks such as Metropolitan Area Network), Wide Area Network (WAN: Wide Area Network) and the like. The network can also include a known World Wide Web (WWW: World Wide Web). However, the network according to the present invention is not limited to the networks listed above, and may include a known wireless data network, a known telephone network, a known wired or wireless television network, and the like.

The service server 100 can provide a moving image service to the terminal device 1 via a network. A plurality of moving image contents that can be provided to the terminal device 1 are stored in the service server 100, and the moving image can be provided to the terminal device 1 in response to a request from the terminal device 1. For example, the service server 100 can provide content such as moving images to be streamed in real time or such content to be downloaded.

When the service server 100 provides the moving image service, the service server 100 can further provide the meta information for the moving image. That is, it is possible to set meta information for the video itself and provide the user with additional information such as characters, stories, genres, etc. of the video, which can be used for video search and recommendation services. Can also be provided.

Here, the service server 100 according to the embodiment of the present invention can set meta information for the content included in the moving image in addition to setting meta information for the moving image itself. That is, the service server 100 can provide the service server 100 so as to search only the section desired by the user in the entire moving image by setting a keyword for each unit interval after separating the moving image into unit sections based on the meaning. it can. It is also possible to provide the user with a summary video summarizing the entire video by collecting unit intervals having the same keyword.

FIG. 2 is a block diagram showing a service server according to an embodiment of the present invention.

Referring to FIG. 2, the service server 100 according to the embodiment of the present invention includes a unit interval separation unit 110, a script character string generation unit 120, a subtitle character string generation unit 130, a keyword generation unit 140, a search unit 150, and a summary moving image. The generation unit 160 can be included.

Hereinafter, the service server 100 according to the embodiment of the present invention will be described with reference to FIG.

The unit interval separation unit 110 can separate the moving image into a plurality of unit intervals. That is, the unit section separation unit 110 can load the target moving image, and can separate the moving image into a plurality of unit sections based on the change in the characteristics of the sound included in the loaded moving image. Here, the change in the characteristics of the voice may be a change in the volume or the sound quality, and depending on the embodiment, the change in the pitch, the tone color, and the like may be included.

Specifically, the unit interval separation unit 110 can track the volume in the moving image in order to confirm the change in the characteristics of the sound. For example, the volume is maintained within a specific range during a certain section of the moving image, and suddenly goes out of the specific range and suddenly increases or decreases. At this time, the unit interval separation unit 110 can track the volume in the moving image and detect the point in the moving image in which the change in volume has occurred. That is, the unit interval separation unit 110 can detect a sharp rise point or a fall point of the volume by using the amount of change in the volume.

Here, the amount of change in volume can be calculated based on the average value of the volume during a certain section in the moving image and the maximum value or the minimum value of the volume appearing in the section. That is, the unit interval separation unit 110 can calculate how much the measured volume has changed by comparing it with a reference such as an average value, and a point where the amount of change in volume has increased above a certain threshold (threshold). Ascending points and decreasing points can be set as descending points. At this time, the threshold values for setting the ascending point and the descending point may be set differently from each other, and the threshold values may be set differently from each other for each moving image.

The unit interval separation unit 110 can separate a moving image into a plurality of unit sections based on a volume rising point or a volume falling point, thereby making a home run scene using a cry when a batter hits a home run in baseball. It can detect, or detect the part of the news that the anchor is talking about and pauses to move on to the next news.

Further, depending on the embodiment, the unit interval separation unit 110 can grasp the sound quality in the moving image in order to confirm the change in the characteristics of the sound included in the moving image. For example, the unit interval separation unit 110 can detect a portion of a moving image in which noise suddenly increases from a state in which the sound quality is good, and can separate the detected portion into unit intervals based on the detected portion. That is, the unit interval separation unit 110 can also separate the moving image based on the change in sound quality that occurs when the microphone is passed to the announcer at the site while the anchor is talking in the news. Further, when a plurality of speakers are present in the moving image, the unit interval separating unit 110 can distinguish each speaker by using a timbre and then separate the speakers into unit intervals for each speaker. In addition to this, the unit section separation unit 110 can detect a change in the characteristics of the voice by various methods and can separate the unit sections accordingly.

On the other hand, in the case of a news movie, the anchor reads the manuscript (script) at a constant speed, and when one paragraph is finished, cuts it for a while and continues reading the next paragraph. That is, each paragraph read by the speaker in the moving image can be distinguished based on the amount of change in the volume of the speaker. Since the same subject content is generally included in the same paragraph, if the moving images are classified based on the contents, the moving images can be separated based on the meaning. Further, even in the case of a moving image without a defined manuscript, it is advantageous to classify the moving image based on the amount of change in the volume of the speaker in order to maintain the context in which the speaker speaks in the moving image. Therefore, in the unit interval separation unit 110, the moving image can be separated into a plurality of unit sections based on the amount of change in the volume of the sound included in the moving image.

For example, as shown in FIG. 4, by using the volume change amount of the anchor in the news video V, the entire video can be divided into a section A in which the anchor speaks and a stop section B in which the utterance is interrupted. Here, since the moving image is separated based on the amount of change in the volume of the anchor, it can be confirmed that a plurality of screen changes can occur in one unit interval.

On the other hand, since the section A spoken by the anchor corresponds to each unit section, the stop section B in which the utterance is interrupted can be set as an editing point (Cutting point) to separate each unit section. Here, the stop section B can be set to a section in which the volume is reduced to less than the set value and the volume reduced to less than the set value is maintained for the reference time or longer. The length of the stop section B may be set to be different for each moving image.

Therefore, the unit section separation unit 110 can determine the stop section B included in the moving image by using the change in the characteristics of the sound, and can use it to separate into a plurality of unit sections.

The script character string generation unit 120 can recognize the voice included in the unit interval and generate a script character string corresponding to the voice. After separating the video into multiple unit intervals, it is necessary to recognize the content contained in each unit interval. For this purpose, the script character string generation unit 120 can recognize the voice spoken by the speaker, convert it into characters, and combine the converted characters to generate a script character string.

Depending on the embodiment, another voice recognition device may be provided in the service server 100, and the script character string generation unit 120 can convert the voice into characters by using the voice recognition device. For example, the voice included in the unit interval can be represented by a voice pattern that is an electrical signal, and a standard voice pattern corresponding to each character may be stored in a voice model database or the like. In this case, the voice recognition device can compare the input voice pattern with the standard voice pattern stored in the voice model database, and can extract the standard voice pattern corresponding to each voice pattern. After that, the extracted standard voice pattern can be converted into the corresponding character, and the converted characters can be combined to generate a script character string. That is, as shown in FIG. 5, the script character string generation unit 120 can recognize the voice spoken by the speaker in the moving image and generate the script character string S1.

However, the method in which the script character string generation unit 120 converts the voice into characters is not limited to this, and the script character string generation unit 120 may convert the voice included in the moving image into characters by various other methods. it can.

The subtitle character string generation unit 130 can recognize the subtitle image included in the unit interval and generate the subtitle character string corresponding to the subtitle image. Subtitle images may be included in the video to emphasize what the speaker is saying or what the video is trying to convey. For example, as shown in FIG. 5, in the case of a news movie, a subtitle image C is included in order to summarize and convey the main contents of the news.

Since the contents of the moving image are summarized and displayed in the subtitle image in this way, it is necessary to recognize the characters included in the subtitle image in order to confirm the contents of each unit interval. However, since the subtitle image is recognized as a shape rather than a character, it is necessary to apply a character recognition algorithm or the like in order to recognize the character included in the subtitle image.

Depending on the embodiment, another character recognition device may be provided in the service server 100, and the subtitle character string generation unit 130 can convert the subtitle image into characters by using the character recognition device. For example, a subtitle image included in a unit interval can be scanned and the distribution of pixel values with respect to the subtitle image can be represented by a shape pattern that is an electrical signal, and a standard shape pattern corresponding to each character is stored in a character model database or the like. It may have been done. In this case, the character recognition device can compare the input shape pattern with the standard shape pattern stored in the character model database, and can extract the standard shape pattern corresponding to each shape pattern. After that, it is possible to generate a subtitle character string by converting each character into a character corresponding to the extracted standard shape pattern. That is, as shown in FIG. 5, the shape included in the subtitle image C in the moving image frame f can be converted into characters and extracted as the subtitle character string S2.

On the other hand, in order for the subtitle character string generation unit 130 to extract the subtitle character string from the subtitle image, it is necessary to determine the presence or absence of the subtitle image in the unit section and the position of the subtitle image in the moving image frame. That is, more efficient character recognition is executed by executing character recognition only in the video frame including the subtitle image and executing character recognition only in the area where the subtitle image is located in the video frame. Can be done. In addition, it is possible to prevent problems such as conversion of other characters that are not subtitle images included in the moving image frame. Therefore, in the subtitle character string generation unit 130, before generating the subtitle character string, first, the video frame including the subtitle image in the unit interval is detected, and the position of the subtitle image included in the video frame is specified. Can be done.

Specifically, the subtitle character string generation unit 130 can set a plurality of landmarks for each moving image frame included in the unit interval. That is, as shown in FIG. 6, the landmarks L can be set to be uniformly located in the moving image frame, and the hue or the brightness can be measured at each landmark L. Specifically, it is possible to receive input such as hue and brightness of each pixel from the pixel corresponding to the position of the landmark L.

After that, if the hue or brightness measured at the landmark corresponds to the reference hue or reference brightness corresponding to the subtitle image, it can be determined that the subtitle image is located in the moving image frame. As shown in FIG. 6, the subtitle image C may be displayed in a form that covers the original image D, and the subtitle image C can be set to have a reference hue and a reference brightness. Here, since the reference hue and the reference brightness of the subtitle image C are set to have a characteristic hue and brightness that are distinguished from the original image D, the subtitle character string generation unit 130 uses the hue and brightness. Subtitle images can be distinguished.

Further, the subtitle character string generation unit 130 can extract, and extract, the landmarks whose reference hue or reference brightness corresponding to the subtitle image is measured from among the plurality of landmarks uniformly distributed on the moving image frame. The position or size of the subtitle image can be specified by using the landmark. That is, the set coordinates in the moving image frame of each landmark may be set in advance, and the subtitle character string generation unit 130 uses the set coordinates of the landmark that detected the subtitle image to display the subtitle image. The position and size can be specified. In this case, the subtitle character string generation unit 130 can be controlled to execute character recognition only within the specified subtitle image area. That is, since the area in which the character recognition is executed can be specified in the entire moving image frame, more efficient character recognition is possible.

On the other hand, the subtitle character string generation unit 130 receives characteristic information such as the reference hue and reference brightness of the subtitle image used for each video, the position and size in the video frame from the video creator, and receives the subtitle image. It can be utilized at the time of extraction. For example, when receiving feature information regarding the position and size of a subtitle image, the landmarks are not set uniformly over the entire video frame, but are limited to the area set where the subtitle image is located. Marks can be set.

The keyword generation unit 140 can apply natural language processing (Natural Language Processing) to the script character string and the subtitle character string to generate a keyword corresponding to the unit interval. That is, after the user confirms the content of the unit interval, the keyword and the annotation can be automatically set based on the meaning for each unit interval, instead of setting the keyword and the annotation corresponding to the content. Here, various methods can be applied to the natural language processing applied to the script character string and the subtitle character string, and depending on the embodiment, machine learning such as word2vec and LDA (Latent Dirichlet Allocation) can be applied. ) Can be applied.

According to one embodiment, the keyword generation unit 140 can realize a word2vec model in which words are embedded using word2vec, and the words extracted from the subtitle character string or the script character string are input words to the word2vec model. Can be set to to extract related words that correspond to the input words. After that, the extracted related words can be set as keywords for the unit interval.

For example, when the moving image provided by the service server 100 is a news moving image, a word2vec model can be realized by embedding a word such as a news article for the last 5 years using word2vec. In the case of Word2vec, each word is embedded in a vector space to represent the word as a vector, and words related to each other are arranged adjacent to each other in the space. That is, the higher the frequency with which each word appears adjacent to each other in a plurality of samples trained by the word2vec model, the more the words can be displayed adjacent to each other in the vector space. For example, in the existing news articles used in the sample, when "Brexit" is often referred to as "UK", "Eurozone", "Leaving", etc., "Brexit" and "UK", "Euro" Vectors corresponding to "zone", "withdrawal", etc. can be embedded adjacent to each other, and it can be determined that they are related to each other.

However, since the script character string contains a plurality of words, the number of keywords may be excessively large in order to set all the related words extracted corresponding to each word included in the script character string as keywords. In order to prevent this, the keyword generation unit 140 can compare the related word and the input word and set only the related word having a high degree of similarity as the keyword.

Specifically, the keyword generation unit 140 calculates the degree of similarity between the input word vector corresponding to the input word input to the word2vec model and the related word vector corresponding to the related word, and the association having a high degree of similarity is calculated. Only words can be extracted and set as keywords.

Through word embedding, each word can be vectorized and distributed in space, and words that are set to be similar or related to each other in the learned sample are located at adjacent positions in vector space. Will be. Therefore, the similarity between the input word vector and the related word vector can be calculated to grasp the relationship between the input word and the related word. Here, the similarity between vectors can be calculated using the cosine similarity, but the similarity is not limited to this, and any similarity between vectors can be applied. You may.

The keyword generation unit 140 can extract a related word vector having a similarity with the input vector of a predetermined value or more, and can set a related word corresponding to the extracted related word vector as a keyword. That is, only related words corresponding to the related word vector having a similarity of a predetermined value or more can be set as a keyword. Further, depending on the embodiment, it is possible to extract an already set number of related word vectors according to the order of similarity with the input vector, and the association corresponding to the extracted already set number of related word vectors. You can also set a word as a keyword. For example, 10 related word vectors having the highest degree of similarity can be extracted, and the extracted 10 related words can be set as keywords.

Further, an embodiment in which the keyword generation unit 140 sets a keyword using real-time search term information is also possible. The real-time search term information may be information for a search term whose search amount has rapidly increased in real time among the search terms used in a search service provided by a portal site or the like. Since each search word included in the real-time search word information is related to the subject currently being an issue, the keyword generation unit 140 can preferentially set a word related to the real-time search word as a keyword. The real-time search term information can be received by the service server 100 from the outside and provided to the keyword generation unit 140.

Specifically, the keyword generation unit 140 extracts the related words corresponding to the search words included in the real-time search word information from the related words extracted from the word2vec model, and the similarity with respect to the extracted related words. A weighted value can be added when calculating. That is, even when the similarity is relatively low, the related words corresponding to the real-time search word information can be set as keywords by weighted values. At this time, weighted values provided to related words corresponding to the search words can be given differently from each other according to the real-time search order of the search words. For example, the weighted values can be set differently for the search word corresponding to the first place and the search word corresponding to the fifth place of the real-time search word.

When the real-time search term information is used when setting keywords, the keyword generation unit 140 can set the keywords to be set for each unit interval so as to be different each time. That is, keywords can be set to reflect the interests and demands of the user, whereby the unit interval related to the content of the issue can be provided so that the user can easily search.

On the other hand, depending on the embodiment, the keyword generation unit 140 can also set keywords by using LDA (Latent Dirichlet Allocation). That is, a script character string and a subtitle character string can be applied to a machine learning model learned by LDA to extract a theme word corresponding to a unit interval, and then the extracted theme word is set as a keyword of the unit interval. can do.

LDA is one of the topic models (topic model), and corresponds to an unsupervised learning algorithm that can classify what kind of subject exists in each document using a plurality of document sets. By modeling using LDA, words corresponding to a specific subject and subjects contained in a specific document can be obtained as a result.

For example, when the moving image provided by the service server 100 is a news moving image, a machine learning model can be realized by learning news articles and the like for the last 5 years using LDA. In this case, it is possible to extract a subject word indicating the subject contained in each article and a set of words corresponding to each subject word. For example, articles about Brexit can be categorized into those that include the subjects "UK", "Eurozone", "Hard Breakfast", and "Dill Breakfast", with the "Dill Breakfast" subject. Related words such as "dill", "proposed agreement", "veto", and "withdrawal" can be set to those included in the subject. Therefore, if the script character string and subtitle character string extracted from any one unit section of the news video are input to the machine learning model, what kind of subject word will the words included in the input script character string and subtitle character string become? It is possible to confirm whether the word is a corresponding word, and thereby it is possible to grasp what kind of subject word the content corresponds to in the news video. After that, the keyword generation unit 140 can set the theme word extracted via the machine learning model as a keyword for the unit interval.

Further, depending on the embodiment, the keyword generation unit 140 may generate a keyword for the entire moving image. Specifically, natural language processing can be applied to the keywords set for each unit interval included in the moving image to generate the keyword corresponding to the moving image. Here, machine learning such as word2vec and LDA can be applied to the natural language processing technique. That is, since it is advantageous for the user to set a keyword for the content of the entire moving image, the keyword generation unit 140 can also generate a keyword for the moving image. At this time, in order to reflect the content of the moving image, the keyword of the moving image can be generated by using the keyword for each unit interval.

The search unit 150 can search for a unit interval corresponding to a keyword input by the user and provide the searched unit interval to the user. Since a keyword is set for each unit interval, the search unit 150 can search for a unit interval including a specific content and provide it to the user. Further, since the search unit 150 can search for each unit interval separated from the moving image, it is possible to provide only the unit interval desired by the user. That is, according to the search unit 150, the user convenience at the time of providing the moving image service can be significantly improved.

The summary moving image generation unit 160 can extract unit intervals corresponding to the reference keywords for the same moving image and combine the extracted unit sections to generate a summary moving image for the moving image. Here, the reference keyword may be preset by the administrator or may be input by the user.

For example, in the case of a soccer broadcast video, if the reference keywords are set to "goal", "score", etc., it is possible to extract only the scoring scene from the unit interval and generate a summary video summarizing the goal scene. By setting the reference keyword to the name of a specific player, it is possible to extract only the unit interval in which the specific player touches the ball and generate a highlight summary video for that specific player. In the case of news videos, the reference keyword is set to "economy" to generate a summary video for the economic field, or news for a specific issue such as "virtual currency" is aggregated into one summary video. It can also be generated in. That is, it is not necessary to perform another editing work on the moving image, and the summary moving image can be easily generated and provided to the user.

On the other hand, as shown in FIG. 3, the service server 100 according to the embodiment of the present invention includes a physical configuration such as a processor 10 and a memory 40, and is executed by the processor 10 in the memory 40. One or more modules configured as such can be included. Specifically, one or more modules can include a unit interval separation module, a script character string generation module, a subtitle character string generation module, a keyword generation module, a search module, a summary moving image generation module, and the like.

The processor 10 can execute various software programs and a set of instruction words stored in the memory 40 to execute various functions and execute a function of processing data. The peripheral interface unit 30 can connect the input / output peripheral devices of the computer device to the processor 10 and the memory 40, and the memory control unit 20 can connect the input / output peripheral devices of the computer device to the memory 40 when the components of the processor 10 and the computer device access the memory 40. You can perform functions that control access. Depending on the embodiment, the processor 10, the memory control unit 20, and the peripheral interface unit 30 may be realized on a single chip or on separate chips.

The memory 40 can include high-speed random access memory, one or more magnetic disk storages, non-volatile memory such as a flash memory device, and the like. Further, the memory 40 can further include storage located away from the processor 10, network attached storage accessed via a communication network such as the Internet, and the like.

On the other hand, as shown in FIG. 3, the service server 100 according to the embodiment of the present invention has a memory 40, an operating system, a unit interval separation module corresponding to an application program, a script character string generation module, and a subtitle character string generation. It can include modules, keyword generation modules, search modules, summary video generation modules, and so on. Here, each module can be stored in the memory 40 as a set of instruction words for executing the above-mentioned functions.

Therefore, in the service server 100 according to the embodiment of the present invention, the processor 10 can access the memory 40 and execute the instruction word corresponding to each module. However, the unit interval separation module, script character string generation module, subtitle character string generation module, keyword generation module, search module, and summary video generation module are the unit interval separation unit, script character string generation unit, and subtitle character string generation unit described above. , Keyword generation unit, search unit, and summary video generation unit, respectively, so detailed description is omitted here.

FIG. 7 is a flowchart showing a method of providing a moving image service according to an embodiment of the present invention.

Referring to FIG. 7, the video service providing method according to the embodiment of the present invention includes a unit interval separation step (S10), a script character string generation step (S20), a subtitle character string generation step (S30), and a keyword generation step (S30). S40), a search step (S50) and a summary video generation step (S60) can be included. Here, the video service providing method according to the embodiment of the present invention can be executed by the service server.

Hereinafter, a method of providing a moving image service according to an embodiment of the present invention will be described with reference to FIG. 7.

In the unit interval separation step (S10), the moving image can be separated into a plurality of unit sections based on the change in the characteristics of the sound included in the moving image. Here, changes in audio characteristics can include changes in volume or sound quality. Specifically, it is possible to extract a stop section in which the speaker's utterance in the moving image is interrupted by using the change in the characteristics of the voice, and to set the stop section as an editing point to separate the moving image. For example, the stop section can be set to a section in which the volume is reduced below the set value and the volume reduced below the set value is maintained for a reference time or longer. That is, when considering the context and the like, it is possible to set one section until the speaker in the moving image stops talking, and for this purpose, the amount of change in volume can be used when the unit intervals are separated.

In the script character string generation step (S20), the voice included in the unit interval can be recognized and the script character string corresponding to the voice can be generated. After separating the video into multiple unit intervals, it is necessary to recognize the content contained in each unit interval. For this purpose, it is possible to recognize the voice spoken by the speaker, convert it into characters, and combine the converted characters to generate a script character string.

Depending on the embodiment, a voice recognition device may be provided, and the voice recognition device can be used to convert the voice into characters. For example, the voice included in the unit interval can be represented by a voice pattern that is an electrical signal, and a standard voice pattern corresponding to each character may be stored in a voice model database or the like. In this case, the voice recognition device can compare the input voice pattern with the standard voice pattern stored in the voice model database, and can extract the standard voice pattern corresponding to each voice pattern. After that, the extracted standard voice pattern can be converted into the corresponding character, and the converted characters can be combined to generate a script character string.

In the subtitle character string generation step (S30), the subtitle image included in the unit interval can be recognized and the subtitle character string corresponding to the subtitle image can be generated. Since the content of the video is summarized and displayed in the subtitle image, it is necessary to recognize the characters contained in the subtitle image. However, since the subtitle image is recognized not by characters but by shape, it is necessary to apply a character recognition algorithm or the like in order to recognize the characters included in the subtitle image. Here, the subtitle character string generation step (S30) can be executed at the same time as the script character string generation step (S20), but the present invention is not limited to this.

Depending on the embodiment, another character recognition device may be provided, and the subtitle image can be converted into characters by using the character recognition device. For example, a subtitle image included in a unit interval can be scanned and the distribution of pixel values with respect to the subtitle image can be represented by a shape pattern that is an electrical signal, and a standard shape pattern corresponding to each character is stored in a character model database or the like. It may have been done. In this case, the character recognition device can compare the input shape pattern with the standard shape pattern stored in the character model database, and can extract the standard shape pattern corresponding to each shape pattern. After that, it is possible to generate a subtitle character string by converting each character into a character corresponding to the extracted standard shape pattern.

On the other hand, in order to extract the subtitle character string from the subtitle image, it is necessary to determine the presence or absence of the subtitle image in the unit interval and the position of the subtitle image in the moving image frame. That is, before generating the subtitle character string, it is possible to first detect the moving image frame including the subtitle image within the unit interval and specify the position of the subtitle image included in the moving image frame. Specifically, in the subtitle character string generation step (S30), a plurality of landmarks are set in the moving image frame included in the unit interval, and the subtitle image is detected by a method of measuring the hue or the brightness in the landmarks. it can. Further, the position of the subtitle image can be specified by extracting the landmarks whose reference hue or reference brightness corresponding to the subtitle image is measured after the landmarks are uniformly distributed on the moving image frame.

In the keyword generation step (S40), natural language processing can be applied to the script character string and the subtitle character string to generate the keyword corresponding to the unit interval. That is, after the user confirms the content of the unit interval, the keyword and the annotation can be automatically set based on the meaning for each unit interval, instead of setting the keyword and the annotation corresponding to the content. Here, various methods can be applied to the natural language processing applied to the script character string and the subtitle character string, and machine learning such as word2vec and LDA can be applied depending on the embodiment.

According to one embodiment, in the keyword generation step (S40), a word2vec model in which words are embedded using word2vec can be realized, and a word extracted from a subtitle character string or a script character string is set as an input word for the word2vec model. Then, the related words corresponding to the input words can be extracted. After that, the extracted related words can be set as keywords for the unit interval.

Here, the keyword generation step (S40) can be restricted so that the related words and the input words are compared and only the related words having a high degree of similarity are set as the keywords. Specifically, the similarity between the input word vector corresponding to the input word input to the word2vec model and the related word vector corresponding to the related word is calculated, and only the related words having a high similarity are extracted. Can be set as a keyword.

Each word can be vectorized and distributed in space through word embedding, so that words that are set to be similar or related to each other in the trained sample are located adjacent to each other in vector space. become. Therefore, the similarity between the input word vector and the related word vector can be calculated to grasp the relationship between the input word and the related word. Here, the similarity between vectors can be calculated using the cosine similarity.

Specifically, a related word vector having a similarity with the input vector of a predetermined value or more can be extracted, and a related word corresponding to the extracted related word vector can be set as a keyword. That is, only related words corresponding to the related word vector having a similarity of a predetermined value or more can be set as a keyword. Further, depending on the embodiment, it is possible to extract an already set number of related word vectors according to the order of similarity with the input vector, and the association corresponding to the extracted already set number of related word vectors. You can also set a word as a keyword. For example, 10 related word vectors having the highest degree of similarity can be extracted, and the extracted 10 related words can be set as keywords.

Further, in the keyword generation step (S40), an embodiment in which a keyword is set using real-time search term information is also possible. For example, among the related words extracted from the word2vec model, the related words corresponding to the search words included in the real-time search word information can be extracted, and the extracted related words are weighted at the time of calculating the similarity. Can be added. That is, even when the similarity is relatively low, the related words corresponding to the real-time search word information can be set as keywords by weighted values. At this time, weighted values provided to related words corresponding to the search words can be given differently from each other according to the real-time search order of the search words.

On the other hand, depending on the embodiment, a keyword can be set using LDA in the keyword generation step (S40). That is, a script character string and a subtitle character string can be applied to a machine learning model trained using LDA to extract a theme word corresponding to a unit interval, and then the extracted theme word can be used as a keyword for the unit interval. Can be set to. However, since the content of setting keywords using the machine learning model learned using LDA is described above, the specific content is omitted here.

Further, depending on the embodiment, a keyword for the entire moving image can be generated in the keyword generation step (S40). That is, it is possible to apply natural language processing to the keywords set for each unit interval included in the moving image to generate the keyword corresponding to the moving image. Here, machine learning such as word2vec and LDA can be applied to the natural language processing technique.

In the search step (S50), the unit interval corresponding to the keyword input by the user can be searched, and the searched unit interval can be provided to the user. Since a keyword is set for each unit interval, it is possible to search for a unit interval containing a specific content and provide it to the user. Further, since the search can be performed for each unit interval separated from the moving image, only the unit interval desired by the user can be provided. That is, it is possible to significantly improve user convenience when providing a video service.

In the summary video generation step (S60), unit intervals corresponding to the reference keywords can be extracted for the same video, and the extracted unit intervals can be combined to generate a summary video for the video. Here, the reference keyword may be preset by the administrator or may be input by the user. That is, it is not necessary to perform another editing work on the moving image, and the summary moving image can be easily generated and provided to the user.

The present invention described above can be realized as a computer-readable code on a medium in which a program is recorded. The computer-readable medium may be one that continuously stores a computer-executable program or a temporary storage for execution or download. Also, the medium may be various recording or storage means in the form of a single piece or a combination of several pieces of hardware, not limited to a medium directly connected to a computer system, but distributed over a network. It may exist. Examples of media include hard disks, magnetic media such as floppy disks and magnetic tapes, optical recording media such as CD-ROMs and DVDs, and magnetic-optical media such as floptic discs. There are optical media), and those configured to store program command words including ROM, RAM, flash memory, and the like. Further, as an example of other media, a recording medium or a storage medium managed by an app store that distributes applications, a site that supplies or distributes various other software, a server, or the like can be mentioned. Therefore, the above detailed description should not be construed in a restrictive manner in all respects and should be taken into account as exemplary. The scope of the invention must be determined by a reasonable interpretation of the appended claims, and all modifications within the equivalent scope of the invention are within the scope of the invention.

The present invention is not limited to the embodiments described above and the accompanying drawings. It is clear that a person having ordinary knowledge in the technical field to which the present invention belongs can replace, modify and change the components according to the present invention without departing from the technical idea of the present invention.

1 ・ ・ ・ Terminal equipment 10 ・ ・ ・ Processor 20 ・ ・ ・ Memory control unit 30 ・ ・ ・ Peripheral interface unit 40 ・ ・ ・ Memory 100 ・ ・ ・ Service server 110 ・ ・ ・ Unit interval separation unit 120 ・ ・ ・ Script character Column generation unit 130 ・ ・ ・ Subtitle character string generation unit 140 ・ ・ ・ Keyword generation unit 150 ・ ・ ・ Search unit 160 ・ ・ ・ Summary video generation unit

Claims (17)

A video service provision method in which a service server provides video to a terminal device.
A unit interval separation step that separates the video into a plurality of unit intervals based on changes in the characteristics of the audio contained in the video.
A script character string generation step that recognizes the voice included in the unit interval and generates a script character string corresponding to the voice.
A subtitle character string generation step that recognizes a subtitle image included in the unit section and generates a subtitle character string corresponding to the subtitle image, and sets a plurality of landmarks in a moving image frame included in the unit section. Then, the subtitle character string generation step of detecting the subtitle image using the hue or brightness measured at the landmark, and applying natural language processing to the script character string and the subtitle character string correspond to the unit interval. A method of providing a video service that includes a keyword generation step to generate a keyword to be generated. The unit interval separation step
Using said characteristic change of voice extracts stop section the speech of the speaker in the video is interrupted, you separate the video by setting the stop section in the editing point, moving according to claim 1 Service provision method. The unit interval separation step
Decreasing the volume of the sound is less than the set value, if it is maintained volume of less than the set value reference time or longer, you determine as the stop section, moving service providing method according to claim 2. The script character string generation step is
Using the speech recognition device, converts voice pattern extracted from the voice into corresponding text, that generates the script text by combining the transformed character, moving service providing method according to claim 1 .. The subtitle character string generation step is
Using the character recognition device, converts the shape pattern extracted from the caption image corresponding character, that generates the caption character string by combining the transformed character, video services provided according to claim 1 Method. The subtitle character string generation step is
From the plurality of landmarks uniformly distributed on the moving image frame, the landmarks whose reference hue or reference brightness corresponding to the subtitle image is measured are extracted, and the extracted landmarks are used to obtain the subtitle image. position that identifies a video service providing method according to claim 1. The keyword generation step is
The word embedding the word2vec model using word2vec caption text or enter an input word extracted from the script the string, and extracts the corresponding related words, to set the associated word to the keyword, claim The method for providing a video service according to 1. The keyword generation step is
The step of calculating the similarity between the input word vector corresponding to the input word and the related word vector corresponding to the related word input to the word2vec model.
A step of extracting a related word vector having a similarity equal to or higher than a predetermined value or an already set number of related word vectors selected according to the order of increasing similarity, and a related word corresponding to the extracted related word vector. the to set the keywords, moving service providing method of claim 7. The step of calculating the similarity is
Extract the relevant words corresponding to the search word included in the real time search word information, you added weight when calculating the degree of similarity for the relevant word the extracted, moving service providing method according to claim 8 .. The keyword generation step is
In response to said search term real-time search ranking, to grant weight to provide the associated word corresponding to the keyword to be different from each other, moving service providing method according to claim 9. The keyword generation step is
LDA (Latent Dirichlet Allocation) by applying the script text and subtitle character string in machine learning models trained using extracts the subject word corresponding to the unit section, to set the subject word to the keyword, The method for providing a video service according to claim 1. The keyword generation step is
By applying natural language processing keyword corresponding to the unit section, that generates a keyword corresponding to the moving picture, moving picture service providing method according to claim 1. Find the unit section corresponding to the keyword input by the user, further including a searching step of providing the retrieved unit segment to the user, video service providing method according to claim 1. For the same video, extracting unit section corresponding to the reference keyword summaries moving image generation step further including generating a summary video by combining the extracted unit sections, video services provided according to claim 1 Method. A computer program that is combined with hardware to cause the hardware to execute the video service providing method according to any one of claims 1 to 14 . A unit interval separator that separates the video into a plurality of unit sections based on changes in the characteristics of the audio contained in the video.
A script character string generator that recognizes the voice included in the unit interval and generates a script character string corresponding to the voice.
A subtitle character string generator that recognizes a subtitle image included in the unit section and generates a subtitle character string corresponding to the subtitle image, and sets a plurality of landmarks in a moving image frame included in the unit section. Then, the subtitle character string generator that detects the subtitle image using the hue or brightness measured at the landmark , and the script character string and the subtitle character string are subjected to natural language processing to correspond to the unit section. A service server that contains a keyword generator that generates the keywords you want. A service server that includes a processor and memory attached to the processor.
The memory comprises one or more modules configured to be executed by the processor.
The one or more modules
The video is divided into a plurality of unit intervals based on the change in the characteristics of the audio contained in the video.
The voice included in the unit interval is recognized, a script character string corresponding to the voice is generated, and the script character string is generated.
Wherein setting a plurality of landmarks in the video frames included in unit section, recognizes the subtitle image included in the unit section to detect a caption image with a hue or luminance was measured in the landmark, Generate a subtitle character string corresponding to the subtitle image,
Natural language processing is applied to the script character string and the subtitle character string to generate a keyword corresponding to the unit interval.
A service server that contains a command word.

Images