KR102267403B1 - Apparatus or Method for Detecting Meaningful Intervals using voice and video information - Google Patents

Abstract

As a system for extracting a meaningful interval according to an embodiment of the present invention, the system includes: a video segmentation module for registering or dividing a video to generate metadata and a video segmentation engine including the video registration module; a metadata auto tagging engine for automatically generating metadata of the registered or divided video; and a section determination engine for extracting a meaningful interval from the interval of the divided video. The interval determination engine can store a set consisting of a predetermined number of continuous caption information as one or more temporary meaningful intervals through calculation of a similarity score based on caption information. The stored temporary meaningful intervals can be composed of one or more shot intervals. Based on a built-in deep learning model, it is possible to quickly extract a scene interval estimated as the same event from a video by using voice information.

Description

Method and apparatus for detecting meaningful intervals using voice and video information {Apparatus or Method for Detecting Meaningful Intervals using voice and video information}

The present invention relates to a method, apparatus or system for detecting a meaningful section using audio and video information, and to more accurately detecting a video section.

With the development of deep learning, research related to artificial intelligence is increasing day by day. Among them, the field of the Turing test, in which a machine understands human language, recognizes a situation, and further explains the situation or background, is positioned as the most important research field in robotics engineering. However, research in the field of the Turing test includes the visual field that recognizes a person's face, object, background, etc., the auditory field that recognizes voice and sound, and the world knowledge field that builds a knowledge base by connecting content related to the real world. The items to be done are organically complex, and in particular, in order to proceed with these studies, a meaningful section (or semantic section) (hereinafter referred to as “scene” or “scene section”) must be intervened by a human to It should be divided into sections according to occurrence. Although human intervention and dividing the scene according to the occurrence of an event is generally highly accurate, it takes a lot of time because it is performed while watching all the videos, and the same error may occur because the criteria for determining that the event is the same for each individual are different. do. Unlike visual or auditory metadata that constructs a large number of identical information, the number of errors is small, and due to the nature of time series data, if one error occurs, it affects 2-3 sections, so the overall error rate increases significantly. .

Therefore, based on the built-in deep learning models, a scene section estimated as the same event is quickly extracted from the video using voice information, and the final metadata is automatically checked and corrected by using the video information in the extracted section information. I would like to suggest a way to make it a product.

The problems to be solved in the present invention are not limited to the problems to be solved, and other problems not mentioned may be clearly understood by those of ordinary skill in the art to which the present invention belongs from the following description. There will be.

A system for extracting a semantic section according to an embodiment of the present invention, the system comprising: a video segmentation engine including a video registration module and a video segmentation module for registering or dividing a video to generate metadata; a metadata auto tagging engine for automatically generating metadata of the registered or divided video; and a section determination engine for extracting a semantic section from the section of the divided video, wherein the section determination engine: A set consisting of a preset number of continuous caption information is calculated by calculating a similarity score based on the caption information. stored as a temporary semantic section, the stored temporary semantic section consists of one or more shot sections, and the section determination engine: sets the start caption information n of the similarity score calculation in the set for calculating the similarity score based on the caption information and, for K pieces of sequential caption information including caption information n to caption information n+K-1 in the set, a similarity score between caption information n and caption information n+1 to n+K-1, respectively, is a preset score n is incremented by 1 until the number of pairs of caption information having a similarity score of less than or equal to a predetermined number is calculated sequentially, and temporally trailing caption information among the last pair of caption information for which the similarity score is calculated is calculated as the similarity score is determined as the end caption information n' of ', and if the length of the caption information section defined by the start caption information and the end caption information is equal to or greater than a predetermined time length, the caption information section is matched with the shot section to give the temporary meaning Stored as an interval, where K is a predetermined constant, and n is an integer greater than 1 and may be a value with an upper bound.

Additionally or alternatively, when there is remaining caption information that is not stored as the temporary semantic section in the set, the section determination engine calculates the similarity score for the remaining caption information and attempts to store the next temporary semantic section. can do.

Additionally or alternatively, even if the number of similarity scores equal to or less than the preset score is smaller than the predetermined number, if there is no more subtitle information for which similarity scores are to be calculated, it is temporally trailing among the last subtitle information pairs for which the similarity score was calculated. The caption information is determined as the end point of the temporary semantic section, and if the length of time of the caption information section defined by the start point and the end point is equal to or greater than the predetermined time length, the caption information section is matched to the shot section and the It can be stored as a temporary semantic interval.

delete

Additionally or alternatively, a plurality of analysis schemes may be used for calculating the similarity score.

Additionally or alternatively, if the length of time of the caption information section defined by the start point and the end point is less than the predetermined time length, the caption information section may not be stored as a temporary semantic section.

Additionally or alternatively, for each of the stored temporary semantic sections, the section determination engine may: store the temporary semantic section as a final semantic section by calculating a similarity score based on image information of the corresponding temporary semantic section, or After the section is further divided, it can be stored as the final semantic section.

Additionally or alternatively, the section determination engine may set the temporally most advanced L shot sections of the corresponding temporary semantic section as a reference section for calculating the similarity score, and determine the similarity between the reference section and the remaining shot sections. have.

Additionally or alternatively, the calculation of the similarity score may include determining the similarity of person or object information or the similarity of position and size information of the person or object between each of the L shot sections and the remaining shot sections.

Additionally or alternatively, all similarity scores of person or object information between each of the L shot sections and the remaining shot sections are less than a preset reference score, and the person or object between each of the L shot sections and the remaining shot sections is less than a preset reference score. When the similarity scores of the location and size information are both less than a preset reference score, the stored temporary semantic section may be divided into two, and the starting point at which the stored temporary semantic section is divided may be the remaining shot sections.
According to another embodiment of the present invention, a method for extracting a semantic section is proposed, the method comprising: registering or dividing a video for generating metadata; automatically generating subtitle information of the registered or divided video; and extracting a temporary semantic section based on the caption information from the segmented video section, wherein the extracting includes calculating a similarity score based on the caption information for a set consisting of a preset number of continuous caption information. storing at least one temporary semantic section, wherein the temporary semantic section consists of one or more shot sections, and the storing of the temporary semantic section includes setting start caption information n for calculating the similarity score in the set step; For K pieces of sequential subtitle information composed of subtitle information n to n+K-1 in the set, the similarity score between each of the subtitle information n and the subtitle information n+1 to n+K-1 is equal to or less than a preset score. incrementing n by 1 until the number of pairs of caption information having scores becomes a predetermined number, and sequentially calculating; determining temporally trailing caption information among the last pair of caption information for which the similarity score is calculated as the ending caption information n' of the similarity score calculation; and if the length of the caption information section defined by the start caption information and the end caption information is equal to or greater than a predetermined time length, matching the caption information section with the shot section and storing the section as the temporary semantic section, Here, K is a predetermined constant, n is an integer greater than 1, and may be a value having an upper limit.

The above problem solving methods are only some of the embodiments of the present invention, and various embodiments in which the technical features of the present invention are reflected are based on the detailed description of the present invention to be described below by those of ordinary skill in the art can be derived and understood as

According to the present invention, there is an advantage that the acquisition of a meaningful section can be performed according to an objective condition, and the time for this can be shortened.

The effects obtainable in the present invention are not limited to the above-mentioned effects, and other effects not mentioned may be clearly understood by those of ordinary skill in the art to which the present invention belongs from the following description. will be.

BRIEF DESCRIPTION OF THE DRAWINGS The accompanying drawings, which are included as a part of the detailed description to help understand the present invention, provide embodiments of the present invention, and together with the detailed description, explain the technical spirit of the present invention.
1 is a block diagram of a meaningful section detection system using image information and audio information.
2 shows the configuration of a video segmentation engine.
3 shows a configuration of a data management engine and a metadata auto-tagging engine.
4 is a flowchart of a method for determining a semantic section using voice information.
5 is a flowchart of a method for determining a semantic section using image information.

Hereinafter, embodiments of the present invention will be described with reference to the accompanying drawings. However, the present invention is not limited to the embodiments described herein and may be implemented in various other forms. The terminology used in this specification is intended to help the understanding of the embodiments, and is not intended to limit the scope of the present invention. Also, singular forms used hereinafter include plural forms unless the phrases clearly indicate the opposite.

Robotics engineering can also be applied in fields that require complex judgment, such as robots that support precision medical care and robots that guide tourists at airports, from very simple fields such as a robot arm that automatically assembles objects in a factory, and a robot that automatically sells soft ice cream. It is widely used. The development of robotics engineering was mainly due to the rapid improvement of artificial intelligence technology, and if you go deeper, it can be seen that it originated from the development of various deep learning architectures. However, no matter how high-performance deep learning architecture is developed, it requires a large amount of metadata of comparable quality. By analogy, it can be expressed as quality food that can be placed in a good bowl, where the bowl becomes a deep learning architecture, and the food can be called metadata. In particular, the field of Turing test, which is receiving the most attention in robotics engineering, is a field that allows robots to listen to a human question and answer a question by calculating it, and furthermore, to provide an explanation of the situation or background knowledge. It is research. The study of the Turing test used in robotics is a combination of very diverse and complex research results. The robot uses a visual engine to distinguish objects or people, understands human language, and outputs internally calculated results in human language. A voice engine and natural language processing engine for the purpose of doing this, a situation knowledge engine that learns to guess the situation based on input visual or voice data, and a world knowledge engine configured to map input/output data with the real world. The number of engines that can be configured according to it is immeasurable, and thus various types of metadata are required.

Essentially, the basis of making robots understand human questions and answer them is, of course, learning human language. To this end, it is considered important to separate an image in units of scene sections based on one event, but all work of segmenting images in units of scenes requires a human to manually view the image. Human intervention in constructing metadata means that it takes a lot of time and money to build it manually, which is one of the main reasons for slowing down the research speed in the field of Turing tests. Hereinafter, a method of dividing a meaningful section using voice and a method of detecting a final meaningful section by reinforcing the reliability of a section using voice using image information will be described.

1 is a block diagram of a meaningful section detection system using image information and audio information. The meaningful section detection system 1 according to the present invention is composed of a video segmentation engine 10, a metadata auto tagging engine 20, a section determination engine 30, and a data management engine 40 interacting with them. do.

The video segmentation engine 10 is composed of a video registration module 11 and a video segmentation module 12 . The video registration module 11 converts and registers an image that requires metadata generation selected by a user, and stores the registered image in the video DB 41 . Referring to FIG. 2 , the function or operation of the video segmentation engine will be described in detail. The video division module 12 may divide the video stored in the video DB 41 in units of frames.

The video segmentation module 12 may extract a section in which a scene of an image stored in the metadata verification server is converted, and extract the image at 10 fps (frame per second) for each section. Here, a section in which a scene of an image is converted is referred to as a “shot”. The video segmentation module may extract and store section information for each section, and the section information may include a start time, an end time, and a shot index of the corresponding section. The extracted section information and images are stored in the image DB, and the stored images can be used when the metadata auto-tagging engine recognizes human faces and objects.

The metadata auto-tagging engine 20 includes a face recognition module 21 , an object recognition module 22 , and a voice recognition module 23 . The metadata auto-tagging engine automatically builds metadata for scene section detection. 3 shows the configuration and interrelationship of a metadata auto-tagging engine.

In the video segmentation engine, the image segmentation is completed by executing the face recognition module, object recognition module and voice recognition module in the metadata auto-tagging engine, and the result of recognizing faces, objects and voices is displayed in the voice DB 42 or image DB 43 ) can be stored in

The face recognition module and object recognition module recognize the face and object of a person in each image by using a deep learning model previously learned based on the image extracted through the video segmentation engine.

The learner of the recognition module outputs a result value with higher accuracy as it learns with high-quality data.

If the metadata that has passed through the face recognition module, object recognition module, and voice recognition module needs to be edited, it is edited in the metadata inspection engine (not shown) and used again as the data set of the learner of the recognition module to develop a new model with higher accuracy. can create

The face recognition module extracts the feature points of the eyes, nose, and mouth of the face in the image, and recognizes the face by vectorizing it using a CNN (Convolutional Neural Network)-based model. For the recognized face, a bounding box may be displayed at a corresponding position on the image, coordinates of the corresponding position may be extracted, and a recognized person name and reliability may be displayed together. That is, the coordinates, the person's name, and the reliability correspond to the automatically generated video metadata, which may be stored in the image DB 43 . Reliability is based on 70%, and results with less than 70% reliability can be stored as failures.

The object recognition module utilizes CNN-based GBD-Net (gated bi-direction CNN) that classifies objects using a feature map extracted from the features of the image, so that the position coordinates of all objects in the image and the names of the corresponding objects are recognize The position coordinates, object name, and reliability of the recognized object are stored in the image DB 43 . As with the face recognition module, a result of less than 70% based on 70% reliability is saved as a failure.

The speech recognition module generates speech start time, end time, length, and subtitles for the voice in the video. The speech recognition process is a pre-processing step that removes noise from the audio of the video, adjusts the size to an appropriate size for a signal that is too loud or too small, detects only the speech section among consecutive speeches, and optimizes it through a decoder and language model scoring. Subtitles are generated by converting speech into text using a domain dictionary construction correction algorithm as a post-processing.

For speaker extraction, speaker identification technology is used during speaker recognition. In order to apply the speaker identification technology, it is necessary to store the speaker's voice in advance. Using the Gaussian Mixture Model-Universal Background Model (GMM-UBM) technique that extracts and models features from the voice signal from the input image, the name of the person closest to the voice registered in the DB is extracted as the speaker. The speaker of the generated voice, utterance start and end times, and caption data may be stored in the voice DB 42 , and all of this information is included within the scope of video metadata.

The metadata generated by the metadata auto-tagging engine may be later used to detect the final semantic section by calculating reliability or similarity in the section determination engine 30 .

The data management engine 40 includes a video DB (management module) 41 , an audio DB (management module) 42 , and an image DB (management module) 43 . The data management engine manages the metadata included in the video, image metadata in 10fps unit extracted from the video, shot section information automatically extracted based on screen transition, auto-tagging results for each image, audio caption information, and audio section information. It stores and plays a pivotal role in performing requests of each module and responses to requests. Before detecting the semantic section, all preprocessing tasks are completed and the section determination engine is called.

Previously, a “scene or scene section” and a “shot or shot section” were defined, and the relationship between the two should be understood to include several “shots or shot sections” in one “scene or scene section”. That is, several shots or shot sections constitute one scene or scene section. Also, as mentioned above, the “scene section” is a term that can also be expressed as a “meaning section”.

The section determination engine 30 includes a section reliability analysis module 31 and a caption reliability analysis module 32 . The section determination engine 30 combines the image information and audio information extracted from each engine to determine reliability (or similarity) and determine a meaningful section. First, a method of determining a semantic section (scene or scene section) using voice information proceeds in the following order. The semantic section determination using the voice information is performed by the section determination engine 30 or the caption reliability analysis module 32 as follows.

On the other hand, as described above, the section determination engine 30 is a shot or shot section extracted from the data management engine 40 by the video segmentation engine 10 , the metadata auto tagging engine 20 , and the like, and metadata therefor. can be used to perform the method or method.

4 is a flowchart of a method for determining a semantic section using voice information.

The section determination engine 30 or the caption reliability analysis module 32 may refine caption information, that is, data (S411). The caption information may be obtained by separating only the audio information from the image, and then converting the dialogue part into a text form through a speech to text (STT) engine. The data purification is a process of normalizing the subtitle information so that only English and numbers remain, and then converting it into a word arrangement form by deleting stopwords such as investigations, conjunctions, and prepositions through morpheme analysis. In addition, in the data purification, after excluding short subtitles that are difficult to read as sentences, which are only 3 words or less, stem extraction that converts words into a circle, that is, stemming, is performed to determine the subtitle information, speaker information, and utterance time. It means that metadata is converted into a dictionary and stored in the form of a list.

Accordingly, a set composed of a predetermined number of continuous caption information is configured, and as a result, the refined data, that is, caption information, can be managed while being numbered in chronological order based on the time information of the video.

Then, the section determination engine 30 or the caption reliability analysis module 32 initializes the sequence number (n) of the start caption information to be subjected to similarity analysis, which will be described later, and the sequence number (i) of the start caption information of the temporary scene section. It can be (S412). Each initialized value is 1.

Among the target caption information, the temporally most advanced K pieces of sequential caption information may be selected, and a caption information section in which the selected K pieces of caption information are subjected to similarity analysis may be set ( S413 ) .

The selection of the K pieces of caption information may be repeated until a preset number (eg, c and c are integers greater than or equal to 1) that a similarity score between caption information is less than (or less than) a predefined score. That is, as a result of the similarity analysis on the K pieces of caption information in S413, if there are less than c pairs of caption information whose similarity score is less than or equal to a predefined score, the caption information section (K caption information) is selected again, but the starting caption information is It will be the caption information having the next sequence number of the start caption information of the previously selected caption information section. That is, if the sequence numbers of the caption information of the first selected caption information section are n to n+K-1, the sequence numbers of the caption information of the next selected caption information section will be n+1 to n+K.

In more detail, the section determination engine 30 or the caption reliability analysis module 32 may calculate a similarity score for a total of K caption information from the first caption (S414). That is, similarity scores for K-1 pairs of caption information may be calculated. In addition, for the K-1 pairs of caption information, similarity scores are sequentially calculated.

The section determination engine 30 or the caption reliability analysis module 32 calculates an average similarity score by performing a method to be described later (hereinafter, referred to as “subtitle similarity analysis”), and depending on the magnitude relationship between the reference value and the scene or scene You can perform segmentation.

In more detail, the caption similarity analysis may be performed by sequentially selecting K pieces of caption information, pairing the most advanced caption information with the remaining K-1 pieces of caption information, respectively. That is, the section determination engine 30 or the caption reliability analysis module 32 may calculate an average similarity score according to the caption similarity analysis for K-1 pairs of caption information ( S414 ). For example, if the sequence number of the first subtitle information is n, (n, n+1), (n, n+2), (n, n+3), ..., (n, n+K-1) ), a caption similarity analysis is performed on K-1 pairs of caption information.

The subtitle similarity analysis will be described in detail below.

To analyze the subtitle similarity, first, the similarity of the K-1 pairs of subtitle information may be compared using TF-IDF (Term Frequency-Inverse Document Frequency) analysis. TF-IDF may be expressed as a product of TF and IDF. TF means the frequency of occurrence of a specific word in a specific document. When applied to the present invention, it means the frequency of occurrence of the word k in the i-th subtitle information. IDF is literally the inverse of DF, and DF is the number of documents (ie, subtitle information) in which the word k appears. IDF is usually expressed as a logarithmic function and is as follows.

In the above equation, n is the total number of documents, that is, the total number of caption information according to the present invention. Since two pieces of caption information are compared with each other, n=2 here. d denotes a document, i.e. individual subtitle information, and t denotes a word. df(t) is the DF described above, and refers to the number of subtitle information in which the word t appears.

The TF-IDF value can be understood as a weight for a specific word, and is always less than 1.

The TF-IDF may be expressed as a vector for each subtitle information. For example, when three words appear in the caption information where n=1, the TF-IDF is obtained for each word, so three values are derived, and accordingly, it can be configured as a three-dimensional vector. That is, since two pieces of caption information are compared, two vectors of two same dimensions or different dimensions can be obtained. These two vectors can be expressed as values using cosine similarity.

Here, A and B respectively refer to two pieces of caption information, and Ai and Bi correspond to TF-IDF values of each caption information. This value can be expressed as a percentage, to be expressed as a similarity score, and will be referred to as A.

Vector similarity analysis is a method of calculating how similar words appearing in each subtitle have with a pre-trained model using Word2Vec. To use Word2Vec, a model that has learned numerical words called embedding vectors is used, and it is based on the results learned using full subtitles in advance. Each word has a weight between 0 and 1, for example, if 6 words appear in one subtitle information, and each weight has 0.043, 0.096, 0.034, 0.084, 0.012, 0.056, the average value of 0.054 to take it When the target for obtaining the weight of a word is the caption information n and the caption information n+1, only the value of the caption information n+1 as a comparison target is calculated. This result value can be expressed as a percentage in order to be expressed as a similarity score, and will be referred to as B.

Association analysis compares the similarity by measuring PMI (Pointwise Mutual Information) based on the frequency of appearance between the obtained main words as in S411. PMI is a measure of the interdependence between two variables and is calculated as follows.

Here, p(x, y) is the joint distribution function of X and Y, and p(x) and p(y) are the marginal distribution functions of X and Y, respectively. According to the present invention, X and Y will be i and j-th caption information, respectively, and x and y will be words included in the i and j-th caption information, respectively.

Since two pieces of caption information are compared, each word included in each caption information is compared, but words included in the same caption information are not compared with each other. For example, there is an array of 5 words (n(1), n(2), n(3), n(4), n(5)) in the nth subtitle, and 9 words in the n+1th subtitle. Array(n+1(1), n+1(2), n+1(3), n+1(4), n+1(5), n+1(6), n+1(7)) , n+1(8), n+1(9)), n(1) and n+1(1), n(1) and n+1(2), ... , compare n(1) to n+1(9), n(2) to n+1(1), n(2) to n+1(2), ... n(2)~n+1(9), ... are sequentially compared. When the result of association analysis of all words is obtained, the average of all values is calculated. The similarity score according to the above association analysis is called C.

In this way, the average similarity score is calculated using the similarity scores according to the above three analyses, and it can be defined as (average similarity score) = A+(B+C)/2.

The section determination engine 30 or the caption reliability analysis module 32 may determine whether the average similarity score is below or below a predefined reference score (eg, 50%), and calculates a count value according to the result It may be changed, and it may be checked whether the count value has reached a preset reference value (c, for example, c=5 mentioned above) (S415). For example, if the average similarity score is less than or equal to a predefined reference score, the count value may be incremented by 1, otherwise the count value is not changed. The initial value of this count is 0, and may be initialized when a temporary semantic section (scene section) is stored.

In addition, the check of the count value may be performed after calculating the similarity of each pair of caption information. Although not shown, S414 and S415 are performed individually and sequentially for each of K-1 pairs of subtitle information. That is, for example, the section determination engine 30 or the caption reliability analysis module 32 calculates a similarity analysis and average similarity score between the caption information n and the caption information n+1 (S414), and the calculated average similarity degree It is determined whether the score is less than or equal to a predefined score (eg, 50%), the count value is also checked (S415), and if the count value does not reach c, subtitle information n+1 and subtitle information n+ again Steps S414 to S415 may be repeated for step 2.

If the count value does not reach the preset reference value, the procedure may be repeated again with respect to the K pieces of subtitle information from the second sequence of subtitle information to S413 to S415 described above. This repetition may be performed until the count value reaches a preset reference value, and the sequence number of the starting caption information, which is the subject of the similarity analysis, is increased by one (S417). That is, similarity analysis will be performed on a total of K subtitles from the n+1 subtitle information to the n+K subtitle information.

Meanwhile, even if the count value has not reached the preset reference value, if the K pieces of caption information that are the subject of similarity analysis in S413 and S414 are the last caption information among the target caption information (S416), the procedure may proceed to S418. That is, it may be determined whether the sequence number of the first subtitle information among the K pieces of subtitle information exceeds P (the total number of subtitle information is P+K). In other words, if the last caption information (subtitle information n+K-1) among the K pieces of caption information is the last caption information, the procedure proceeds to S418.

When the count value reaches a preset reference value, the section determination engine 30 or the caption reliability analysis module 32 sets the temporally trailing caption information among the caption information pairs that are the subject of the last similarity score calculation to the temporary scene section. It can be set as the last subtitle (n') of (S418).

Then, the section determination engine 30 or the caption reliability analysis module 32 determines whether the length of the corresponding time exceeds or exceeds a preset length through the start and last caption information of the temporary scene section. It can be done (S419).

The section determination engine 30 or the caption reliability analysis module 32 may store information on the temporary scene section if the length of the time is equal to or greater than a preset length ( S420 ). When the information on the temporary scene section is stored, it may be matched with the above-described shot section and stored based on the time information of the start caption information and the last caption information. In general, since the caption information exists from the first frame or a subsequent frame in one shot section, the temporary scene section is highly likely not to coincide with the start or end time of the previously divided shot section. Since the final goal of the subtitle similarity analysis is to accurately divide a scene section and a shot section included therein, it is necessary to set the temporary scene section to be composed of individual shot sections. There is a need. For example, when the temporary scene section corresponds to the range of 00:03 to 02:30 in one image, the actual shot section may belong to 00:01 to 02:55 (shot sections 2 to 40). Accordingly, according to the matching result, the temporary scene section may be stored as shot sections 2 to 40.

Then, the section determination engine 30 or the caption reliability analysis module 32 sets the order of the caption similarity analysis to n'+1 (the caption information after the last caption information) in order to analyze the remaining caption information. ), and the sequence number of the start caption information for the temporary scene section may also be updated to n'+1 (S421).

Alternatively, if the length of the time is shorter than the preset length, the section determination engine 30 or the caption reliability analysis module 32 does not store information on the temporary scene section. Then, the section determination engine 30 or the caption reliability analysis module 32 sets the order of the first caption information of the caption similarity analysis to n'+1 (after the last caption information) in order to analyze the remaining caption information. of subtitle information), and the sequence number of the start subtitle information for the temporary scene section may also be updated to n'+1 (S421).

Then, the section determination engine 30 or the caption reliability analysis module 32 may check whether the K pieces of caption information exceed the target caption range of the caption similarity analysis according to the sequence of the updated start caption information ( S422). That is, it is determined whether additional analysis of the subtitle similarity is necessary (check whether n>Q, Q=P+K is the total number of subtitle information). If additional analysis is required, the procedure proceeds to S413; Otherwise, the procedure may be terminated.

According to the above-described temporary scene period determination method, the temporary scene period may be divided into one or more individual temporary scene periods.

Next, a method of determining the extracted temporary scene sections described above by using image information to determine them as the final section will be described.

5 is a flowchart of a method for determining a semantic section using image information.

The section determination engine 30 or the caption reliability analysis module 32 may purify data on the temporarily stored semantic section, ie, temporary scene sections, using the aforementioned voice (caption) information (S511). That is, the section determination engine 30 or the caption reliability analysis module 32 may store time information, metadata, location, size, label, reliability, etc. for all shot sections of the temporary scene sections as a list in advance. .

In addition, the temporary scene sections may be managed while being numbered in chronological order.

Then, the sequence number n of the starting scene section information, which is the target of image information similarity analysis to be described later, may be initialized (S512). The initialization value is 1. Also, the number of shot sections in each of the temporary scene sections may be set.

The section determination engine 30 or the caption reliability analysis module 32 may select the first L shot sections of the corresponding scene section and set them as the reference shot section (S513). That is, the first L shot sections of each temporary scene section serve as a criterion for image information similarity analysis, which will be described later.

The section determination engine 30 or the caption reliability analysis module 32 searches all shot sections of the temporary scene section using voice information (subtitle information) while comparing the reference shot section and the next shot section with each other. , the scene section may be finally determined by checking whether the type of person or object has a degree of similarity equal to or greater than a preset degree, and the size and location of the person or object has a degree of similarity higher than or exceeding a preset degree.

More specifically, image information similarity analysis is performed using i) person or object information (that is, a person or object that can be identified by name, name or type, etc.), ii) location information and size information of the person or object. can

The section determination engine 30 or the caption reliability analysis module 32 calculates a similarity analysis and similarity score between the reference shot information and the next shot section, and the person or object information can be used for this (S514).

To this end, object or person information of each shot section of the reference shot section may be stored in an array form, and object or person information of each shot section of the reference shot section and object or person information of the remaining shot sections A comparison of can be performed. That is, if the reference shot section consists of L shot sections, and each order is referred to as n, n+1, ..., n+L-1, the order of the next remaining shot section is n +L, and accordingly, the image information similarity analysis compares shot sections n and n+L, n+1 and n+L, ..., n+L-1 and n+L.

The section determination engine 30 or the caption reliability analysis module 32 may determine whether there is a shot section in which the similarity score exceeds or exceeds a preset reference score R1 (eg, 70%) (S515). When the similarity score exceeds or exceeds the preset score, the section determination engine 30 or the caption reliability analysis module 32 determines the position of a person or object for each shot section and the remaining shot sections of the reference shot section. A similarity analysis and a similarity score may be calculated based on the information and the size information (S516).

The section determination engine 30 or the caption reliability analysis module 32 may determine whether there is a shot section in which the similarity score exceeds or exceeds a preset reference score R2 (eg, 70%) (S517).

If there is a shot section in which the similarity score exceeds or exceeds the preset reference score, the similarity analysis may be performed on the next shot section (M+1=L+2). Accordingly, M=M+1 is set (S518). If there is no longer a shot section for similarity analysis for the n-th temporary scene section (S519), the section determination engine 30 or the caption reliability analysis module 32 can check whether a temporary scene section for similarity analysis remains. There is (S520). For example, when the total number of temporary scene sections is T, it may be determined whether n is equal to or greater than T. If there is a temporary scene section for the similarity analysis remaining, the section determination engine 30 or the subtitle reliability analysis module 32 performs the procedures of S513 to S519 or S522 to S523 for the next temporary scene section (n+1). Can be repeated. That is, n=n+1 and P=P _n may be set ( S521 ), and the procedure proceeds to S513 .

In S516 and S517, if there is no shot section in which the similarity score exceeds or exceeds the preset reference score, the section determination engine 30 or the caption reliability analysis module 32 divides the scene section from the next shot section as a starting point It can be done (S522). According to the division of temporary scene sections, the number T of temporary scene sections may be increased by one (S523). In this case, since one temporary scene section is divided into two, _{information on the number of shot sections (P n} ) of the temporary scene section must also be updated. That is, P _n =L, and P _n+1 =P _n -L.
Then, the section determination engine 30 or the caption reliability analysis module 32 may check whether a temporary scene section to be analyzed for similarity remains ( S520 ). If there is a temporary scene section for the similarity analysis remaining, the section determination engine 30 or the subtitle reliability analysis module 32 performs the procedures of S513 to S519 or S522 to S523 for the next temporary scene section (n+1). Can be repeated. That is, n=n+1 and P=P _n may be set ( S521 ), and the procedure proceeds to S513 .

When there is no longer a temporary scene section or shot section for which similarity analysis is to be performed, the section determination engine 30 or the caption reliability analysis module 32 may store the information stored as the temporary scene section as the final scene section (S524). That is, the finally set values of T, Pn, etc. will indicate the configuration of the final scene section.

In the above specification, "system" or "engine" or "module" belonging thereto has been described as performing the method or procedure, etc., but "system", "engine" or "module" is only a name and the scope of rights is limited thereto. not dependent. That is, in addition to the system, engine, and module, the method or procedure may be performed as a device or a component belonging thereto, and as well as software for reviewing or editing video metadata, or code readable by a computer or other machine, device, etc. The method or manner may be performed by

In addition, as another aspect of the present invention, the above-described proposal or operation of the invention is performed by a "computer" (a comprehensive concept including a system on chip (SoC) or (micro) processor, etc.) Code that can be implemented, implemented or executed, or a computer-readable storage medium or computer program product storing or including the code, may also be provided, and the scope of the present invention is that the code or the storage or storage of the code is provided. a computer-readable storage medium including a computer-readable storage medium or a computer program product.

Claims (10)

As a system for extracting a semantic interval,
a video segmentation engine including a video registration module and a video segmentation module for registering or dividing a video for generating metadata;
a metadata auto tagging engine for automatically generating metadata of the registered or divided video; and
and a section determination engine for extracting a semantic section from the section of the divided video,
The section determination engine: stores a set consisting of a preset number of continuous caption information as one or more temporary semantic sections by calculating a similarity score based on the caption information;
The stored temporary semantic section consists of one or more shot sections,
The section determination engine: For calculating a similarity score based on the subtitle information
Set the starting caption information n of calculating the similarity score in the set,
For K pieces of sequential subtitle information composed of subtitle information n to n+K-1 in the set, the similarity score between each of the subtitle information n and the subtitle information n+1 to n+K-1 is equal to or less than a preset score. n is incremented by 1 until the number of pairs of subtitle information having a score becomes a predetermined number, and sequentially calculate,
determining temporally trailing caption information among the last pair of caption information for which the similarity score is calculated as the ending caption information n' of the similarity score calculation;
If the length of the caption information section defined by the start caption information and the end caption information is greater than or equal to a predetermined time length, matching the caption information section to the shot section and storing the section as the temporary meaning section,
where K is a predetermined constant and n is an integer greater than 1 and a value having an upper bound. The method of claim 1, wherein the interval determination engine comprises:
and when there is remaining subtitle information that is not stored as the temporary semantic section in the set, the similarity score calculation is performed on the remaining subtitle information to attempt to store the next temporary semantic section. The method according to any one of claims 1 to 2, wherein the interval determination engine comprises:
Even if the number of pairs of caption information having a similarity score equal to or less than the preset score is smaller than the predetermined number, if there is no more caption information for calculating a similarity score,
determining temporally trailing caption information among the last pair of caption information for which the similarity score is calculated as end caption information of a temporary semantic section;
If the length of the caption information section defined by the start caption information and the end caption information is equal to or greater than the predetermined time length, the caption information section is matched with the shot section and stored as the temporary semantic section. . The method according to any one of claims 1 to 2,
A semantic interval extraction system in which a plurality of analysis methods are used to calculate the similarity score. The semantic section extraction according to claim 1, wherein if the length of the caption information section defined by the start caption information and the end caption information is less than the predetermined time length, the caption information section is not stored as a temporary semantic section. system. The interval determination engine according to any one of claims 1 to 2, wherein for each of the stored temporary semantic intervals:
A semantic section extraction system for storing the temporary semantic section as a final semantic section by calculating a similarity score based on image information of the corresponding temporary semantic section, or further dividing the temporary semantic section and storing the temporary semantic section as the final semantic section. The method of claim 6, wherein the interval determination engine comprises:
A semantic section extraction system for setting the temporally most advanced L shot sections of a corresponding temporary semantic section as a reference section, and determining a similarity between the reference section and the remaining shot sections. The method of claim 7, wherein the calculation of the similarity score comprises:
and determining the degree of similarity of person or object information or position and size information of person or object between each of the L shot sections and the remaining shot sections. 8. The method of claim 7,
All of the similarity scores of person or object information between each of the L shot sections and the remaining shot sections are less than a preset reference score, and the similarity of the position and size information of the person or object between each of the L shot sections and the remaining shot sections If all the scores are less than the preset reference score, the stored temporary semantic section is divided into two,
The starting point at which the stored temporary semantic section is divided is the remaining shot section. As a method for extracting a semantic interval,
registering or dividing a video for which metadata is to be generated;
automatically generating subtitle information of the registered or divided video;
extracting a temporary semantic section based on the subtitle information from the segmented video section,
The extracting includes storing one or more temporary semantic sections by calculating a similarity score based on the subtitle information for a set consisting of a preset number of continuous subtitle information,
The temporary semantic section consists of one or more shot sections,
The step of storing the temporary semantic section includes:
setting start caption information n for calculating the similarity score in the set;
For K pieces of sequential subtitle information composed of subtitle information n to n+K-1 in the set, the similarity score between each of the subtitle information n and the subtitle information n+1 to n+K-1 is equal to or less than a preset score. incrementing n by 1 until the number of pairs of caption information having scores becomes a predetermined number, and sequentially calculating;
determining temporally trailing caption information among the last pair of caption information for which the similarity score is calculated as the ending caption information n' of the similarity score calculation; and
If the length of the caption information section defined by the start caption information and the end caption information is equal to or greater than a predetermined time length, matching the caption information section to the shot section and storing the section as the temporary meaning section,
wherein K is a predetermined constant and n is an integer greater than 1 and a value having an upper bound.

Images