KR101072347B1 - Method and device of annotating scene context information of a moving picture - Google Patents

Abstract

In order to annotate the video status information, a predetermined scene in the video corresponding to the specific elapsed time is extracted by using time information about the elapsed time provided by the timed text including the elapsed time and the text, and at least appearing in the predetermined scene. After recognizing the speaker and the non-talker from the face of one character, the action type in the predetermined scene of the video is identified from the information on the recognized speaker, non-speaker, and text of the timed text, and the action type is used to determine the motion picture. Create scene context information about the scene situation. Accordingly, it is possible to annotate a situation for a predetermined scene, and use this to easily search for a desired position in the video and to effectively reduce the content of the video.

Annotation, Face Recognition, Video, Situation Information, Speaker Recognition

Description

METHOOD AND DEVICE OF ANNOTATING SCENE CONTEXT INFORMATION OF A MOVING PICTURE}

The present invention relates to a video situation information annotation method and apparatus, and more particularly, to a video situation information annotation method and apparatus using face recognition.

Recently, due to the development of the Internet and digital multimedia, video generated from various media such as video produced by various service providers and personally made UCC video has been increased exponentially through various means such as the Internet.

Accordingly, the necessity for finding a desired video from the numerous videos as described above and the need for shortening of the video have been continuously increased. In order to search and shorten such a video, there is an urgent need to annotate the video information by extracting meaning from the video information.

In general, the annotation method of the video is classified into a text-based method and a visual information method. remind The text-based method extracts the contents of an image from text information around the image, and the visual information method extracts and annotates visual information from the image itself .

The text-based method has a certain limitation because there is a case where there is no information related to the video or information related to the video around the video. In addition, since the conventional visual information method extracts and annotates visual information from the image itself, there is an advantage in extracting the information directly from the video itself even if there is no related information in the surroundings, but studies on the information extraction method are still insufficient.

Accordingly, an object of the present invention is to provide a video situation information annotation method capable of extracting meaning within a video itself.

Another object of the present invention is to provide a video contextual information annotation apparatus that can extract meaning within a video itself.

In accordance with an exemplary embodiment of the present invention, a video status information annotation method includes an elapsed time and a specific elapsed time using time information about the elapsed time provided by a timed text including text. Extracting a predetermined scene in a video corresponding to the method, recognizing a speaker and a non-speaker from a face of at least one character appearing in the predetermined scene, and recognizing a text of the recognized speaker and non-speaker and the timed text. Identifying an operation form in a predetermined scene of the video from the information and generating scene situation information regarding a situation of the predetermined scene of the video using the operation form.

In one embodiment, the step of recognizing the speaker and the non-speaker, detecting the position of the left and right eyes from the face of the character, calculating a first distance between the left and right eyes, from the position of the left eye A position spaced apart by the first distance below is set to the upper left position of the box representing the mouth of the character, and spaced apart from the position of the right eye by the distance of the first distance plus a second distance which is a predetermined constant value. Setting the position to the lower right position of the box representing the mouth of the character and using the histogram change of the inner image of the box defined by the upper left position and the lower right position to present the face with the change as the speaker. And determining that the face does not have a change as the non-talker.

The operation type in the predetermined scene of the video may be grasped based on a predetermined rule by using the number of speakers, the number of non-talkers, and the presence or absence of the text.

The scene situation information may be generated as an XML file including at least one of the elapsed time, the name of the speaker, the name of the non-speaker, the operation form, and the content of the text.

The video situation information annotation apparatus according to an exemplary embodiment of the present invention includes an image processor, a text processor, and a scene situation generator. The image processor recognizes the speaker and the talker from the face of at least one character appearing in a predetermined scene in the video, and provides the first context information about the recognized speaker and the talker. The text processing unit provides second context information regarding timed text including elapsed time and text. The scene situation generation unit generates scene situation information about a situation of a predetermined scene of the video from the first situation information provided from the image processing unit and the second situation information provided from the text processing unit.

In one embodiment, the image processor may include a face recognition unit and a speaker determiner. The face recognition unit extracts a face of the character from a predetermined scene in the video and recognizes who the extracted face is. The speaker determiner determines who is the speaker and the non-talker among the faces recognized by the face recognizer.

The video situation information annotation device may further include a face database. The face database stores faces of characters appearing in the video. In this case, the face recognition unit may recognize who the extracted face is using the face database.

In one embodiment, the text processing unit may include a time information unit and a text information unit. The time information unit includes time information about the elapsed time with respect to a predetermined scene of the video. The text information unit includes text information regarding the content of text corresponding to the elapsed time of the video. The time information unit may transmit the time information to the image processor, and the image processor may determine the predetermined scene by using the time information transmitted from the time information unit.

In one embodiment, the scene situation generator may include a context information extractor and an XML generator. The contextual information extractor extracts the scene contextual information by using the first contextual information and the second contextual information respectively provided from the image processor and the text processor. The XML generator converts the extracted scene situation information into an XML file and stores the XML file.

The video situation information annotation apparatus may further include a rule database storing rules for setting an operation type by using the number of speakers, the number of non-talkers, and the presence or absence of the text. The scene situation information may include the rules. It may include an operation type set using rules stored in a database.

The XML file may include, for example, at least one of the elapsed time, the name of the speaker, the name of the non-speaker, the operation form, and the content of the text.

According to the present invention, after recognizing a face of a character in a specific scene of a moving picture, a speaker and a non-talker are determined, and together with the text information, scene situation information regarding a situation of a predetermined scene is generated. The situation can be extracted from the image itself and annotated, and this can be used to easily search for a desired position within the video and to effectively reduce the content of the video.

Therefore, it can be used for a video search service, an effective application service is possible in a movie or UCC video service providing site, and the like, and can be widely used for hearing impaired people and foreign language education software.

In addition, when the scene situation information is generated in the XML file format, the portability is excellent, and thus it may be used more easily and conveniently in various systems.

As the inventive concept allows for various changes and numerous embodiments, particular embodiments will be illustrated in the drawings and described in detail in the text. However, this is not intended to limit the present invention to the specific disclosed form, it should be understood to include all modifications, equivalents, and substitutes included in the spirit and scope of the present invention.

Terms such as first and second may be used to describe various components, but the components should not be limited by the terms. The terms are used only for the purpose of distinguishing one component from another. For example, without departing from the scope of the present invention, the first component may be referred to as a second component, and similarly, the second component may also be referred to as a first component.

The terminology used herein is for the purpose of describing particular embodiments only and is not intended to be limiting of the invention. Singular expressions include plural expressions unless the context clearly indicates otherwise. In this application, the terms "comprise" or "having" are intended to indicate that there is a feature, number, step, action, component, part, or combination thereof described in the specification, and that one or more other features It should be understood that it does not exclude in advance the possibility of the presence or addition of numbers, steps, actions, components, parts or combinations thereof.

Unless defined otherwise, all terms used herein, including technical or scientific terms, have the same meaning as commonly understood by one of ordinary skill in the art.

Terms such as those defined in the commonly used dictionaries should be construed as having meanings consistent with the meanings in the context of the related art, and shall not be construed in ideal or excessively formal meanings unless expressly defined in this application. I do not.

DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS Reference will now be made in detail to the preferred embodiments of the present invention, examples of which are illustrated in the accompanying drawings.

1 is a flowchart illustrating a video situation information annotation method according to an embodiment of the present invention.

Referring to FIG. 1, in order to annotate video status information according to an embodiment of the present invention, a corresponding elapsed time is first used by using information about an elapsed time provided by a timed text. The predetermined scene in the video is extracted (S110).

The timed text refers to text synchronized with a medium such as audio and video. The timed text includes an elapsed time corresponding to an elapsed time based on the beginning of an image and a text synchronized with the elapsed time. The elapsed time may be the actual elapsed time or the number of elapsed frames. For example, the elapsed time may be expressed in milliseconds.

In one embodiment, the timed text is subtitle information stored with the video, DVD subtitle information, SMI format (for example, Windows Media Player), RealText (for example, RealPlayer), W3C of the video player of the computer It may include the proposed DFXP.

Subsequently, a person who is not a speaker and a speaker (hereinafter referred to as “non-speaker”) is recognized from the face of at least one character appearing in the predetermined scene (S120).

The speaker and the non-talker can be recognized through the change of the face of the character. In one embodiment, the speaker and the non-talker can be determined using the movement of the mouth. The non-talker has little change in the frame over time because there is little movement around the mouth. On the other hand, since the speaker moves around the mouth and the mouth, the change is relatively large with respect to the frame over time. Therefore, the mouth and the movement around the mouth may be determined using, for example, a histogram graph for each frame.

FIG. 2 is a flowchart illustrating a specific example of recognizing a speaker and a talker of FIG. 1, and FIG. 3 is a conceptual diagram illustrating a step of recognizing the speaker and the talker of FIG. 2.

2 and 3, in order to recognize the speaker and the non-talker, the position of the left eye ((x1, y1)) and the position of the right eye from the face of the character 110 appearing in the scene 100 of the video first. ((x2, y2)) is detected, and a first distance d1 between the left eye and the right eye is calculated (S122). Since human eyes are much easier to detect than the mouth due to the color characteristics, the position of the left and right eyes can be detected first, and then the relative position of the mouth can be detected therefrom. In a general person, the position of the mouth may be determined based on the first distance d1.

Subsequently, the position spaced apart from the position of the left eye by the first distance d1 is set to the upper left position ((x1, y1 + d1)) of the box representing the mouth of the character (S124). The position spaced apart from the position by the distance plus the predetermined second distance d2 is set to the lower right position ((x2, y2 + d1 + d2)) of the box representing the mouth of the character. (S126). The second distance d2 may be selected and used from a value smaller than the first distance d1. For example, the second distance d2 may be set in a range of 1/3 to 2/3 of the first distance. Thereby, the box containing the mouth of a character can be set.

Then, using the histogram change of the inner image of the box defined by the upper left position and the lower right position, the face with the change is determined as the speaker and the face without the change is determined as the non-talker ( S128). That is, when there is a change in the histogram of the inner box image, it may be determined that the mouth is moving, and such a case may be determined as the speaker's mouth.

The method of recognizing the speaker and non-speaker as described above may be appropriately modified and used when the side of the character appears. That is, instead of using the distance between the left eye and the right eye, the size of one eye can be detected and the position of the mouth can be used to observe the movement of the mouth.

Meanwhile, face recognition may be preceded by comparing a person who appears in the specific scene with the face database of a predetermined character during or before recognizing the speaker and the non-speaker. Can be.

Referring back to FIG. 1, next, the operation type in a predetermined scene of the video is determined from information about the recognized speaker, non-speaker, and text of the timed text (S130).

The operation mode may be grasped based on a predetermined rule using, for example, the number of speakers, the number of non-talkers, and the presence or absence of the text. As a specific example, the operation type may be grasped based on the following rule. Table 1 shows an example of a rule for determining the action type between characters.

number Characters Number of speakers Non-talkers Text Operation mode One 2 One One has exist (Speaker) say to 2 One 0 One has exist (Listener) listen 3 One One 0 has exist (Speaker) speak alone 4 3 or more One 2 or more has exist (Narrator) talk with (listeners) 5 2 or more 0 2 or more has exist (Listeners) listen 6 0 0 0 has exist nothing (or open caption) 7 - - - none nothing

Referring to Table 1, the operation type can be determined from the number of speakers and non-talkers and the presence or absence of text.

If there is text while the speaker is recognized and the speaker is also recognized, if the speaker and the speaker are recognized as one person as shown in the number 1, it can be judged as 'say to' speaking between the speaker and the speaker (listener). If, as shown in the number 4, the speaker is one person and the speaker is recognized as two or more, the speaker can be determined as 'talk with' when the speaker speaks or discusses the speaker.

If the text is present when the speaker is not recognized and only the non-speaker is recognized, there is a speaker that does not appear in the video, so the non-visible speaker is the listener and the listener is listening to the invisible speaker. Judging by At this time, when the non-speaker is recognized as one as shown in the number 2, the non-speaker (listener) is answering the call, or the criminal is questioned. ) Corresponds to a scene where you are listening to a lecture or taking a class.

If the text is present in the state in which the non-speaker is not recognized and only the speaker is recognized as shown in the number 3, it may be determined that the speaker speaks alone in the scene. In this case, the talker may be talking on the phone or speaking.

If both the speaker and the non-talker are not recognized as shown in the number 6 and only the text exists, it can be determined as 'nothing' in which no situation related to the character exists in the scene. In this case, the text may correspond to a simple description or narration of the scene, such as an open caption, or a translation of a character appearing in the scene.

The number 7 is a case in which there is no text regardless of the speaker or non-speaker's perception, and it can be determined as 'nothing' in which no dialogue is performed between the characters in the scene. If neither the speaker nor the speaker is recognized, it may be a scene that represents a landscape, etc. If the speaker or the speaker is recognized (the speaker is recognized because there is no text, it may be misidentified or only the shape of the mouth). It may be in a standby state or a state facing each other.

Referring back to FIG. 1, scene situation information regarding a situation of a predetermined scene of the video is generated using the operation mode (S140).

The scene situation information may include, for example, at least one of the elapsed time, the name of the speaker, the name of the non-speaker (listener), the operation form, and the content of the text (subtitle). The scene situation information may be provided in various forms for annotation, and may be generated as an XML file as an example.

4 illustrates an example of an XML file that is a result of generating scene situation information of FIG. 1.

Referring to FIG. 4, the XML file 200 includes an elapsed time 210, a speaker's name 220, a speaker's name 230, an operation form 240, and text content 250.

For example, using the scene situation information of the XML file 200 shown in FIG. 4, when the time of "1000" has elapsed, the speaker "Gap-d" is the listener "Gap-soon". You can see that "I'm talking".

The XML file 200 may further include an ID 260 for identifying scene situation information at the elapsed time 210. In FIG. 4, the ID 260 is set to the same value as the elapsed time 210.

The scene situation information generated as described above may be used to annotate a situation of the predetermined scene. The scene situation information may be used to easily search for a desired position in the video and to effectively reduce the content of the video.

5 is a block diagram illustrating a video contextual information annotation apparatus according to an embodiment of the present invention.

Referring to FIG. 5, the video situation information annotation apparatus 300 according to an embodiment of the present invention includes an image processor 310, a text processor 320, and a scene situation generator 330.

The image processor 310 recognizes the speaker and the talker from the face of at least one character appearing in a predetermined scene in the video, and provides the first context information regarding the recognized talker and the talker.

In an embodiment, the image processor 310 may include a face recognition unit 312 and a speaker determiner 314.

The face recognition unit 312 extracts a face of the character from a predetermined scene in the video and recognizes who the extracted face is. In this case, the video situation information annotation apparatus 300 may further include a face database 340. The face database 340 stores faces of characters appearing in the video. The face recognition unit 312 may recognize who the extracted face is among the characters using the face database 340 in which the face of the character is stored.

The speaker determiner 314 determines who is the speaker and the non-talker among the faces recognized by the face recognizer 312. The speaker determiner 314 may determine the speaker and the non-speaker using various methods, and may determine the speaker and the non-speaker using the method described with reference to FIGS. The first situation information may include information about the speaker and the talker determined, and the information about the speaker and the talker may be provided to the scene situation generator 330 to be described later.

The text processing unit 320 provides second context information regarding timed text including elapsed time and text. Since the description of the timed text is substantially the same as that described with reference to FIG. 1, detailed descriptions thereof will be omitted.

In one embodiment, the text processing unit 320 may include a time information unit 322 and a text information unit 324.

The time information unit 322 includes time information about the elapsed time with respect to a predetermined scene of the video. The time information about the elapsed time may be information about an elapsed time or the number of elapsed frames as described with reference to FIG. 1. The time information unit 322 may transfer the time information to the image processor 310, and the image processor 310 may determine the predetermined scene by using the time information transmitted from the time information unit 322. That is, the image processor 310 may extract a predetermined scene in a video corresponding to a specific elapsed time by using the time information transmitted from the time information unit 322.

The text information unit 324 includes text information regarding the content of text corresponding to the elapsed time of the video. For example, the text information is the content of text that appears in the elapsed time. The second situation information may include the text information, and the text information may be provided to the scene situation generation unit 330 to be described later.

The scene situation generating unit 330 obtains scene situation information about a situation of a predetermined scene of the video from the first situation information provided from the image processing unit 310 and the second situation information provided from the text processing unit 320. Create

In an embodiment, the scene situation generator 330 may include a context information extractor 332 and an XML generator 334.

The context information extractor 332 extracts the scene context information using the first context information and the second context information provided from the image processor 310 and the text processor 320, respectively. The scene situation information may include an operation form of a character in a predetermined scene of the video. The operation type may be grasped based on a preset rule.

For example, the video situation information annotation apparatus 300 may further include a rule database 350. The rules database 350 stores rules that set an operation type by using the number of speakers, the number of non-talkers, and the presence or absence of the text. The scene situation information may include an operation form of a character set using rules stored in the rule database 350. The rules may be set, for example, as described in Table 1.

The XML generator 334 converts the extracted scene situation information into an XML file and stores the XML file. The XML file may include, for example, at least one of the elapsed time, the name of the speaker, the name of the non-speaker, the operation form, and the content of the text. For example, as shown in FIG. 4, the XML file may include an elapsed time 210, a speaker's name 220, a speaker's name 230, an operation form 240, and text content 250. have.

The scene situation information generated as described above may be used to annotate a situation of the predetermined scene. The scene situation information may be used to easily search for a desired position in the video and to effectively reduce the content of the video.

While the present invention has been described in connection with what is presently considered to be practical and exemplary embodiments, it is to be understood that the invention is not limited to the disclosed embodiments, but, on the contrary, It will be apparent to those skilled in the art that various modifications and variations can be made in the present invention without departing from the spirit and scope of the invention. Therefore, the above description and the drawings below should be construed as illustrating the present invention, not limiting the technical spirit of the present invention.

1 is a flowchart illustrating a video situation information annotation method according to an embodiment of the present invention.

2 is a flowchart illustrating a specific example of a step of recognizing a speaker and a non-speaker of FIG. 1.

FIG. 3 is a conceptual diagram illustrating a step of recognizing a speaker and a non-speaker shown in FIG. 2.

4 illustrates an example of an XML file that is a result of generating scene situation information of FIG. 1.

5 is a block diagram illustrating a video contextual information annotation apparatus according to an embodiment of the present invention.

<Short Description of Main Drawing Numbers>

100: Movie Scene 200: XML File

300: video status information annotation device

310: image processor 312: face recognition unit

314: speaker determination unit 320: text processing unit

322: time information unit 324: text information unit

330: Scene situation generation unit 332: Context information extraction unit

334: XML generator 340: face database

350: rules database

Claims (12)

delete Extracting a predetermined scene in a video corresponding to a specific elapsed time by using the time information about the elapsed time provided by the timed text including the elapsed time and the text; Recognizing a speaker and a non-speaker from a face of at least one character appearing in the predetermined scene; Identifying an operation type in a predetermined scene of the video from the recognized speaker, non-speaker, and information about the text of the timed text; And Generating scene situation information about a situation of a predetermined scene of the video using the operation form; Recognizing the speaker and non-speakers, Detecting positions of left and right eyes from the face of the character and calculating a first distance between the left and right eyes; The position spaced apart from the position of the left eye by the first distance is set to the upper left position of the box representing the mouth of the character, and the second distance, which is a predetermined constant value, is set downward from the position of the right eye. Setting the spaced apart distance by a further distance to the lower right position of the box representing the mouth of the character; And Determining the face with the change as the speaker and the face without the change as the non-talker using a histogram change of the inner image of the box defined by the upper left position and the lower right position. Annotation method for video status information, characterized in that the. Extracting a predetermined scene in a video corresponding to a specific elapsed time using time information on the elapsed time provided by the timed text including the elapsed time and the text; Recognizing a speaker and a non-speaker from a face of at least one character appearing in the predetermined scene; Identifying an operation form in a predetermined scene of the video from the recognized speaker, non-speaker, and information about text of the timed text; And Generating scene situation information about a situation of a predetermined scene of the video using the operation form; The operation form in the predetermined scene of the video, The video situation information annotation method of claim 1, wherein the video is identified based on a predetermined rule by using the number of speakers, the number of non-talkers, and the presence or absence of the text. Extracting a predetermined scene in a video corresponding to a specific elapsed time using time information on the elapsed time provided by the timed text including the elapsed time and the text; Recognizing a speaker and a non-speaker from a face of at least one character appearing in the predetermined scene; Identifying an operation type in a predetermined scene of the video from the recognized speaker, non-speaker, and information about the text of the timed text; And Generating scene situation information about a situation of a predetermined scene of the video using the operation form; The scene situation information, And an XML file including at least one of the elapsed time, the name of the speaker, the name of the non-speaker, the operation type, and the text. delete An image processor for recognizing a speaker and a speaker from at least one character's face appearing in a predetermined scene in the moving image and providing first situation information regarding the recognized speaker and the speaker; A text processing unit for providing second context information about timed text including elapsed time and text; And A scene situation generating unit generating scene situation information about a situation of a predetermined scene of the video from the first situation information provided from the image processing unit and the second situation information provided from the text processing unit; The image processor, A face recognition unit extracting a face of the character from a predetermined scene in the video and recognizing who the extracted face is; And And a speaker determiner configured to determine who is the speaker and the non-talker among the faces recognized by the face recognition unit. An image processor for recognizing a speaker and a speaker from at least one character's face appearing in a predetermined scene in the moving image and providing first situation information regarding the recognized speaker and the speaker; A text processing unit for providing second context information about timed text including elapsed time and text; A scene situation generating unit generating scene situation information about a situation of a predetermined scene of the video from the first situation information provided from the image processing unit and the second situation information provided from the text processing unit; And A face database storing a face of a character appearing in the video, And the face recognition unit recognizes who is the extracted face using the face database. An image processor for recognizing a speaker and a speaker from at least one character's face appearing in a predetermined scene in the moving image and providing first situation information regarding the recognized speaker and the speaker; A text processing unit for providing second context information about timed text including elapsed time and text; And A scene situation generating unit generating scene situation information about a situation of a predetermined scene of the video from the first situation information provided from the image processing unit and the second situation information provided from the text processing unit; The text processing unit, A time information unit including time information about the elapsed time with respect to the predetermined scene of the video; And And a text information unit including text information on content of text corresponding to the elapsed time of the video. The method of claim 8, The time information unit transmits the time information to the image processing unit, And the image processing unit determines the predetermined scene using the time information transmitted from the time information unit. An image processor for recognizing a speaker and a speaker from at least one character's face appearing in a predetermined scene in the moving image and providing first situation information regarding the recognized speaker and the speaker; A text processing unit for providing second context information about timed text including elapsed time and text; And A scene situation generating unit generating scene situation information about a situation of a predetermined scene of the video from the first situation information provided from the image processing unit and the second situation information provided from the text processing unit; The scene situation generation unit, A context information extracting unit extracting the scene context information using the first context information and the second context information provided from the image processor and the text processor, respectively; And And an XML generator for converting and extracting the extracted scene context information into an XML file. An image processor for recognizing a speaker and a speaker from at least one character's face appearing in a predetermined scene in the moving image and providing first situation information regarding the recognized speaker and the speaker; A text processing unit for providing second context information about timed text including elapsed time and text; A scene situation generating unit generating scene situation information about a situation of a predetermined scene of the video from the first situation information provided from the image processing unit and the second situation information provided from the text processing unit; And A rule database storing rules for setting an operation type by using the number of speakers, the number of non-talkers, and the presence or absence of the text; And the scene situation information includes an operation form set using rules stored in the rule database. delete

Images