KR101911613B1 - Method and apparatus for person indexing based on the overlay text of the news interview video - Google Patents

Abstract

The present invention relates to an overlay text-based person indexing method and apparatus for automatically generating and collecting person indexing information using a news interview image,
The method includes extracting only a sub clip including an overlay text from a news interview image; Extracting the person information text and the keyword text of the interviewee from the overlay text of the sub clip; Obtaining a face thumbnail image of the interviewee in the sub clip; And generating a person indexing table based on the person information text, the keyword text, and the face thumbnail of the person to be interviewed.

Description

BACKGROUND OF THE INVENTION 1. Field of the Invention The present invention relates to a method and apparatus for indexing text based on an overlay text of a news interview image,

The present invention relates to an overlay text-based person indexing apparatus for a news interview image, which enables automatic generation and collection of person indexing information using a news interview image.

In general, text in a video sequence is divided into overlay text and scene text. Overlay texts, called graphical texts or subtitles, are created graphically on editing and are overlaid artificially over video images.

Examples of the overlay text include a subtitle of a news video and a score of a sports game. Examples of the scene text may include various texts recorded on the image pickup result of the camera such as a street sign, a truck license plate, and a shirt letter.

Overlay text is an essential problem in automated content analysis systems such as scene understanding, indexing, searching and searching, because overlay text is used primarily for the purpose of providing additional important or summary information about a video sequence.

Particularly, in the case of the news interview image, as shown in FIG. 1, the interview information has additional features such as name and age of the person to be interviewed, and the content of the interview is additionally provided through the overlay text.

Therefore, although the overlay text provided through the news interview image provides a lot of information necessary for indexing the person, no technology has been developed so far to utilize it.

However, only the concept of extracting additional information about a moving image character is disclosed only in a separate means such as a subtitle / script information database in Korean Patent Laid-Open No. 10-2011-0032347.

Korean Patent Laid-Open No. 10-2011-0032347 (Published date: March 30, 2011)

Accordingly, the present invention provides an overlay text-based person indexing method and apparatus for automatically generating and collecting person indexing information using a news interview image.

The objects of the present invention are not limited to the above-mentioned objects, and other objects not mentioned can be clearly understood by those skilled in the art from the following description.

According to an aspect of the present invention, there is provided a method for extracting only a sub clip including an overlay text from a news interview image. Extracting the person information text and the keyword text of the interviewee from the overlay text of the sub clip; Obtaining a face thumbnail image of the interviewee in the sub clip; And generating a person indexing table of the interviewee based on the person information text, the keyword text, and the face thumbnail. The present invention also provides an overlay text-based person indexing method of a news interview image.

The step of extracting the sub clip may include: converting an edge map image of the news interview image, and tracking an edge density change based on the edge map image; Initiating extraction of the sub-clip if the edge density increment is greater than or equal to a first set value; And terminating extraction of the sub-clip if the edge density reduction amount is equal to or greater than a second set value.

Wherein the extracting of the person information text and the keyword text comprises: detecting an overlay text area of the sub clip using a multiple edge map algorithm; Obtaining a horizontal projection histogram of the overlay text area, and separating the character information line and the keyword line based on the horizontal projection histogram; And extracting the person information text and the keyword text from the person information line and the keyword line, respectively.

Wherein detecting the overlay text area of the subclip comprises: extracting a plurality of reference frames from the subclip; A step of logically multiplying the edge map images of the reference frames to obtain a multi-edge map image from which background image edges have been removed; And detecting the overlay text area based on the number of monochrome transitions of the multi-edge map image.

The step of detecting an overlay text area of the sub clip may further include correcting the overlay text area through a morphological operation.

Wherein the step of distinguishing the character information line from the keyword line comprises: obtaining a horizontal projection histogram of the overlay text area; And dividing the first horizontal line among the horizontal lines having the horizontal projection histogram by more than a predetermined value into a character information line and the remaining horizontal lines by a keyword line.

Wherein the acquiring of the horizontal projection histogram acquires a horizontal projection histogram only for the overlay text region located in a lower region of the news image.

The step of extracting the character information text and the keyword text may include extracting texts of the character information line and the keyword line through OCR (Optical Character Recognition), respectively. And extracting the keywords of the person information of the interviewee and the interview contents by performing the object name recognition operation on the text extraction result.

The step of generating the person indexing table may further include performing an Internet information search based on the person information of the person to be interviewed and further storing the Internet information search result in the person indexing table.

The creating of the person indexing table may further include storing the news interview image and the person indexing table together in a database.

According to another embodiment of the present invention, a subtitle extracting unit extracts only a subclip including an overlay text in a news interview image. A text extraction unit for extracting a person information text and a keyword text of an interviewee in an overlay text area of the sub clip; A face thumbnail extracting unit for extracting a face thumbnail image of the interviewee from the sub clip; And a person indexing table generating unit for generating a person indexing table based on the person information text, the keyword text, and the face thumbnail image of the person to be interviewed. The present invention also provides an overlay text based person indexing apparatus for a news interview image.

By using the news interview image of the present invention, it is possible to automatically generate and collect person indexing information, so that a large amount of person indexing information can be constructed and provided with minimum time and cost.

1 is a view showing an example of a general news interview image.
FIG. 2 is a diagram for explaining an overlay text-based person indexing method of a news interview image according to an exemplary embodiment of the present invention.
FIG. 3 is a diagram for explaining a step of extracting a subclip according to an embodiment of the present invention in more detail.
FIG. 4 is a view for explaining a method of separating an overlay text existing section of a news interview image in a step of extracting a sub clip according to an embodiment of the present invention.
5 is a view for explaining a change in edge density according to an overlay text existence period of a news interview image according to an exemplary embodiment of the present invention.
6 is a diagram for explaining a step of detecting an overlay text area according to an embodiment of the present invention in more detail.
7 is a diagram for explaining a reference frame acquisition method according to an embodiment of the present invention.
FIG. 8 is a diagram illustrating a multi-edge map image and an overlay text area detection result according to an embodiment of the present invention.
9 is a diagram for explaining the step of extracting person information text according to an embodiment of the present invention in more detail.
FIG. 10 is a diagram for explaining the step of extracting a keyword text according to an embodiment of the present invention in more detail.
11 is a diagram for explaining a step of extracting a face thumbnail image according to an embodiment of the present invention in more detail.
12 is a view showing a person indexing table according to an embodiment of the present invention.
13 is a diagram for explaining an overlay text-based person indexing apparatus of a news interview image according to an embodiment of the present invention.

It is noted that the technical terms used in the present invention are used only to describe specific embodiments and are not intended to limit the present invention. In addition, the technical terms used in the present invention should be construed in a sense generally understood by a person having ordinary skill in the art to which the present invention belongs, unless otherwise defined in the present invention, Should not be construed to mean, or be interpreted in an excessively reduced sense. In addition, when a technical term used in the present invention is an erroneous technical term that does not accurately express the concept of the present invention, it should be understood that technical terms can be understood by those skilled in the art. In addition, the general terms used in the present invention should be interpreted according to a predefined or prior context, and should not be construed as being excessively reduced.

Furthermore, the singular expressions used in the present invention include plural expressions unless the context clearly dictates otherwise. In the present invention, terms such as "comprising" or "comprising" and the like should not be construed as encompassing various elements or various steps of the invention, Or may further include additional components or steps.

Hereinafter, preferred embodiments of the present invention will be described in detail with reference to the accompanying drawings, wherein like reference numerals refer to the same or similar elements, and redundant description thereof will be omitted.

In the following description, well-known functions or constructions are not described in detail since they would obscure the invention in unnecessary detail. It is to be noted that the accompanying drawings are only for the purpose of facilitating understanding of the present invention, and should not be construed as limiting the scope of the present invention with reference to the accompanying drawings.

FIG. 2 is a diagram for explaining an overlay text-based person indexing method of a news interview image according to an exemplary embodiment of the present invention.

Referring to FIG. 2, the figure indexing method of the present invention includes a step of receiving a news interview image (S10), extracting only a sub clip including an overlay text area in a news interview image (S20) Extracting a character information text in an overlay text area (S40), extracting a keyword text in an overlay text area (S50), extracting a face thumbnail image in a sub clip (S60) And creating a person indexing table based on the extracted person information, keyword, and face thumbnail (S70).

That is, in the present invention, not only the face image of the interviewee but also the person information and the keyword of the interviewee are extracted from the news interview image, and then the person indexing information is automatically generated using the information.

FIG. 3 is a diagram for explaining a step of extracting a subclip according to an embodiment of the present invention in more detail.

Referring to FIG. 3, in operation S20 of extracting a subclip of the present invention, an edge map image of a news image is generated through a Canny edge detector (S21) So that the change is tracked (S22). When the edge density increase amount is equal to or larger than the first value (S23) as a result of the edge density variation amount tracing step S22, the sub clip is started to be extracted (S24). After a predetermined time has elapsed, (S25), the sub clip extraction is ended (S26).

The overlay text of the news image is produced and provided according to the rules of news program production. For example, it is displayed on the lower or upper left corner of the frame to minimize the occurrence of image blurring. Also, in the case of the interview video, there is a characteristic that the person information of the interviewee is described in the first line, and the interview contents are described through the remaining lines.

In addition, in the case of a news video, overlay text is not included in every frame of the news video, and the feature disappears after a certain period of time after being displayed gradually or suddenly. As shown in FIG. 4, the provision period of the news image includes a non-text period in which overlay text is not displayed, a transition period in which the text starts to be overlaid, And a text period.

Accordingly, in the present invention, the text detection and recognition operation is performed only on the frames belonging to the text period, so that the time and cost required for text detection and recognition can be minimized.

Since the text includes many edges, the frame containing the overlay text has a very large edge density value compared to the frame that does not contain the overlay text. For example, as shown in FIG. 5, the edge density rapidly increases from the frame 13 to the frame 20 belonging to the transition period, and after the start frame 21 belonging to the text existence period (Text period) It can be seen that the density difference hardly occurs.

Therefore, in the present invention, the start frame can be detected by tracking the edge density variation of the news interview image, and the frame after the start frame is started to be extracted as the sub clip. In addition, by tracking the edge density variation of the news interview image on the same principle, the extraction of the subclips after the generation of the end frame is ended, so that only the frames belonging to the text period can be easily extracted into subclips.

6 is a diagram for explaining a step of detecting an overlay text area according to an embodiment of the present invention in more detail.

Referring to FIG. 6, the step S30 of detecting an overlay text area according to the present invention includes: selecting n reference frames from a subclip (S31); ANDing all edge map images of n reference frames, A step S32 of obtaining a multi-edge map image from which a background edge component is removed, a step S33 of detecting an overlay text area based on the number of monochrome transitions of the multi-edge map image, and a morphological closing closing and dilation operations to correct the overlay text area (S34).

If the subclip is played at f frames / sec, the overlay text will be in a fixed position for at least 2f frames. Assuming that k is an integer not less than f, consecutive k frames may be defined as one round.

In the present invention, for calculation simplification, n reference frames are extracted by repeatedly sampling subclips belonging to the first round at a predetermined period (for example, k / n-1 period). That is, as shown in FIG. 7, four reference frames can be extracted by repeatedly sampling the subclips belonging to the first round at intervals of 1, [k / 3], 2 [k / 3], and 3 [k / 3] .

When the n reference frames are thus obtained, the edge map image of each of the n reference frames is zigzag scanned to eliminate the horizontal line or the vertical line which is larger than the width (w) or the height (h) First, remove it.

Then, all the edge map images of the n reference frames are logically multiplied to obtain a multi-edge map image from which all the background edge components have been removed as shown in FIG. 8A.

After acquiring the number of monochrome transitions (N _trans ) using Equation (1) while zigzag scanning the multi-edge map image, an overlay text area is obtained by combining all the regions having the number of monochrome transitions greater than a predetermined value.

[Equation 1]

In this case, w and h are the width and height of the scanning window, b (o) is the multi-edge map image, and the size of the scanning window is determined depending on the size of the character.

Finally, morphological closing and dilation operations are sequentially performed to correct the overlay text area as shown in FIG. 8 (b). In other words, it solves the problem of loss of some pixels of characters caused by the logical multiplication of all the reference frame edge images through morphological closing and expansion operations.

9 is a diagram for explaining the step of extracting person information text according to an embodiment of the present invention in more detail.

As shown in FIG. 9, in step S40 of extracting the person information text of the present invention, a horizontal projection histogram is obtained by performing a horizontal projection on the overlay text area, and a horizontal projection histogram (S41) of detecting a text written on the character information line by performing OCR (Optical Character Recognition) on the character information line (S42), detecting an object name And extracting the person information of the person to be interviewed by performing the operation (S43).

A news video is a video produced by a professional organization such as a broadcasting station, and overlay text is generated and provided under a news program production rule. In particular, the overlay text corresponding to the interview image is displayed in the lower area of the upper area, the middle area, and the lower area of the frame, the person information of the person to be interviewed is described in the first line, and the contents of the interview are described through the remaining lines Feature.

Accordingly, in the present invention, the horizontal projection is performed only for the overlay text area located in the lower area of the news image, rather than the entire area of the news image, so that the time and cost required for text detection and recognition can be further reduced.

In consideration of the fact that the person information of the person to be interviewed is described in the first line of the overlay text, the first line of the overlay text is a character information line and the remaining line is a keyword line .

In addition, the object name recognizing operation of the present invention recognizes object names such as a person's name, affiliation / institution, occupation / position, etc. based on Conditional Random Fields (CRF) machine learning.

For example, if you enter the text "Ernest / White House spokesperson" and you receive an object named "Person: Ernest, Organization / Organization: White House, Occupation: Spokesperson, It is possible to obtain the recognition result of the name.

FIG. 10 is a diagram for explaining the step of extracting a keyword text according to an embodiment of the present invention in more detail.

10, in the step S50 of extracting the keyword information text of the present invention, the horizontal projection histogram is obtained through the horizontal projection of the overlay text area, and the horizontal projection histogram of the first line (S52) of performing text recognition on the keyword line by performing OCR on the keyword line (S52), and performing an object name recognition operation on the text detection result And extracting a related keyword (S53).

That is, in the present invention, the first line of the overlay text is acquired as the character information line, the remaining lines are acquired as the keyword line, and then the object name recognition operation described above is performed in the same manner, So that at least one keyword that can define contents can be extracted.

For example, "Jeffrey Paul is returning home for family reunion with the permission to leave North Korea," he welcomed North Korea's decision to release him, saying, "Jeffrey Paul, North Korea, permission, family , Reunion, home, release, decision, illusion ".

11 is a diagram for explaining a step of extracting a face thumbnail image according to an embodiment of the present invention in more detail.

11, step S60 of detecting a face image of the present invention includes obtaining a start frame image of a subclip (S61), performing a face detection operation on a start frame image, (S62), storing the face detection result of the step S62 as a thumbnail image (S63), and the like.

However, if the face detection of the interviewee fails through the start frame image, the face detection of the interviewee can be performed through the next frame image.

In addition, if necessary, a face detection operation may be performed on various images (for example, n reference frames), and then one face detection result may be selected based on the face detection accuracy. The accuracy of face detection can be calculated considering the point of time of interview, face clipping, degree of tremor, and screen resolution.

12 is a view showing a person indexing table according to an embodiment of the present invention.

Referring to FIG. 12, the person indexing table of the present invention may include a face thumbnail, a name of an interviewee, a name / affiliation / organization, a job / position, other information, and a keyword.

In the face thumbnail, the face detection result of the interviewee will be automatically displayed, the person information detection result is automatically input in the name, affiliation / institution, job / position of the interviewee, and the keyword is automatically inputted .

The person indexing table of the present invention allows the person information detection result excluding name, affiliation / institution, and occupation / position to be stored in other information.

In the present invention, an Internet person DB (for example, Naver person information or Wikipedia person information) is searched based on the name of the person information of the person to be interviewed, and an Internet site in which person information related to the person to be interviewed is disclosed is found , And to provide additional link addresses for them. Also, the acquired person indexing table is stored in the database together with the input original news interview image so that the third person can browse the clip later.

13 is a diagram for explaining an overlay text-based person indexing apparatus of a news interview image according to an embodiment of the present invention.

13, the person indexing apparatus of the present invention includes a news interview image input unit 10 for receiving a news interview image, a sub clip extracting unit 20 for extracting a sub clip including an overlay text area in a news interview image A face thumbnail extracting unit 30 for extracting a face thumbnail image of a person to be interviewed in the sub clip, a text extracting unit 30 for extracting the person information and the keyword of the interviewee in the overlay text area after detecting the overlay text area of the sub clip, And a person indexing table creating unit 40 for creating a person indexing table based on the face thumbnail extracting unit 30 and the text extracting unit 40 and storing the same in the DB 60 together with the original news interview image, (50), and the like.

Also, the person indexing apparatus of the present invention may include various data input / output devices, a person indexing table generating unit 50 for receiving data provided from a user or an external apparatus, or providing a result of operation of the apparatus to a user or an external apparatus in real time And a communication device capable of connecting to the Internet network so as to be able to access the Internet network and retrieve information related to the interviewee.

The person indexing device configured in this way may be implemented as a single hardware device, but may be implemented as an embedded system coupled to an existing computing device, or software downloaded and installed in an existing computing device, if necessary.

The features, structures, effects and the like described in the foregoing embodiments are included in at least one embodiment of the present invention and are not necessarily limited to one embodiment. Further, the features, structures, effects, and the like illustrated in the embodiments may be combined or modified in other embodiments by those skilled in the art to which the embodiments belong.

Therefore, it should be understood that the present invention is not limited to these combinations and modifications. While the present invention has been particularly shown and described with reference to exemplary embodiments thereof, it is clearly understood that the same is by way of illustration and example only and is not to be construed as limiting the scope of the present invention. It can be seen that various modifications and applications are possible. For example, each component specifically shown in the embodiments may be modified and implemented. It is to be understood that the present invention may be embodied in many other specific forms without departing from the spirit or essential characteristics thereof.

Claims (11)

Extracting only the subclips including the overlay text from the news interview image;
Extracting the person information text and the keyword text of the interviewee from the overlay text of the sub clip;
Obtaining a face thumbnail image of the interviewee in the sub clip; And
And generating a person indexing table of the person to be interviewed based on the person information text, the keyword text, and the face thumbnail. 2. The method of claim 1, wherein extracting the sub-
Transforming the news interview image into an edge map image and tracking an edge density change based on the edge map image;
Initiating extraction of the sub-clip if the edge density increment is greater than or equal to a first set value; And
And terminating the extraction of the sub clip if the edge density reduction amount is equal to or greater than a second set value. 2. The method of claim 1, wherein extracting the person information text and the keyword text comprises:
Detecting an overlay text area of the subclip using a multiple edge map algorithm;
Obtaining a horizontal projection histogram of the overlay text area, and separating the character information line and the keyword line based on the horizontal projection histogram; And
And extracting the character information text and the keyword text from the character information line and the keyword line, respectively. 4. The method of claim 3, wherein detecting an overlay text area of the subclip comprises:
Extracting a plurality of reference frames from the subclip;
A step of logically multiplying the edge map images of the reference frames to obtain a multi-edge map image from which background image edges have been removed; And
And detecting the overlay text area based on the number of black-and-white transitions of the multi-edge map image. 5. The method of claim 4, wherein detecting an overlay text area of the subclip comprises:
Further comprising the step of correcting the overlay text area through morphological operations. ≪ Desc / Clms Page number 19 > 4. The method of claim 3, wherein the step of distinguishing the character information line from the keyword line comprises:
Obtaining a horizontal projection histogram of the overlay text area; And
The method comprising the steps of: dividing a first horizontal line among the horizontal lines having the horizontal projection histogram equal to or greater than a predetermined value into a character information line and a remaining horizontal line into a keyword line; . 7. The method of claim 6, wherein obtaining the horizontal projection histogram comprises:
And a horizontal projection histogram is acquired only for the overlay text area located in a lower area of the news interview image. 4. The method of claim 3, wherein the step of extracting the character information text and the keyword text comprises:
Extracting text of each of the character information line and the keyword line through OCR (Optical Character Recognition); And
And extracting keywords of the person information and the interview contents of the interviewee by performing an object name recognition operation on the text extraction result. 2. The method of claim 1, wherein generating the person indexing table comprises:
Further comprising: performing Internet information search based on the person information of the person to be interviewed, and further storing an Internet information search result in the person indexing table. 2. The method of claim 1, wherein generating the person indexing table comprises:
Further comprising the step of storing the news interview image and the person indexing table together in a database. A sub clip extracting unit for extracting only subclips including the overlay text in the news interview image;
A text extraction unit for extracting a person information text and a keyword text of an interviewee in an overlay text area of the sub clip;
A face thumbnail extracting unit for extracting a face thumbnail image of the interviewee from the sub clip; And
And a person indexing table generating unit for generating a person indexing table based on the person information text, the keyword text, and the face thumbnail of the person to be interviewed.

Images