KR20090112020A - System and method for extracting caption candidate and system and method for extracting image caption using text information and structural information of document - Google Patents

Abstract

PURPOSE: A system and a method for extracting caption candidate, and a system and a method for extracting image caption using text information, and structural information of document are provided to improve image caption extracting function by calculating the probability that the image-caption candidate pair is the correct answer pair. CONSTITUTION: A system and a method for extracting caption candidate, and a system and a method for extracting image caption using text information comprises an image-caption candidate pair generating unit, a feature extracting unit, and an image caption selecting unit. The image-caption candidate pair generating unit creates the included in a document image-caption candidate pair about an image using a text and structural information of a document(S801). The feature extracting unit extracts the generated feature about the image-caption candidate pair(S802). The image caption selecting unit selects the image caption from the image-caption candidate pair of which is generated using the probability according to the extracted feature(S803).

Description

SYSTEM AND METHOD FOR EXTRACTING CAPTION CANDIDATE AND SYSTEM AND METHOD FOR EXTRACTING IMAGE CAPTION USING TEXT INFORMATION AND STRUCTURAL INFORMATION OF DOCUMENT}

The present invention relates to a caption candidate extraction system and method using text information and structural information of a document, and an image caption extraction system and method. In more detail, a caption candidate extraction system and method for extracting a caption candidate for each image of a document to generate an image-caption candidate, and extracting an image caption by calculating probabilities according to pre-learning for the generated image-caption candidate. And an image caption extraction system and method.

Recently, there is a growing interest in a technique for searching for an image included in a document. However, since a web document includes various images, when searching for an image actually required by a searcher in the web document, the web document includes various images.

In this case, when extracting an image caption which is a word describing an image rather than an attribute of the image itself and searching for the image caption, the performance of the image search may be improved. As a result, the performance of image caption extraction may affect the performance of the image retrieval system.

However, since the image caption is one of a plurality of texts included in the document, a criterion for selecting an image caption among the texts is required. It is also necessary to select an image that can be an image caption extraction target in advance. In addition, since there may be many texts describing the image, a criterion for selecting an optimal image caption from such text is also required.

The present invention provides a caption candidate extraction system and method and an image caption extraction system and method for efficiently extracting an image caption by generating an image-caption candidate pair of a document using text information and structural information of the document.

The present invention provides a caption candidate extraction system and method and an image caption extraction system and method that can improve caption candidate extraction performance by using rules according to image attributes and text attributes.

The present invention provides a caption candidate extraction system and method and an image caption extraction system and method for efficiently extracting a caption candidate for an image by structurally parsing a document and extracting actual position values of document components on a browser.

The present invention provides an image caption extraction system and method for improving image caption extraction performance by extracting an image caption by calculating a probability that an image-caption candidate pair generated through a pre-trained probability-based classification model becomes a correct answer pair.

A caption candidate extraction system according to an embodiment of the present invention includes a document parsing unit for parsing a document and extracting text information and structural information of the document, and an image for determining an image that is an image caption extraction target among images included in the document. The image filtering unit may perform filtering and a caption candidate determiner to determine a caption candidate for an image that is an image caption extraction target from text included in the document.

The image caption extraction system according to an embodiment of the present invention is an image-caption candidate pair generator for generating an image-caption candidate pair for each of the images included in the document using text information and structural information of the document. It may include a feature extraction unit for extracting a feature for each of the generated image-caption candidate pair and an image caption selection unit for selecting an image caption from the generated image-caption candidate pair by using the probability according to the extracted feature .

The caption candidate extraction method according to an embodiment of the present invention includes parsing a document to extract text information and structural information of the document, and performing image filtering to determine an image that is an image caption extraction target among images included in the document. And performing a caption candidate for the image that is the image caption extraction target from the text included in the document.

The image caption extraction method according to an embodiment of the present invention comprises the steps of generating an image-caption candidate pair for each of the images included in the document using text information and structural information of the document, the generated image-caption candidate Extracting a feature for each pair and selecting an image caption from the generated image-caption candidate pair using a probability according to the extracted feature.

According to the present invention, there is provided a caption candidate extraction system and method and an image caption extraction system and method for efficiently extracting an image caption by generating an image-caption candidate pair of the document using text information and structural information of the document. .

According to the present invention, there is provided a caption candidate extraction system and method and an image caption extraction system and method capable of improving caption candidate extraction performance by using rules according to image attributes and text attributes.

According to the present invention, there is provided a caption candidate extraction system and method and an image caption extraction system and method for efficiently extracting a caption candidate for an image by structurally parsing a document and extracting actual position values of document components on a browser. .

According to the present invention, there is provided an image caption extraction system and method for improving image caption extraction performance by extracting an image caption by calculating a probability that an image-caption candidate pair generated through a pre-learned probability-based classification model becomes a correct answer pair. do.

Hereinafter, with reference to the contents described in the accompanying drawings will be described in detail an embodiment according to the present invention. However, the present invention is not limited or limited by the embodiments. Like reference numerals in the drawings denote like elements. Caption candidate extraction method according to an embodiment of the present invention may be performed by a caption candidate extraction system. And, the image caption extraction method according to an embodiment of the present invention can be performed by the image caption extraction system.

1 is a diagram illustrating a process of extracting an image caption list using an image caption extraction system according to an embodiment of the present invention.

Referring to FIG. 1, the image caption extraction system 103 of the present invention uses image information and structural information in the periphery of an image to capture an image caption for each image included in each of the HTML web documents 102-1 to 102-n. May be extracted to generate an image caption list 104. Although the web 100 is illustrated in FIG. 1, other online environments may be applied in addition to the web.

In one example, images included in HTML web documents 102-1 through 102-n may or may not have image captions. The image caption extracted for each image may have at most one image caption. At this time. Image caption refers to text given separately from the body of HTML web documents 102-1 through 102-n to describe an image.

The image caption extraction system 103 may determine a caption candidate for each image included in the HTML web documents 102-1 through 102-n. In this case, the caption candidate may mean unit text that may become an image caption among texts included in the HTML web documents 102-1 to 102-n with respect to the image.

According to an embodiment of the present invention, the image caption extraction system 103 structurally parses the HTML web documents 102-1 to 102-n to web browsers 101-1 to 101-n of the image and caption candidates. The actual position value (structural information) of the image can be extracted. That is, when the HTML web document is provided through the web browser, the image caption extraction system 103 may extract the actual position of the image or caption candidate on the web browser. At this time, the types of web browsers 101-1 to 101-n are not limited. In addition, the HTML web documents 102-1 to 102-n are merely examples, and may be replaced with documents implemented in other languages (for example, XML, etc.) that can structurally construct the document according to a tag. .

As a result, the present invention uses the structural information extracted by parsing the HTML web documents 102-1 to 102-n, thereby reflecting the actual distance visually felt through the web browsers 101-1 to 101-n. More accurate image captions can be extracted for. In addition, the present invention can extract the image caption which is a wide unit of text in a wider range by using the structural information.

2 is a block diagram showing the overall configuration of a caption candidate extraction system according to an embodiment of the present invention.

Referring to FIG. 2, the caption candidate extraction system 200 may include a document parser 201, an image filter 202, and a caption candidate determiner 203. In this case, the caption candidate extraction system 200 may be a component of the image caption extraction system 103.

The document parsing unit 201 may parse the document and extract text information and structural information of the document. In this case, the text information may refer to a text form such as a length, a width, and a font of the text included in the document. In addition, the structural information may mean the location of the components of the document.

In one example, the document parsing unit 201 may parse the document to extract the actual position value of the document component on the browser. In this case, the document parsing unit 201 may extract a position value of each of the image and the text included in the document, thereby determining the actual distance on the web browser.

In this case, the document may include an HTML web document. For example, the document parsing unit 201 may parse an HTML web document composed of HTML using a parser according to the HTML interface.

The image filtering unit 202 may perform image filtering to determine an image, which is an image caption extraction target, among images included in a document. In this case, the image filtering unit 202 may perform image filtering to determine an image that is an image caption extraction target by using a filter according to the attributes of the images.

For example, the filter according to the attributes of the images may include a filter based on the size of the image, a filter based on the aspect ratio of the image, a filter based on the file name of the image, and a filter based on the overlapping arrangement of the images in the document.

For example, the image filtering unit 202 may filter an image in which the horizontal and vertical products of the image are smaller than a preset value through a filter according to the size of the image. The image filtering unit 202 may filter the image when the aspect ratio of the image is greater than or equal to a preset value through a filter based on the aspect ratio of the image.

In addition, the image filtering unit 202, the filter according to the image file name of the image, such as banner, icon, menu, logo, top, bottom, left, right, such as a menu, logo or banner on the document including the file name of the image If so, you can filter the image. In addition, the image filtering unit 202 may filter the images appearing in the document through a filter according to the overlapping arrangement of the images in the document. However, since the image of the array type may have an image caption describing the entire image of the array type, it may not be filtered exceptionally.

That is, when an image included in a document is highly likely to correspond to ancillary components of the document, the caption for the image may be meaningless. Accordingly, the filters according to the attributes of the images may refer to filters for removing an image corresponding to an additional component of the document.

The caption candidate determiner 203 may determine a caption candidate for the image that is the image caption extraction target from the text included in the document. In this case, the caption candidate may mean unit texts adjacent to an image caption extraction target image among the text included in the document.

For example, the caption candidate determiner 203 may determine the caption candidate for the unit text adjacent to each direction of the image that is the image caption extraction target through the image filtering unit 202. In this case, the caption candidate determiner 203 may extract the unit text by using the tag of the document, and determine the caption candidate by using the length of the text and the distance from the image in the unit text.

For example, the caption candidate determiner 203 may extract unit text using a tag for a paragraph or a table in the document. However, a tag for a paragraph or a table corresponding to a lower node of the tag may be excluded.

The caption candidate determiner 203 determines whether the extracted unit text corresponds to a preset length of text and a distance from the preset image (for example, two times or less of an average value of the width and height of the image). Caption candidate may be determined in consideration of whether the information is included in the "

For example, the caption candidate determiner 203 may determine the caption candidate for the unit text adjacent to each direction of the image. When the unit text is one or more in each direction, the caption candidate determiner 203 may determine the unit text closest to the image. Caption candidates may be determined. If there is at least one unit text closest to the image, the unit text may be determined as a caption candidate.

Here, when the criteria for determining the caption candidate among the text included in the document are not satisfied, the caption candidate may not be determined for a specific image that is an image caption extraction target. That is, at least one caption candidate determined for each image that is an image caption extraction target may or may not exist.

The present invention is not limited to the above example for determining the caption candidate for the unit text adjacent to the image, and various methods may be applied.

3 is a block diagram showing the overall configuration of an image caption extraction system according to an embodiment of the present invention.

Referring to FIG. 3, the image caption extraction system 300 may include an image-caption candidate pair generator 301, a feature extractor 302, and an image caption selector 303. In this case, the image-caption candidate pair generator 301 may be replaced with the caption candidate extraction system 200 illustrated in FIG. 2.

The image-caption candidate pair generator 301 may generate an image-caption candidate pair for each of the images included in the document using text information and structural information of the document. When a caption candidate is determined for the image according to a preset rule, an image-caption candidate pair is generated for each image.

In this case, the image-caption candidate pair generator 301 may include a document parser 304, an image filter 305, and a caption candidate determiner 306. That is, the image caption candidate pair generation unit 301 may determine a caption candidate for an image of the document, and generate an image-caption candidate pair for each image according to the determined caption candidate.

The document parsing unit 304 may parse the document and extract text information and structural information of the document. At this time, the document parsing unit 304 may parse the document to extract the actual position value of the document component on the browser. For example, when the document is an HTML web document, the document parsing unit 304 may parse the document composed of HTML using a parser according to a hypertext markup language (HTML) interface.

The image filtering unit 305 may perform image filtering to determine an image, which is an image caption extraction target, among images included in a document. For example, the image filtering unit 305 may perform image filtering to determine an image that is an image caption extraction target by using a filter according to the attributes of the images. The filter according to the attributes of the images may refer to the description of FIG. 2.

The caption candidate determiner 306 may determine a caption candidate for the image that is the image caption extraction target from the text included in the document. In this case, the caption candidate determiner 306 may determine a caption candidate based on unit text adjacent to each direction of the image that is the image caption extraction target.

When caption candidates are determined for an image that is an image caption extraction target, when there is no text that satisfies a predetermined caption candidate determination criterion among texts included in a document, a caption candidate may not be determined for the image. The extraction of the feature from the image-caption candidate pair and the selection of the image caption are not performed on the image for which the caption candidate is not determined.

For example, the caption candidate determiner 306 may extract the unit text by using a tag of the document, and determine the caption candidate using the length of the text and the distance from the image in the unit text. A specific example of determining the caption candidate may refer to FIG. 2.

The feature extractor 302 may extract a feature for each of the generated image-caption candidate pairs. For example, the feature extractor 302 may extract a feature using text information and structural information about the generated image-caption candidate pair. Here, the feature may be an attribute of an image, an attribute of a caption candidate, and an attribute according to a relationship between the image and the caption candidate.

At this time, the feature extraction unit 302 is the image size, image aspect ratio, image format, the length of the caption candidate, whether the caption candidate is repeated, whether the font tag is used, the image and the caption for the generated image-caption candidate pair Features including the distance to the candidate, the caption candidate direction, the width of the caption candidate relative to the image width, whether to use an anchor tag, whether a caption keyword exists or whether there is a termination code may be extracted.

The above-mentioned features are only examples and may be changed according to the configuration of the system. Each of the above-mentioned features is described in detail in FIG. 6.

The image caption selector 303 may select an image caption from the generated image-caption candidate pair using the probability according to the extracted feature. As an example, the image caption selector 303 may calculate a probability value that the image-caption candidate pair will be a correct answer pair using the extracted feature according to a pre-learned probability-based classification model.

In this case, the pre-trained probability-based classification model means learning a process of selecting an image caption from an image-caption candidate pair through training data. In this case, the probability-based classification model is to determine the probability that the caption candidate is an image caption through the probability according to each feature for the image-caption candidate pair, it can be pre-learned through the training material.

The image caption selector 303 may select an image caption candidate by selecting an image-caption candidate pair having the largest probability value from among the generated image-caption candidate pairs having the calculated probability value equal to or greater than a preset threshold value. . Thus, if all of the probability values calculated for the image-caption candidate pair are not greater than or equal to a preset threshold, the image caption for the image may not be selected.

4 illustrates a process of parsing a document according to an embodiment of the present invention.

Referring to FIG. 4, the document 401 may include an image 403 and text 404. As already mentioned above, the document 401 may be a web document in HTML. The present invention may extract an image caption describing an image from text 404 adjacent to image 403. In this case, it may be determined whether the text included in the document 401 is adjacent to the image 403 using the structural information of the document 401.

The document parsing unit 304 may parse the document 401 to extract text information and structural information of the document 401. At this time, the document parsing unit 304 may parse the document to extract the actual position values of the document components (images and texts) on the browser. In this case, the structural information may include an actual position value of the document elements on the web browser.

For example, when the document is an HTML web document, the document parsing unit 304 may parse the document 401 using a parser according to a hypertext markup language (HTML) interface.

The tag data 402 represents the structure of the document 401 through the tag of the document 401. In this case, the document parsing unit 304 may parse the document 401 using the tag data 402 of the document 401.

In one example, image 403 may be parsed via a tag called img (123.jpg in FIG. 4). In addition, the text 404 may be parsed through a <p> tag or a <td> tag that is a table tag (XYZ in FIG. 4). As a result, text information and structural information of the document 401 may be extracted through the document parser 304.

Then, the image filtering unit 305 may perform image filtering to determine the image that the parsed image 403 is an image caption extraction target among images included in the document through a filter based on the image property. The candidate determiner 306 may determine whether the parsed text 404 is unit text capable of being a caption candidate. The image filtering unit 305 and the caption candidate determiner 306 may utilize text information and structural information extracted by parsing the document 401.

5 is a diagram for describing a process of generating an image-caption candidate pair by determining a caption candidate according to an embodiment of the present invention.

The image-caption candidate pair generator 301 may generate an image-caption candidate pair for each of the images included in the document using text information and structural information of the document.

In FIG. 5, the image 501 included in the document 500 refers to an image that has undergone an image filtering process through the image filtering unit 305. For example, the image 501 of FIG. 5 is subjected to an image filtering process through a filter according to an image size, a filter based on an aspect ratio of an image, a filter based on a file name of an image, and a filter based on overlapping arrangement of images in a document. As an image, an image that is an image caption extraction target. That is, the image filtering process may be referred to as a process for determining an image that is an image caption extraction target among images included in a document.

In addition, the texts 502, 503, 504, and 505 included in the document 500 mean unit text adjacent to the image 501. In this case, the caption candidate determiner 306 may extract unit texts 502, 503, 504, and 505 adjacent to each direction by using the tag of the document with respect to the image 501, which is an image caption extraction target. At this time, each direction of the image shown in Figure 5 is up, down, left, right, but the present invention is not limited to the above direction. As mentioned above, the caption candidate determiner 306 may extract the unit texts 502, 503, 504, and 505 using tags of a document including the paragraph tag <p> and the table tag <td>.

The caption candidate determiner 306 may determine a caption candidate based on the extracted unit texts 502, 503, 504, and 505. If the unit texts 502, 503, 504, and 505 do not satisfy the criteria for determining the caption candidate, the caption candidate for the image 501 may not be determined. The extraction of the feature from the image-caption candidate pair and the selection of the image caption are not performed on the image for which the caption candidate is not determined.

In this case, the caption candidate determiner 306 may determine the caption candidate using the length of the text and the distance from the image of the extracted unit text. That is, the criterion for determining the caption candidate may be set by the length of the unit text and the distance from the image. Since the caption candidate is text that may be an image caption describing the image 501, the length of the text may mean that the caption is a length enough to be a descriptive text. The distance from the image may also mean a distance enough to be an explanatory phrase.

For example, when the length of the limited text is 2 bytes to 500 bytes, if the length of the text 2 503 is 600 bytes, the text 2 503 is not determined as the caption candidate. When the limited distance to the image is less than twice the average of the width and height of the image, the distance between the text 3 504 and the image 501 is three times the average of the width and height of the image 501. If not, text 3 504 is not determined to be a caption candidate.

When caption candidates 502 and 505 are determined through the caption candidate determiner 306, an image-caption candidate pair may be generated for the image 501. The generated image-caption candidate pair may be at least one.

Then, the feature extractor 303 may extract a feature for each of the generated image-caption candidate pairs. For example, the extracted feature may include an image size, an image aspect ratio, an image format, a length of a caption candidate, whether there is a repetition of the caption candidate, whether a font tag is used, a distance between the image and the caption candidate, The caption candidate direction, the width of the caption candidate relative to the image width, whether to use an anchor tag, whether a caption keyword exists or whether there is a termination code may be included.

The image size may be determined by the product of the width and length of the image. The image aspect ratio may be determined as the ratio between the width and the length of the image. The image format may mean a file format of an image (jpg, gif, tiff, etc.). The length of the caption candidate may be determined by the length of the caption candidate, which is text adjacent to the image.

The presence or absence of the caption candidate may be determined by whether the caption candidate frequently appears in the document corresponding to the repeated phrase (title, content, creation date, etc.) to describe the image.

In addition, whether the font tag is used means whether effects such as font, size, and thickness of the caption candidate text are applied. The caption candidate direction means in which direction of the image the caption candidate is located. The width of the caption candidate to the image width may be determined as a ratio between the width of the image and the width of the caption candidate.

Whether to use an anchor tag is determined whether a hyperlink to which a caption candidate connects to a specific link has been applied. The presence or absence of a caption keyword is determined by whether or not a keyword (eg, a figure, a scene, a photo, etc.) including a meaning describing the image exists in the caption candidate. The presence or absence of a termination code is determined by whether a termination code such as a period exists at the end of the caption candidate.

6 is a diagram illustrating a process of selecting an image caption from an image-caption candidate pair according to an embodiment of the present invention. Referring to FIG. 6, examples 601 and 602 of selecting an image caption are shown.

The image caption selector 303 may select an image caption from the generated image-caption candidate pair using the probability according to the extracted feature. As an example, the image caption selector 303 may calculate a probability value that the image-caption candidate pair will be a correct answer pair using the extracted feature according to a pre-learned probability-based classification model.

For example, the probability-based classification model may be trained to select an image caption from an image-caption candidate pair through training materials. For example, the image caption selector 303 may calculate a probability value that the image-caption candidate pair will be a correct pair according to Equation 1 below.

The image caption selector 303 may select an image caption by selecting an image-caption candidate pair having the largest probability value from among the extracted image-caption candidate pairs having the calculated probability value equal to or greater than a preset threshold value. For example, the image caption selector 303 may select an image caption according to Equation 2 below.

는 이미지에 대한 이미지 캡션을 의미하고,

는 이미지-캡션 후보 쌍을 의미한다. 그리고,

는 하나의 캡션 후보를 의미하고,

는 캡션 후보의 집합을 의미할 수 있다.

는 이미지-캡션 후보 쌍이 정답 쌍이 될 확률을 의미한다.here,

Means an image caption for the image,

Means an image-caption candidate pair. And,

Means one caption candidate,

May mean a set of caption candidates.

Denotes the probability that the image-caption candidate pair will be a correct pair.

)이 80%이라고 가정할 수 있다. 예제(601)에서, 이미지 1에 대한 캡션 후보가 캡션 후보 1, 캡션 후보 2가 존재할 때, 이미지 1-캡션 후보 1 쌍이 정답 쌍이 될 확률값은 90%이고, 이미지1-캡션 후보 2 쌍이 정답 쌍이 될 확률값은 95%이다. 상기 수학식 2를 적용하면, 선택되는 이미지 캡션은 캡션 후보 2이다.Referring to Figure 6, the threshold value (

) Can be assumed to be 80%. In example 601, when the caption candidate for image 1 is caption candidate 1 and caption candidate 2, there is a 90% probability that the pair of image 1-caption candidates will be the correct pair, and the pair of image 1-caption candidates will be the correct pair The probability value is 95%. Applying Equation 2, the selected image caption is caption candidate 2.

In example 602, when the caption candidate for image 2 is caption candidate 3 and caption candidate 4, the probability value that the pair of image2-caption candidates 3 will be the correct pair is 40%, and the pair of image2-caption candidates will be the correct pair The probability is 75%. In the case of Example 602, the probability value of the four pairs of image2-caption candidates to be the correct pair is the largest, but since there is no probability value that is greater than or equal to the threshold value, it can be said that there is no image caption selected in the example 602.

However, the present invention is not limited to the above-mentioned examples, and various methods may be applied.

7 is a flowchart showing the overall configuration of a caption candidate extraction method according to an embodiment of the present invention.

The caption candidate extraction method according to an embodiment of the present invention may parse the document to extract text information and structural information of the document (S701). In this case, the text information may refer to a text form such as a length, a width, and a font of the text included in the document. In addition, the structural information may mean the location of the components of the document.

At this time, the step of parsing the document to extract the text information and structural information of the document (S701) can parse the document to extract the actual position value of the document component on the browser. For example, parsing a document and extracting text information and structural information of the document (S701) may parse the document composed of HTML using a parser according to a hypertext markup language (HTML) interface. Can be.

The caption candidate extraction method according to an embodiment of the present invention may perform image filtering to determine an image, which is an image caption extraction target, among images included in a document (S702).

Performing image filtering to determine an image that is an image caption extraction target (S702) may perform image filtering to determine an image that is an image caption extraction target using a filter according to attributes of the images. In this case, the filter according to the attributes of the images may include a filter based on the size of the image, a filter based on the aspect ratio of the image, a filter based on the file name of the image, and a filter based on the overlapping arrangement of the images in the document.

That is, when an image included in a document is highly likely to correspond to ancillary components of the document, the caption for the image may be meaningless. Accordingly, the filters according to the attributes of the images may refer to filters for removing an image corresponding to an additional component of the document.

The caption candidate extraction method according to an embodiment of the present invention may determine a caption candidate for an image that is the image caption extraction target from text included in a document (S703).

In this case, the caption candidates may be unit texts adjacent to an image which may become an image caption among texts included in the document. For example, in the determining of the caption candidate (S703), the caption candidate may be determined based on unit text adjacent to each direction of the image that is the image caption extraction target.

In operation S703, the caption candidate may be extracted using the tag of the document, and the caption candidate may be determined using the length of the text in the unit text and the distance from the image.

For example, determining the caption candidate (S703) may extract unit text using a tag for a paragraph or a table in the document. However, a tag for a paragraph or a table corresponding to a lower node of the tag may be excluded.

The determining of the caption candidate may include determining whether the extracted unit text corresponds to a preset length of text and a distance from the preset image (for example, 2 of an average value of the width and height of the image). The caption candidate may be determined in consideration of whether the information is included in the same or less times.

When determining a caption candidate for an image that is an image caption extraction target, and when there is no text among the text included in the document that satisfies a predetermined caption candidate determination criteria (length of unit text and distance from the image), the image The caption candidate may not be determined for.

As a result, an image-caption candidate pair may be generated by determining a caption candidate for an image from text included in a document through a caption candidate determination method.

8 is a flowchart illustrating the overall configuration of an image caption extraction method according to an embodiment of the present invention.

The image caption extraction method according to an embodiment of the present invention may generate an image-caption candidate pair for each of the images included in the document using text information and structural information of the document (S801). That is, in operation S801 of generating an image-caption candidate pair, a caption candidate may be determined for an image of a document, and an image-caption candidate pair may be generated for each image according to the determined caption candidate.

For example, generating an image-caption candidate pair in operation S801 may include parsing a document and extracting text information and structural information of the document. At this time, parsing the document to extract the text information and structural information of the document may parse the document to extract the actual position value of the document component on the browser.

For example, the generating of the image-caption candidate pair in operation S801 may include performing image filtering to determine an image, which is an image caption extraction target, among images included in the document. For example, in operation S801 of generating an image-caption candidate pair, image filtering may be performed to determine an image that is an image caption extraction target by using a filter according to attributes of the images. The filter according to the attributes of the images may refer to the description of FIG. 2.

For example, generating an image-caption candidate pair (S801) may include determining a caption candidate for an image that is an image caption extraction target from text included in a document. The determining of the caption candidate may determine the caption candidate for the unit text adjacent to each direction of the image that is the image caption extraction target. A specific example of determining the caption candidate may refer to FIG. 2.

The image caption extraction method according to an embodiment of the present invention may extract a feature for each of the generated image-caption candidate pairs (S802). Here, the feature may be an attribute of an image, an attribute of a caption candidate, and an attribute according to a relationship between the image and the caption candidate.

In one example, extracting the feature (S802) may include image size, image aspect ratio, image format, length of caption candidate, presence of caption candidate repetition, font tag use, image and caption for image-caption candidate pairs. Features including the distance to the candidate, the caption candidate direction, the width of the caption candidate relative to the image width, whether to use an anchor tag, whether a caption keyword exists or whether there is a termination code may be extracted. The above-mentioned features are only examples and may be changed according to the configuration of the system. See FIG. 6 for each of the above-mentioned features.

The image caption extraction method according to an embodiment of the present invention may select an image caption from an image-caption candidate pair generated using the probability according to the extracted feature (S803).

Selecting an image caption (S803) may calculate a probability value of the image-caption candidate pair to be a correct answer pair using the extracted features according to a pre-learned probability-based classification model.

In this case, the pre-trained probability-based classification model means learning a process of selecting an image caption from an image-caption candidate pair through training data. In this case, the probability-based classification model is to determine the probability that the caption candidate becomes an image caption through the probability of each of the features for the image-caption candidate pair, it can be pre-learned through the training material.

In this case, selecting an image caption (S803) may be performed by selecting an image-caption candidate pair having the largest probability value from among the extracted image-caption candidate pairs having a value greater than or equal to a preset threshold. You can choose. Thus, if all of the probability values calculated for the image-caption candidate pair are not greater than or equal to a preset threshold, the image caption for the image may not be selected.

Parts not described in FIGS. 7 and 8 may refer to the contents described with reference to FIGS. 1 to 6.

In addition, the caption candidate extraction method and the image caption extraction method according to an embodiment of the present invention includes a computer readable medium including program instructions for performing various computer-implemented operations. The computer readable medium may include program instructions, data files, data structures, etc. alone or in combination. The medium or program instructions may be those specially designed and constructed for the purposes of the present invention, or they may be of the kind well-known and available to those having skill in the computer software arts. Examples of computer-readable recording media include magnetic media such as hard disks, floppy disks, and magnetic tape, optical media such as CD-ROMs, DVDs, and magnetic disks, such as floppy disks. Magneto-optical media, and hardware devices specifically configured to store and execute program instructions, such as ROM, RAM, flash memory, and the like. Examples of program instructions include machine code, such as that produced by a compiler, as well as high-level language code that can be executed by a computer using an interpreter.

As described above, the present invention has been described by way of limited embodiments and drawings, but the present invention is not limited to the above-described embodiments, which can be variously modified and modified by those skilled in the art to which the present invention pertains. Modifications are possible. Accordingly, the spirit of the present invention should be understood only by the claims set forth below, and all equivalent or equivalent modifications thereof will belong to the scope of the present invention.

1 is a diagram illustrating a process of extracting an image caption list using an image caption extraction system according to an embodiment of the present invention.

2 is a block diagram showing the overall configuration of a caption candidate extraction system according to an embodiment of the present invention.

3 is a block diagram showing the overall configuration of an image caption extraction system according to an embodiment of the present invention.

4 illustrates a process of parsing a document according to an embodiment of the present invention.

5 is a diagram for describing a process of generating an image-caption candidate pair by determining a caption candidate according to an embodiment of the present invention.

6 is a diagram illustrating a process of selecting an image caption from an image-caption candidate pair according to an embodiment of the present invention.

7 is a flowchart showing the overall configuration of a caption candidate extraction method according to an embodiment of the present invention.

8 is a flowchart illustrating the overall configuration of an image caption extraction method according to an embodiment of the present invention.

<Explanation of symbols for the main parts of the drawings>

100: WEB

101-1 to 101-n: web browser

102-1 through 102-n: HTML Web Documents

103: image caption extraction system

104: image caption list

Claims (28)

An image-caption candidate pair generator for generating an image-caption candidate pair for each of the images included in the document using text information and structural information of the document; A feature extractor for extracting a feature for each of the generated image-caption candidate pairs; And An image caption selector which selects an image caption from the generated image-caption candidate pair using a probability according to the extracted feature Image caption extraction system comprising a. The method of claim 1, The image-caption candidate pair generation unit, A document parsing unit which parses a document and extracts text information and structural information of the document; An image filtering unit configured to perform image filtering to determine an image, which is an image caption extraction target, among images included in the document; And Caption candidate determination unit for determining a caption candidate for the image that is the image caption extraction target from the text included in the document Image caption extraction system comprising a. The method of claim 2, The document parsing unit, Parsing the document to extract the actual position values of document components on a browser. The method of claim 2, The document parsing unit, And parsing the document using a parser. The method of claim 2, The image filtering unit, And image filtering to determine an image that is an image caption extraction target using a filter according to the attributes of the images. The method of claim 5, The filter according to the attributes of the images, An image caption extraction system comprising a filter according to an image size, a filter according to an aspect ratio of an image, a filter based on a file name of an image, and a filter based on overlapping arrangement of images in a document. The method of claim 2, The caption candidate determiner, And caption candidates are determined based on unit text adjacent to each direction with respect to the image that is the image caption extraction target. The method of claim 2, The caption candidate is And image caption extraction unit text adjacent to the image caption extraction target image among the text included in the document. The method of claim 7, wherein The caption candidate determiner, And extracting the unit text by using the tag of the document, and determining a caption candidate using the length of the text and the distance from the image of the unit text. The method of claim 1, The feature extraction unit, Image size, image aspect ratio, image format, length of caption candidate, presence of repetition of caption candidate, use of font tag, distance between image and caption candidate, caption candidate direction, image width for the image-caption candidate pair And extracting a feature including a width of a contrast caption candidate, whether an anchor tag is used, whether a caption keyword exists, or whether a terminator exists. The method of claim 1, The image caption selector, And calculating a probability value of the image-caption candidate pair to be a correct answer pair using the extracted features according to a pre-learned probability-based classification model. The method of claim 11, The image caption selector, And selecting an image caption by selecting an image-caption candidate pair having the largest probability value from among the generated image-caption candidate pairs having the calculated probability value equal to or greater than a preset threshold. Parsing a document to extract textual information and structural information of the document; Performing image filtering to determine an image, which is an image caption extraction target, among images included in the document; And Determining a caption candidate for an image that is an image caption extraction target from text included in the document; Caption candidate extraction method comprising a. The method of claim 13, The caption candidate is Caption candidate extraction method characterized in that the unit of the text contained in the document adjacent to the image caption extraction target image. The method of claim 13, Parsing the document to extract text information and structural information of the document, And parsing the document to extract the actual position value of the document component on the browser. The method of claim 13, Parsing the document to extract text information and structural information of the document, Caption candidate extraction method characterized in parsing the document using a parser. The method of claim 13, Performing image filtering to determine an image that is the image caption extraction target, Caption candidate extraction method characterized in that for performing the image filtering to determine the image that is the image caption extraction target using a filter according to the properties of the images. The method of claim 17, The filter according to the attributes of the images, Caption candidate extraction method comprising a filter according to the size of the image, a filter according to the aspect ratio of the image, a filter according to the file name of the image and a filter according to the overlapping arrangement of the image in the document. The method of claim 13, Determining a caption candidate for the image that is the image caption extraction target, Caption candidate extraction method characterized in that for determining the caption candidate for the unit text adjacent to each direction for the image that is the image caption extraction target. The method of claim 19, Determining a caption candidate for the image that is the image caption extraction target, And extracting the unit text by using the tag of the document, and determining a caption candidate using the length of the text and the distance from the image of the unit text. Generating an image-caption candidate pair for each of the images included in the document using textual information and structural information of the document; Extracting features for each of the generated image-caption candidate pairs; And Selecting an image caption from the generated image-caption candidate pair using a probability according to the extracted feature Image caption extraction method comprising a. The method of claim 21, Generating the image-caption candidate pair, Parsing a document to extract textual information and structural information of the document; Performing image filtering to determine an image, which is an image caption extraction target, among images included in the document; And Determining a caption candidate for an image that is an image caption extraction target from text included in the document; Image caption extraction method comprising a. The method of claim 22, Parsing the document to extract text information and structural information of the document, Parsing the document to extract an actual position value of a document component on a browser. The method of claim 22, Determining a caption candidate for the image that is the image caption extraction target, And caption candidates are determined based on unit text adjacent to each direction with respect to the image caption extraction image. The method of claim 21, Extracting features for each of the generated image-caption candidate pairs may include: Image size, image aspect ratio, image format, length of caption candidate, presence of repetition of caption candidate, use of font tag, distance between image and caption candidate, caption candidate direction, image width for the image-caption candidate pair And extracting a feature including a width of a contrast caption candidate, whether an anchor tag is used, whether a caption keyword exists, or whether a terminator exists. The method of claim 21, Selecting an image caption from the generated image-caption candidate pair, And a probability value of the generated image-caption candidate pair to be a correct answer pair using the extracted features according to a pre-learned probability-based classification model. The method of claim 26, Selecting the image caption, And selecting an image caption by selecting an image-caption candidate pair having the largest probability value from among the extracted image-caption candidate pairs, wherein the calculated probability value is equal to or greater than a preset threshold value. A computer-readable recording medium in which a program for executing the method of any one of claims 13 to 27 is recorded.

Images