KR20110087620A - Layout based page recognition method for printed medium - Google Patents

Abstract

PURPOSE: A printing medium page recognition method based on a layout is provided to apply an algorithm according to an area through layout information of the printed page. CONSTITUTION: The layout of an image is analyzed(S10). A feature point is extracted from a picture area according to an analysis result(S20). A descriptor of the picture area is generated by using the extracted feature point(S40). The feature point is extracted from a character area according to the analyzed result(S30).

Description

Layout Based Page Recognition Method for Printed Medium}

The present invention relates to a method for recognizing a page of a printing medium to which augmented reality is applied. More specifically, the present invention relates to a layout-based digitlog-book page recognition method for effectively using a digitlog-book that combines analog books and digital contents to allow users to experience real books and multimedia contents simultaneously. will be.

In general, an e-book refers to a digital book that is produced through XML, digital images, multimedia, etc., rather than a book printed on paper, and viewed through terminal screens such as computers, PDAs, and mobile phones. Regulations defined by the Korea Electronic Publishing Association (KISA) require that electronic books be stored in digital data on electronic book recording media or storage devices such as CD-ROMs or DVDs. 'You can see and hear the contents using a computer or a mobile terminal via'.

Recently, various e-books that can be used as terminals such as computers, mobile phones, and PMPs have been developed. First-generation e-books such as PDF and XML, and second-generation e-books that combine multimedia such as Flash have been actively developed and have been spotlighted as a means of replacing existing paper books. However, contrary to the initial expectation that these e-books will completely replace the existing paper books, the use of paper books is increasing. The reason is that digital e-books have advantages such as convenience of collection and convenience of movement, but they cannot replace various advantages of existing analog books (paper books).

Meanwhile, with the development of Augmented / Mixed Reality, which is being studied based on technologies such as computer graphics and computer vision, there have been attempts to produce various books combining paper books and digital contents by applying these technologies. . Japan's Rekimoto introduced an augmented reality system using a two-dimensional matrix code in 1998 (JunRekimoto, "Matric: A Realtime Object Identification and Registration Method for Augmented Reality," Asia Pacific Computer Human Interaction (APCHI 98), pp. 63-68, 1998). In 2001, Bill Billinghurst and Kato introduced the first Magic Book in the form of a book (Mark Billinghurst, Hero Kato, Ivan poupyrev, "The Magic Book: A Transitional AR Interface," IEEE Computers and Graphics, pp. 745-753, 2001). Later, Saso presented a magic book called "Little Red" (enji Lguchi, Masa Inakage, Tomoki Saso, "Little Red MR: Storytelling in Mixed Reality," SIGGRAPH 2003), and Shibata. ) Showed "Vivid Encyclopedia" (Shibata Fumihisa, Yoshida Yusuke, Furuno Koki, Sakai Toshio, Kiguchi Kenji, Kimura Asako, Tamura Hideyuki, "Vivide Encyclopidea: MR Pictorial Book of Insects", Vitual Reality Society of Japan Annual Conference, pp 611-612, 2004).

The term "digilog-book" used in the present invention refers to a publication (paper book) in an augmented reality environment by incorporating multimedia content that stimulates human vision, hearing, and tactile sense to provide additional information that cannot be provided in a paper book. Means the book to offer. The Digilog-Book has a presence, which is an advantage of paper books, and a multimedia book that can be used as an advantage of electronic books. In addition, information that cannot be obtained from a paper book can be obtained through various multimedia contents, and various multimedia scenarios according to the purpose and situation can be combined with the same paper book, so that a personalized digital log book can be produced. Can be.

This digitlog-book provides visual information using augmented reality technology. Therefore, page recognition and tracking techniques are the most important part of the digitlog-book. In the existing digitlog-book, a marker and texture-based natural feature matching method is mainly used to recognize a page.

Marker-based techniques are robust to page recognition and tracking, but there is a problem in that the content is not augmented when a part of the marker is hidden as well as a marker having a specific pattern attached to the printed publication. In addition, it is difficult to apply to a book having a large amount of pages due to the limitation of the marker pattern that can be implemented.

In order to solve this problem, a technique that applies recognition and tracking method based on natural feature points in the picture area is introduced. However, the page recognition technique based on the feature points of the texture has a problem in that it is difficult to apply to pages composed of letters only because of the high similarity of each feature descriptor. In particular, in the case of LLAH (Locally Likely Arrangement Hashing) using geometric information between feature points, there is a problem that the repeatability is poor in the picture area.

In view of the above-mentioned problem, the present invention provides a method of applying a suitable algorithm according to an area by utilizing layout information of a page printed on a digit log book.

The present invention also provides a method of recognizing a page by dividing a page into a text area and a picture area according to the layout of a page of a digitlog-book and matching a descriptor generated from each area with a predetermined descriptor. It is a technical problem.

However, the technical problem of the present invention is not limited to the above-mentioned matters, and other objects not mentioned will be clearly understood by those skilled in the art from the following description.

In order to achieve the above technical problem, a method of recognizing a page of a Digilog-Book according to the present invention includes: (a) detecting a margin of an acquired image, and (b) based on the detected margin Dividing the image into at least one block, and (c) dividing the divided block into blocks of a text area and / or blocks of a picture area.

Here, the step (a), (a1) changing the obtained image to a binary image (binary) image and (a2) detecting the horizontal margin and vertical margin of the binarized image changed in the step (a1) It may include a step.

In the step (b), (b1) synthesizing the horizontal and vertical margins detected in the step (a2) and (b2) dividing the image into blocks based on the margin, And removing the vertical margin if the vertical margin of the divided block is less than or equal to a predetermined size to form one block in the direction.

Also preferably, the step (a1) may remove the noise of the binarized image using a Gaussian filter.

In the step (c), when the histogram of the image of the block is obtained and the vertex value detected in the histogram is equal to or greater than a predetermined reference value, the block is recognized as a block of the text area, and the vertex value is previously determined. If it is less than the determined reference value, the block may be recognized as a block of the picture area.

Meanwhile, in order to achieve the above technical problem, the method of recognizing a page of a digit log-book according to the present invention includes: (a) analyzing a layout of an image obtained from a page and separating the image into a picture area and / or a text area; (b) extracting feature points of the separated picture area and / or text area, and (c) matching descriptors generated using the extracted feature points with predetermined descriptors.

In the step (a), (a1) detecting the margin of the obtained image and dividing it into at least one block; and (a2) obtaining a histogram of the image of the block to obtain a vertex value detected in the histogram. When the reference value is greater than or equal to the predetermined reference value, the block is divided into blocks of the text area. When the vertex value is less than the predetermined reference value, the block is preferably divided into blocks of the picture area.

Here, in step (a1), (a11) changing the obtained image to a binary image (binary) and (a12) detecting a horizontal margin and a vertical margin of the binarized image changed in step (a11). It may also include a step.

Also preferably, in the step (b), the feature points of the picture area may be extracted by using a corner from the image of the picture area, and the feature points of the text area may be extracted by using a midpoint of a word included in the text area. You will be able to extract it.

In addition, the feature points extracted in step (b) are preferably feature points irrelevant to the change in the size of the image.

The step (c) may include: (c1) calculating first results of matching the first descriptors generated from the feature points of the picture area among the descriptors and the predetermined descriptors, and (c2) the descriptors. Calculating second results matched by the second descriptors generated by the feature points of the character region and the predetermined descriptors, and (c3) combining the first results with the second results. Do.

Further, in the step (c3), it is more preferable to generate a matching set by synthesizing the first results and the second results alternately.

Meanwhile, the page recognition method of the digitlog-book according to the present invention may include various computer readable media including program instructions for performing computer-implemented operations.

According to the present invention grasped from the contents described in the present specification, since the page is recognized by applying a suitable algorithm according to the area by utilizing the layout information of the printed page, it can be applied without modifying a previously published book or document.

In addition, the present invention provides a method for recognizing a page of a digit log-book based on layout tracking, which can be applied to any text area and a picture area printed on a page.

1 is a flowchart illustrating a layout analysis method for page recognition of a print medium according to an embodiment of the present invention;
2 is a view illustrating a process of dividing an image into blocks according to an embodiment of the present invention;
3 is a view showing a state in which an image is formed into a binarized image according to an embodiment of the present invention;
4 is a view illustrating a process of detecting a margin of an image according to an embodiment of the present invention;
5 and 6 illustrate histograms of blocks obtained in accordance with one embodiment of the present invention;
7 is a diagram illustrating a state in which an image is divided into blocks according to an embodiment of the present invention;
8 is a flowchart illustrating a page recognition method according to an embodiment of the present invention;
9 is a view illustrating a feature point of an image is extracted according to an embodiment of the present invention;
FIG. 10 is a diagram illustrating a matching result according to each area of a digit log-book; FIG.
FIG. 11 is a diagram for comparing a page matching rate of a digitlog-book including a picture area and a text area according to the present invention with a matching rate when a conventional method is used.

Hereinafter, exemplary embodiments of the present invention will be described in detail with reference to the accompanying drawings. In the description herein, when a component is described as being connected to another component, this means that the component may be directly connected to another component or an intervening third component may be interposed therebetween. First, in adding reference numerals to the components of each drawing, it should be noted that the same reference numerals are assigned to the same components as much as possible, even if shown on different drawings. At this time, the configuration and operation of the present invention shown in the drawings and described by it will be described as at least one embodiment, by which the technical spirit of the present invention and its core configuration and operation is not limited.

Before describing the method for recognizing a page of a digitlog-book in the present invention, an outline of the augmented reality technology underlying the present invention will be described.

Augmented reality systems generally include registration, tracking, and camera calibration. Registration refers to a process of registering a position to augment a virtual object and a virtual object to be augmented in a virtual environment. Typically, for registration, a two-dimensional marker, a three-dimensional image or a two-dimensional image representing a virtual object is placed in a real environment and then identified.

In general terms, three-dimensional coordinates of a point in the real world are required for registration, and the coordinates may be calculated as camera coordinates based on a camera. In order to obtain these three-dimensional coordinates, camera parameters for calculating the exact correlation between the singularity obtained from the camera input image and the camera should be obtained. This process is called camera calibration. Camera calibration creates a camera model using internal variables, such as the focal length of the camera, and external variables calculated by detecting the position of the Digilog-Book. Next, singular points are obtained from the camera input image and the three-dimensional coordinates of the singular points are tracked. This is called tracking, and it is possible to augment a virtual graphic object even in a video input of a moving camera by tracking a position every image frame. Meanwhile, in order to calculate an accurate correlation between the singularity obtained from the camera input image and the camera, camera parameters should be obtained.

However, it should be noted that since the present invention relates to a page recognition method of a digitlog-book, the above-described camera correction is not described separately. In addition, the present invention is not limited to a particular type of print medium that is the object of page recognition, it should be noted that the following description will be described as a digit-book.

1 is a flowchart illustrating a layout analysis method for recognizing a page of a print medium according to an embodiment of the present invention.

As shown in FIG. 1, the method of analyzing a layout of a print medium may include obtaining an image printed on a digit log book using a camera (S11), and changing the obtained image into a binary image ( S12), detecting the vertical and horizontal margins of the image (S13), dividing the image into blocks using the margin detection result (S14), and detecting the histogram of each block (S15). do.

Step S11 is a step of acquiring an image by photographing a page printed on a real digit log-book by a camera, and by an image acquisition device (usually a camera) disposed around a display device in which a page content of a digit log-book is represented. Acquire an image.

Step S12 is a step of converting the acquired image into a binarized image, where binarization is a process of determining a certain threshold value and expressing it as 1 when the brightness value of the pixel is smaller than the threshold value and as large as 0. And controlled. Here, it is possible to remove noise by using a Gaussian filter on the binarized image.

3 is a diagram illustrating a state in which an image obtained as a binarization image is formed according to an embodiment of the present invention. As shown in FIG. 3, in operation S12, the acquired image A is converted into a binarized image B in operation S12.

Step S13 detects the vertical and horizontal margins existing in the image and detects the margin using the binarized image. To describe this in more detail, reference is made to FIGS. 2 and 4. 2 is a diagram illustrating a process of dividing an image into blocks according to one embodiment of the present invention, and FIG. 4 is a diagram illustrating a process of detecting a margin of an image according to an embodiment of the present invention. Drawing.

As shown in FIG. 2, the binarized image A is represented in binary by comparing the brightness value and the threshold value of the pixel. Therefore, as described above, the portion indicated by 0 in the binarization image A corresponds to the margin. In step S13, horizontal margin detection B and vertical margin detection C are performed on the binarized image A. FIG. As shown in the example shown in FIG. 2, the horizontal margin detection B is displayed as 1 when there are less than 4 consecutive zeros in the horizontal direction of the binarization image A. FIG. In addition, the vertical margin detection C is indicated as 1 when there are less than two consecutive zeros in the vertical direction of the binarized image A. FIG. Here, Constraint Run Length Algorithm (CRLA) is applied as an algorithm for detecting vertical and horizontal margins. The result of detecting the margin of the image using the CRLA is shown in FIG. 4. In Figure 4 (A) is a result of detecting the horizontal margin, (B) is a result of detecting the vertical margin, (C) is a result of combining (A) and (B) and (D) is a horizontal direction To indicate that only one block is formed, when the margin existing in the horizontal direction in (C) is less than or equal to the predetermined size, it is determined that it is not a margin.

Next, the horizontal margin detection result and the vertical margin detection result are synthesized (see D of FIG. 2). Here, as shown in FIG. 2, 0 representing the margin is prioritized to 1 representing the content of the image. After that, in the obtained image, a small space between words and words existing in the text area is removed so that only one block can be formed in the text area in the horizontal direction based on the space. In other words, in the binary array included in the block of the character area, 0 arranged horizontally is less than a certain number in a row as 1 to remove the margin (E). In the example illustrated in FIG. 2, the process of removing margins based on the number of consecutive zeros in the horizontal direction of the character area has been described. However, the removal of the horizontal margin of the character area is determined by the width and height of the predetermined pixel. The determined size can be used. In FIG. 4D, a small margin formed in the block existing in the horizontal direction in (C) is removed by the above-described method.

In step S14, the image is divided into blocks based on the margin detected in step S13. In step S14, the image is divided into blocks in advance to separate the image into a picture area and a text area.

In step S15, a histogram for each block is detected to determine whether the divided block corresponds to a block of the text area or a block of the picture area.

That is, here, the normalized histogram of each block is used to determine whether the block is a block of a picture area or a block of a character area.

This will be described in detail with reference to FIGS. 5 and 6.

5 and 6 are histograms of blocks obtained according to an embodiment of the present invention. As shown in Fig. 5, the histogram (B) of block (A) of the text area is one dominant because characters in a page printed in a digitlog-book typically show brightness differences that are clearly distinguished from their background. It has a dominant peak. On the other hand, when the block is a block of the picture region, as shown in FIG. 6, the histogram B of the block A of the picture region does not have a dominant vertex value as shown in FIG. 5.

As described above, the present invention analyzes the layout of the image obtained from the digitlog-book, divides the image area into the text area, extracts the feature points of each area, and matches the predetermined descriptor to recognize the page. A process of generating a descriptor from feature points of each region and matching the generated descriptor with a predetermined descriptor will be described below.

Hereinafter, a method of recognizing a page by extracting feature points of each region will be described.

8 is a flowchart illustrating a page recognition method according to an embodiment of the present invention.

As shown in FIG. 8, the page recognition method includes analyzing a layout of an image (S10), extracting feature points from a picture area according to the analysis result of step S10 (S20), and extracting feature points extracted from a picture area. Generating a descriptor of the picture area using the feature point extracted from the text area (S30), extracting a feature point from the text area according to the analysis result of step S10, and generating a descriptor of the text area using the feature points extracted from the text area (S40). And performing matching for each region with a descriptor of an image which is predetermined and stored in a database by using the descriptor of the picture region and the descriptor of the character region. Here, steps S20 and S30 will be described as first executing step S20 for convenience of description, but it should be noted that the implementation of the present invention may change the order of steps S20 and S30 or simultaneously execute them.

Step S10 is a step of analyzing the layout of the image of the digit log-book obtained by using a camera and dividing it into a picture area and / or a text area. Since the description thereof is overlapped with the above, it will be omitted here.

Step S20 is a step of extracting feature points of the picture area and step S30 is a step of extracting feature points of the text area. This will be described with reference to 9. 9 is a view illustrating a feature point of an image is extracted according to an embodiment of the present invention. In FIG. 9, the feature points extracted from the picture area are represented by blue dots, and the feature points extracted from the text area are represented by red dots. In this case, it should be noted that, in the case of a page composed of only a text area or only a picture area, only feature points corresponding to the corresponding area may be extracted according to the page characteristics of the digit-book.

Here, SURF (Speeded Up Robust Features) can be used for the picture area, and it is preferable to extract FAST corners by applying FAST corner detection to shorten the processing time.

SURF is an image detector or descriptor used in object recognition and three-dimensional design in the field of computer vision.

FAST corner detection is also a method of detecting corners of an image by detecting feature points within a computer vision system. For example, the maximum degree of density in an image detects a feature point using a minimum of isolated points, an ending portion of a curved line, and extracts a point where two contours intersect each other in the image area.

In the character area, the feature point is extracted from the center point of each word. The extracted feature points can be named as LLAH (Locally Likely Arrangement Hashing) feature points. In this case, LLAH is a type of geometric hashing based on the arrangement of feature points for the accuracy of geometric search and the reduction of processing time.

Extraction of the LLAH feature points converts the image of the character region obtained in step S10 into a binarized image and removes noise by applying a Gaussian filter thereto. Each word is then formed into separate blocks and the center point of the block is extracted as the LLAH feature.

In step S40, the descriptors of the drawing region are generated using the extracted feature points of the drawing region. The descriptor of the picture region is a descriptor generated corresponding to the feature points of the picture region and is a SURF descriptor (first descriptor).

In step S50, the descriptors of the text area are generated using the extracted feature points of the text area. The descriptors of the character area are descriptors generated corresponding to the feature points of the character area and are LLAH descriptors (second descriptors).

In step S60, the first descriptors and the second descriptors generated in steps S40 and S50 are matched with descriptors of a database in which images of respective pages of the digit log book are stored in advance.

In more detail, the first descriptors are matched with predetermined descriptors in the database to produce first results. Similarly for the second descriptors, the second results are calculated as a result of matching the descriptors in the database. Next, the first results and the second results are synthesized to recognize the page stored in the database. Here, when synthesizing the first results and the second results, one matching set is obtained by alternately synthesizing any first result included in the first results and any second result included in the second results. )

That is, the algorithm used in the present invention eliminates outliers that fail pose estimation and matching based on homography, and thus obtains an improved homography. Normally, four points are required to calculate homography, where the more homogeneous the four points cover, the better the homography. To this end, if a match is brought one by one in the text area and the picture area, the probability of covering a wide area is increased, and thus, the first and second results are alternately synthesized in the present invention.

In addition, here, the first and second results of each area are not randomly selected one by one, but are arranged by applying a different score function to each matching set of the character area and the picture area. The priority is to bring a match with quality.

According to this method, a higher matching rate is achieved because the pages are recognized by combining the results generated in different areas rather than the pages of the digit log book stored in the database using only the results generated in one area. do.

FIG. 10 is a diagram illustrating a matching result according to each area of a digit log-book. In FIG. 10, (A) shows a matching result of the layout-based page recognition method according to the present invention, (B) shows a matching result using only LLAH matching, and (C) shows a matching result using only SURF matching. Here, the green line is the line showing the matching result, and the red line is the line showing the non-matching result. The matching result (B) by the method using only LLAH matching shows the relatively accurate matching result in the text area. Does not match correctly. In addition, the matching result C by the method using only SURF matching shows a relatively accurate matching result in the picture area, but does not accurately match in the text area. Therefore, when the LLAH matching is used to match the character area and the SURF matching is used in the picture area (A), the matching ratio of the digit log-book consisting of the text area and the picture area is improved. According to the present invention, a high matching ratio is achieved by separating an image acquired through an image acquisition device into a picture region, a text region, or an image having both a picture region and a text region at an early stage, and using a matching method that is most suitable for each region. .

FIG. 11 is a diagram for comparing a page matching rate of a digitlog-book including a picture area and a text area according to the present invention with a matching rate when a conventional method is used.

As shown in FIG. 11, the image area and the text area are separated and separated according to the present invention, rather than the case of recognizing a page of a digit log-book consisting of the picture area and the text area by applying only the SURF method or the LLAH method. Higher matching rates are achieved when different matching techniques are applied. In particular, as the degree of inclination of the digit-book increases, it can be seen that the layout-based page recognition method according to the present invention has a relatively high matching rate.

Meanwhile, the method of recognizing a page of a digitlog-book according to the present invention includes various computer readable media including program instructions for performing a computer-implemented operation. The computer readable medium may include program instructions, data files, data structures, etc. alone or in combination. The media may be program instructions that are specially designed and constructed for the present invention or may be available to those skilled in the art of computer software. Examples of computer readable recording media include magnetic media such as hard disks, floppy disks and magnetic tape, optical media such as CD-ROMs and DVDs, and magnetic disks such as floppy disks. Magneto-optical media, and hardware devices specifically configured to store and execute program instructions, such as ROM, RAM, flash memory, and the like. Examples of program instructions include not only machine code generated by a compiler, but also high-level language code that can be executed by a computer using an interpreter or the like.

As described above, the present invention has been described by way of limited embodiments and drawings, but the present invention is not limited to the above-described embodiments, which can be variously modified and modified by those skilled in the art to which the present invention pertains. Modifications are possible. Accordingly, it is intended that the scope of the invention be defined solely by the claims appended hereto, and that all equivalents or equivalent variations thereof fall within the spirit and scope of the invention.

Claims (13)

In the page recognition method of the print media,
(a) detecting the margins of the acquired image;
(b) dividing the image into at least one block based on the detected margins; And
and (c) dividing the divided block into blocks of a text area and / or blocks of a picture area. According to claim 1, wherein step (a),
(a1) changing the obtained image to a binary image; And
and (a2) detecting the horizontal and vertical margins of the binarized image changed in step (a1). The method of claim 2, wherein step (b) comprises:
(b1) synthesizing the horizontal and vertical margins detected in step (a2); And
(b2) dividing the image into blocks based on the margin, and removing the horizontal margin when the horizontal margin of the divided image is equal to or smaller than a predetermined size to form one block in the horizontal direction of the image Page recognition method comprising a. The method according to any one of claims 1 to 3, wherein step (c) comprises:
A histogram of an image of the block is obtained, and when the vertex value detected in the histogram is greater than or equal to a predetermined reference value, the block is recognized as a block of a text area. When the vertex value is less than a predetermined reference value, the block is illustrated. A page recognition method, characterized in that the recognition as a block of the area. In the page recognition method of the print media,
(a) analyzing a layout of an image obtained from a page and separating the image into a picture area and / or a text area;
(b) extracting feature points of the separated picture area and / or text area; And
and (c) matching descriptors generated using the extracted feature points with predetermined descriptors. The method of claim 5, wherein step (a) comprises:
(a1) detecting and dividing the margin of the obtained image into at least one block; And
(a2) if the vertex value detected in the histogram is equal to or greater than a predetermined reference value by acquiring a histogram of the image of the block, and if the vertex value is less than the predetermined reference value, the block is divided into A page recognition method comprising dividing into blocks of a picture area. The method of claim 6, wherein step (a1),
(a11) changing the obtained image to a binary image; And
(a12) a page recognition method comprising detecting a horizontal margin and a vertical margin of the binarized image changed in step (a11). The method of claim 5, wherein step (b) comprises:
Extracting feature points of the picture area by using a corner of an image of the picture area, and extracting feature points of the text area by using an intermediate point of a word included in the text area. The method of claim 5,
And the feature points extracted in step (b) are feature points irrelevant to a change in the size of the image. The method of claim 5, wherein step (c) comprises:
(c1) calculating first results of matching the first descriptors generated by the feature points of the picture area among the descriptors and the predetermined descriptors;
(c2) calculating second results of matching the second descriptors generated from the feature points of the character area among the descriptors and the predetermined descriptors; And
(c3) synthesizing the first results and the second results. The method of claim 10,
In step (c3), the first result and the second result are synthesized alternately to generate a matching set. The method of claim 11,
In the step (c3), the first results and the second results are synthesized using a score function. A computer-readable recording medium having recorded thereon a program for executing the page recognition method according to any one of claims 5 to 12.

Images