JP2012022577A - Image processing apparatus, control method, and computer program - Google Patents

Abstract

PROBLEM TO BE SOLVED: To extract a document area of exact original size by estimating a correct border line between a document and a background even when a natural image includes a plurality of documents whose borders with the background are indistinctive.SOLUTION: An image processing apparatus includes: character area extraction means of extracting a character area which is a circumscribed rectangle of a character included in a document area from input image data; first search range determination means of determining a first search range for searching for a document area end pixel as a document area end based upon the position of the extracted character area; document area end pixel extraction means of extracting the document area end pixel from the first search range; second search range determination means of determining a second search range for searching a document area end line as a document area end based upon the position of the extracted document area end pixel; document area end line extraction means of extracting the document area end line from the second search range; and document area estimation means of determining the document area according to the document area end line.

Description

  The present invention relates to a technique for extracting a region of a document in a natural image shot with a digital camera or the like.

  In recent years, due to the spread of networks, the opportunity for electronic distribution of documents has increased, and along with this, a technology that enables paper documents to be electronically distributed by scanning has become widespread. However, there are documents that are difficult to scan, such as posted posters, whiteboards used in meetings, and large-size imitation paper. In view of this, a technique for electronically documenting images taken with a camera has been considered. In that case, a trapezoidal distortion occurs in the obtained image due to the positional relationship between the camera and the document to be copied, and thus a technique for correcting the distortion is required.

  For example, there is a technique for acquiring an edge from a color difference, detecting a line segment having a certain length or more as a document frame, and correcting distortion (see Patent Document 1). By using this technique, it is possible to detect a document from a captured image and correct distortion. Further, there is a technique for obtaining an adjacent side from a relative position on a captured image plane of a detected line segment candidate when a document on a pedestal is captured (Patent Document 2). There is also a technique for estimating the edge of a document when the edge of the document cannot be extracted because it is hidden behind another object in the photographed document. First, an edge is extracted from the input image, and the approximate position of the document edge is detected. Background color candidates and background color candidates are estimated from the peak values of the histogram generated from the color information of the entire document, and the final document edge is obtained from the estimated background color candidates and background color candidates (Patent Document 3).

JP 2003-058877 A JP 2007-58634 A JP 2004-096435 A

  When a document is photographed with a camera, it is difficult to accurately face the object and the camera, so that the document in the photographed image has a trapezoidal distortion due to a three-dimensional tilt. Therefore, in order to extract a document from a captured image in a form that is easy to read, it is necessary to accurately extract the frame of the document. As a method of extracting a frame, there is a method of detecting a straight line component using Hough transform or the like and estimating a frame from four straight lines.

  However, many straight lines are extracted depending on the background of the document. In particular, when the background color of the document area 301 and the document area 302 and the background of the entire image 300 are approximate as shown in FIG. Must be estimated. In addition, a plurality of documents may be reflected in a natural image. For this reason, there is a problem that the number of straight line combination candidates constituting the frame increases and it is difficult to estimate a correct document frame.

  Further, even if a general area division is performed and only a character area is extracted, only an area smaller than the original document area can be extracted as shown in FIG. For this reason, the document area cannot be extracted with the original size. Furthermore, the document that should be the same is divided in character area units, and there is a problem that it is not possible to extract the document area in accordance with the user's intention.

  In order to solve the above problems, the present invention has the following configuration. An image processing apparatus for extracting the document area from image data having a document area, wherein the character area extracting unit extracts a character area of characters constituting the document area from input image data, and the extracted character First search range determination means for determining a first search range for searching for a document region end pixel serving as a document region end with reference to the position of the region, and extracting the document region end pixel from the first search range And a second search range determining means for determining a second search range for searching for a document area end line serving as a document area end with reference to the position of the extracted document area end pixel. And document area end line extracting means for extracting the document area end line from the second search range, and document area determining means for determining the document area from the document area end line.

  Even if there are a plurality of document areas in which the boundary between the background and the background is not clear in one natural image, the correct boundary line between the document area and the background can be estimated, and the document area can be extracted with the original size.

The example figure of the system configuration concerning the first and second embodiments. The figure of the flowchart which concerns on 1st, 2nd embodiment. The figure which shows an example of the color input image which concerns on 1st, 2nd embodiment. The figure which shows the area | region division result of the color input image of FIG. The block diagram of the detailed function of document area extraction which concerns on 1st embodiment. The figure which showed the 1st search range with respect to the character area 401. FIG. The detailed flowchart of a density | concentration change point extraction part. The figure which showed the document area edge pixel of the character area 401. FIG. The figure which showed the 2nd search range of the character area 401. FIG. The figure which showed the document area | region end line of the character area 401. FIG. The figure which showed the straight line estimated from the document area end line of the character area 401. FIG. The figure which showed the document area of the character area 401. FIG. 6 is a diagram illustrating an example of distortion correction processing and electronic file conversion of document areas 1001 and 1301. FIG. The detailed functional block diagram of the document area extraction which concerns on 2nd embodiment. The detailed flowchart of a character area integration part. The figure which showed the 1st search range with respect to the character area 402, and a document area end pixel. The figure which showed the document area edge pixel with respect to the character areas 402 and 403, the 2nd search range, and the document area edge line. The figure which showed the document area of character area 402,403.

<First embodiment>
FIG. 1 shows an example of a system configuration for carrying out an embodiment of the present invention. The CPU 101 controls the entire system and executes an execution program in which each process is defined. In a RAM (Random Access Memory) 102, processing programs and input / output data are expanded and processed. The storage device 103 stores image data to be processed and processed electronic files. The input device 104 is used for inputting processing data from the outside. The output device 105 is used to output processing data to the outside.

  Image data input from an input device 104 such as a digital camera is stored as input data 1032 in a storage device 103 such as a hard disk. The processing program 1031 stored in the storage device 103 is expanded in a processing program expansion area 1021 on the RAM 102 and executed by the CPU 101. The processing program calls input data from the storage device 103 and expands the input data area 1022 on the RAM 102. The processing program processes the input data, outputs the processing result to the output data area 1023 on the RAM 102, and saves it as output data 1033 in the storage device 103. The output data 1033 is output to the output device 105 such as a display or a printer as necessary.

[Processing flow]
FIG. 2 is a flowchart in the first embodiment of the present invention. Details of the processing procedure will be described with reference to FIG. In S201, the input image is determined. The image determination method classifies a natural image and an image with characters depending on whether characters exist in the input image. Regarding a method for extracting characters from image data, for example, Japanese Patent Laid-Open No. 2002-042055 “Character recognition method from color document” can be used. Note that the present invention is not limited to this method, and other image determination methods may be used as long as the present invention is applicable. Here, the character area refers to an area including one or more characters and is a rectangle including characters. The document area refers to an area including one or more character areas, and some images include a trapezoidal distortion due to an influence of an angle at which the image is captured.

  If there is a character in the image data, the processing from S202 is performed. If no character is present in the image data (in the case of a natural image), the process is terminated. In S202, a document area that is an area including characters in the image data is extracted. A detailed functional configuration of S202 is shown in FIG. An image 501 is an image determined as an image with characters in S201. In the present embodiment, an example of an image with characters is an image 300 in FIG. 3, and the details of the subsequent processing will be described assuming that the image 300 is input.

  As the color input, the image 300 includes a document area 301 in which “ABC” is written and a document area 302 in which “The are some words in this paper” are written. Specifically, a paper memo or the like may be assumed as an example of the document area here. The document area 301 and the document area 302 are placed in a table having a color similar to these two background colors, and the boundary between the document area and the table is not clear.

  A character area extraction unit 502 divides an input image into areas and extracts character areas. An image 501 is input to the character region extraction unit 502. As a specific example of the area dividing process, there is a process described in US Pat. No. 5,680,478. In the above example, a set of black pixel connected components and white pixel components in the document image is extracted, and characteristic areas such as characters, pictures, diagrams, tables, frames, and lines are extracted from the shape, size, set state, etc. is doing. In the present embodiment, a binary image is generated from the image 300 by a known binarization technique, and region segmentation is performed.

  FIG. 4 is a diagram showing the result of dividing the image 300 into regions for each attribute. In this result, the character areas 401, 402, and 403 are divided as areas. The output of the character area extraction unit 502 is a character area, that is, a pixel connected component corresponding to the character and a circumscribed rectangle of the character pixel connected component. The extracted character area is transferred to the document area edge pixel extraction unit 503.

  The document area edge pixel extraction unit 503 detects a document area edge pixel that can be a pixel at the boundary between the document area and the background. The document area edge pixel extraction unit 503 includes a first search range determination unit 504 and a density change point extraction unit 505. The input to the document area edge pixel extracting unit 503 is the image 501 and the character area output from the character area extracting unit 502.

[First search range determination]
First, the first search range determination unit 504 determines a range for searching for a document area end pixel in the image 501. The search range of the document area end pixel is a line segment that is radially drawn from the scanning start point (start point) inside the character area. Here, scanning in general image processing refers to Z scanning or line scanning for scanning all images. In the present embodiment, by setting the scanning direction to be radial, the number of pixels to be searched becomes smaller than when the entire image is scanned, and the processing amount can be reduced. Furthermore, since more document area end pixels can be obtained than by simple vertical and horizontal scanning, it is easier to estimate the document area. Further, by setting the scanning start point as the inner point of the character area, there is an advantage that the target pixel at the start of scanning is surely on the document area, and it is easy to extract a minute density change between the document area and the background.

  In this embodiment, the scanning start point is the center of the circumscribed rectangle of the character area, and the search range is a line segment scanned in 8 directions in 45 degree increments from this center point. Note that the starting point of the scanning line to be searched may be anywhere inside the circumscribed rectangle of the character area. Further, the scanning end point is set to the edge of the image or a pixel that intersects a circumscribed rectangle of another character area.

  FIG. 6 shows a first search range based on the character area 401 included in the document area 301. Scan lines 601 to 608 indicate scan lines that are the first search range. The scanning end points of the scanning lines 601 to 604 and the scanning lines 606 to 608 are image data ends, and the scanning end point of the scanning line 605 intersects the circumscribed rectangle of the character region 402 which is a different character region in the image data. Up to pixels.

  In this embodiment, after all the scanning lines for searching the document region end pixel are first determined, the process proceeds to the density change point extraction unit 505 that performs the next process. However, as soon as one search range is determined, the process may be repeated so that the process proceeds to the density change point extraction unit 505 and another unprocessed search range is determined after the density change point is extracted.

  As described above, in extracting the document area of the character area 401, the description will be continued assuming that the document area end pixels are searched for the eight scanning lines 601 to 608.

[Density change point extraction]
The density change point extraction unit 505 extracts density change points on the scanning line in the input image that is the determined first search range. The density change point extraction unit 505 will be described in detail with reference to the flow of FIG. The density change point extraction unit 505 determines the background color of the document area and refers to the background color and the density change of the pixel on the scanning line to determine whether the target pixel is the document area end pixel.

  First, in S5051, a character pixel connected component close to the pixel that becomes the scanning start point is acquired from the character region, and the process proceeds to S5052. In step S5052, the background color in the document area is determined. Specifically, the peripheral pixels of the acquired character pixel connected component are acquired from the input image 501 and the average value of the color information (pixel value) of the character peripheral pixels is calculated. Then, the calculated color information is stored as the document background color, and the flow shifts to S5053.

  In S5053, the scanning line of the previously determined search range is acquired, and the process proceeds to S5054. In S5054, one pixel on the scanning line is set as a target pixel for density change point determination, and the process proceeds to S5055. In S5055, the character pixel is excluded from the density change determination. The purpose of this processing is to eliminate the pixels of the character itself that is assumed to have a clearly large density change, and to improve the accuracy of extracting density change points of the background and document background color. Specifically, it is determined whether the target pixel corresponds to a character connection pixel in the character area. If YES, the process proceeds to S5058, and if NO, the process proceeds to S5056.

  In step S5056, the document area end pixel is determined. Specifically, the difference between the color information of the target pixel and the previously obtained document background color is obtained, and it is determined whether the difference is larger than the threshold. If YES, the process proceeds to S5057, and if NO, the process proceeds to S5058. Note that a Manhattan distance or the like can be used to calculate a color (pixel value) difference when performing document region edge pixel determination. In step S5057, the target pixel is set as a density change point, and is stored in association with the document region end pixel in the character region, and the flow advances to step S50510.

  In step S5058, the scanning line direction is advanced by one pixel, and the flow advances to step S5059. In S5059, it is determined whether this pixel is within the search range. Specifically, it is determined whether this pixel is an image edge or an intersection with a circumscribed rectangle of another character area. If YES, the process proceeds to S5054, and if NO, the process proceeds to S50510. In S50510, it is determined whether there is a search scanning line that has not been scanned yet. If YES, the process proceeds to S5053, and if NO, the density change point extraction for this character area is terminated.

  The processing result of the character area 401 is shown in FIG. Density change points of the character area 401, that is, document area end pixels 701 to 708 are obtained. The document area edge pixel extraction unit 503 outputs a document area edge pixel associated with one character area, and is delivered to the document area edge line extraction unit 506.

  The document area end line extraction unit 506 detects a document area end line that can be a boundary line between the document and the background. The document area end line extraction unit 506 includes a second search range determination unit 507 and an edge extraction unit 508. Further, the input of the document area end line extraction unit 506 is an image 501 and a document area end pixel associated with the character area output by the document area end pixel extraction unit 503.

[Second search range determination]
First, the second search range determination unit 507 determines a range for searching for a document area end line in the input image 501. The search range for the document area end line is a small rectangle centered on the document area end pixel. In setting the size of the small rectangle, an upper limit size is determined in advance as a predetermined size of the small rectangle. If a character connected component is included in the upper limit size small rectangle, the size of the small rectangle is reduced and adjusted. By making the extraction range of the document area end line a small area centered on the document area edge pixel, it is possible to extract minute density change points of the document area and the background color without generating other extra edges that cause noise. There are advantages.

  FIG. 9 shows a second search range with respect to the character area 401 based on the extracted document area end pixels. Small rectangles 801 to 808 indicate small rectangles in the search range for each of the extracted document region end pixels. In this embodiment, after all the small rectangles for searching the document area end line are first determined, the process proceeds to the edge extraction unit 508 that performs the next process. However, the transition to the edge extraction unit 508 may be performed immediately after one search range is determined, and the determination of another search range may be repeated after edge extraction.

  As described above, in extracting the document area of the document area 301, the description will be continued assuming that the document area end lines are searched for the eight small rectangles 801 to 808.

[Edge Extraction]
The edge extraction unit 508 extracts a weak edge inside the small rectangle in the input image that becomes the determined second search range. The weak edge can be detected using a known method as follows. That is, a pixel corresponding to a boundary portion of a document in an input image is made to stand out by an edge enhancement method using a Sobel filter or a Laplacian filter.

  FIG. 10 shows a weak edge of the document area 301, that is, an end line of the document area. The document area end line extraction unit 506 outputs a document area end line associated with one character area and passes it to the document area determination unit 509.

[Confirm document area]
The document area determination unit 509 determines the document area using the document area end line. A straight line can be detected by using a well-known straight line extraction method such as a Hough transform or a minimum approximation method for a document area end line. If the edge information is only for the second search range, the edge information of other character areas in the same input image will not be mixed, which greatly increases the amount of calculation for determining the combination of lines that determine the boundary of the document area. There is an advantage that can be reduced. FIG. 11 shows a straight line estimated from the document area end line. From this, a closed rectangle is detected and determined as a document area. FIG. 12 shows a document area 1001 for the character area 401 extracted by the above processing. This corresponds to the document area 301 in the input image 300.

  The document area extraction process in S202 of FIG. 2 has been described above. These processes may be executed one by one for each character area existing in the input image until the output of the document area information, or after each character area existing in the input image is processed for each function, the process proceeds to the next function. You may do it.

  The extracted document area is cut out from the input image in S203, distortion correction such as reverse perspective transformation is performed, and the process proceeds to S204. In step S204, the corrected document area is converted into an electronic file. FIG. 13A shows an example in which the document area 1001 for the character area 401 is corrected for distortion and converted into an electronic file.

  As described above, in the present embodiment, in the case where a single natural image includes a document whose boundary with the background is not clear, by limiting the search range of the image edge, other extra noise is generated. Can be suppressed. Therefore, it is possible to extract a weak edge for extracting a boundary where the natural image and the background are not clear. Further, since the number of combinations of document edge lines can be reduced, the amount of calculation can be reduced. Furthermore, since the document boundary is calculated by associating the character area with the document end line, the document area can be extracted even when there are a plurality of documents in the natural image.

<Second Embodiment>
In the first embodiment, an example in which there is one character area for one document area has been described. In the present embodiment, the function of the document area determination unit when a single character area is divided into a plurality of character areas as in the character areas 402 and 403 in FIG. 4 will be described.

  FIG. 14 shows a detailed functional configuration of S202 in FIG. The functional blocks that are the same as the numbers in FIG. 5 are the same as those in the first embodiment, and will not be described. The character area integration unit 5011 in FIG. 14 integrates the character areas from the relationship between the scanning line that is the first search range and the document area end pixels. The input of the character region integration unit 5011 uses the character region output from the document region end pixel extraction unit 503, the document region end pixel associated therewith, and the scanning end point information of the scanning line that is the first search range.

[Character area integration processing]
A detailed flow of the character area integration unit 5011 is shown in FIG. The character area integration unit 5011 determines the end point of the scanning line and whether there is a document area end pixel serving as a boundary on the scanning line, and determines whether the two character areas are included in the same document area. If the end point of the scanning line that is the first scanning range is the end of the image itself, there is no character area to be integrated.

  If the end point of the scanning line in the first scanning range is another character area, the two character areas may be on the same document area. Accordingly, if there is a document area edge pixel on the scanning line located between the two character areas, it can be determined that the document is not an integration target because there is a boundary as a document. From the above, the document area integration condition is that “the end point of the scan line is another character area” and “the end pixel of the document area is not present on the scan line whose end point is another character area”. Become.

  In step S <b> 1101, it is determined whether there is a scanning line from which the document area end pixel has not been extracted for the input character area. If YES in S1101, the process proceeds to S1102, and if NO, the process flow ends. In S1102, it is determined whether or not the end point of the scanning line from which the document area end pixel has not been extracted is another character area. If YES, the process proceeds to S1103, and if NO, the process flow ends.

  In S1103 and S1104, the character area determined to be on the same document area and the document area end pixels associated with the character area are integrated, and this processing flow ends. Note that, since the integration condition is satisfied even if the order of S1101 and S1102 is switched, whichever of the order of S1101 and S1102 may be used.

  FIG. 16 shows the first search range and document area end pixel for the character area 402. Scan lines 1201 to 1208 indicate a search range. The scanning end points of the scanning lines 1201 to 1204 and the scanning lines 1206 to 1208 are image ends, and the scanning end point of the scanning line 1205 is up to a pixel that intersects the circumscribed rectangle of the character area 403. Further, the document area end pixels are extracted in the scanning lines 1201 to 1204 and the scanning lines 1206 to 1208, respectively. However, the scanning line 1205 cannot extract the document area end pixels. For this reason, the character region 402 and the character region 403 are determined as character regions existing in the same document region, and the two document region end pixels are integrated.

  FIG. 17 shows the integrated document region end pixel output from the character region integration unit 5011, the small rectangle that is the second search range, and the document region end line estimated in the document region determination unit 509. The second search range determination unit 507 of the present embodiment has a function of enlarging the size into a rectangle that includes these when the document region end pixels are closer than a certain distance. The small rectangle 1301 and the small rectangle 1306 in FIG. 17 are determined to be close to the document region end pixel, and are larger than the size of the small rectangle.

  A character area 1401 in FIG. 18 indicates a document area including character areas 402 and 403. FIG. 13B shows an electronic file obtained by performing distortion correction on this document area. The character area 1401 corresponds to the document area 302 in FIG.

  As described above, even if characters exist apart from each other in the same document and are divided when the character region is extracted, the character regions can be integrated and extracted as a single document region. In addition, by integrating character areas, more lines can be used to determine the document area, so that the document area can be extracted more reliably. Further, the amount of calculation can be reduced because the number of areas for performing the document area determination process can be reduced.

<Other embodiments>
The present invention can also be realized by executing the following processing. That is, software (program) that realizes the functions of the above-described embodiments is supplied to a system or apparatus via a network or various storage media, and a computer (or CPU, MPU, or the like) of the system or apparatus reads the program. It is a process to be executed.

Claims (8)

An image processing apparatus for extracting a document area from image data having a document area,
A character region extraction means for extracting a character region that is a circumscribed rectangle of characters included in the document region from the input image data;
First search range determining means for determining a first search range for searching for a document region end pixel serving as a document region end with reference to the position of the extracted character region;
Document area edge pixel extracting means for extracting the document area edge pixel from the first search range;
Second search range determining means for determining a second search range for searching for a document area end line serving as a document area end with reference to the position of the extracted document area end pixel;
Document area end line extracting means for extracting the document area end line from the second search range;
An image processing apparatus comprising: a document area determining unit that determines the document area from the document area end line.   The first search range determination means scans in a plurality of directions starting from an inner point of the character area that is a circumscribed rectangle and going outward from the circumscribed rectangle, and the first search range is determined from the inner point. The image processing apparatus according to claim 1, wherein the image processing apparatus includes an image end or an intersection with a different character area in the image.   2. The image according to claim 1, wherein the second search range determination unit sets a small rectangle having a predetermined size centered on the extracted pixel in the document area as the second search range. Processing equipment. The document area edge pixel extracting means includes:
A background color determining means for determining a pixel value to be a background color of the document area from a pixel value of a peripheral pixel of a character connected component adjacent to a pixel that is a search start point of the document area edge pixel;
Document area end pixel determining means for scanning the first search range and determining whether the pixel is the document area end pixel using the pixel value of the target pixel and the pixel value as the background color. The image processing apparatus according to claim 1, wherein:   If it is determined from the determined first search range and the document area end pixel whether a plurality of character areas are included in the same document area, and the plurality of character areas are determined to be included in the same document area, The image processing apparatus according to claim 1, further comprising: a character area integrating unit that integrates the character areas.   In the first search range located between the plurality of character regions, when the document region end pixel is not extracted, the character region integration unit includes the plurality of character regions included in the same document region. The image processing apparatus according to claim 5, wherein the determination is performed. A control method of an image processing apparatus for extracting the document area from image data having a document area,
A character region extraction step in which the character region extraction means of the image processing device extracts a character region that is a circumscribed rectangle of a character included in the document region from the input image data;
A first search for determining a first search range in which a first search range determining unit of the image processing apparatus searches for a document region end pixel serving as a document region end with reference to the extracted position of the character region. A range determination process;
A document area edge pixel extracting step in which the document area edge pixel extracting means of the image processing apparatus extracts the document area edge pixel from the first search range;
Second search range determining means of the image processing apparatus determines a second search range for searching for a document area end line serving as a document area end with reference to the extracted position of the document area end pixel. A search range determination step of
A document area end line extracting means for extracting the document area end line from the second search range;
A control method comprising: a document region determining step of determining the document region from the document region end line, wherein the document region determining means of the image processing apparatus. Computer
A character area extracting means for extracting a character area that is a circumscribed rectangle of a character included in the document area from the input image data;
First search range determining means for determining a first search range for searching for a document region end pixel serving as a document region end with reference to the position of the extracted character region;
Document area edge pixel extracting means for extracting the document area edge pixel from the first search range;
Second search range determining means for determining a second search range for searching for a document area end line serving as a document area end with reference to the position of the extracted document area end pixel;
Document area end line extracting means for extracting the document area end line from the second search range;
A computer program for functioning as document area determining means for determining the document area from the document area end line.

Images