JP5563390B2 - Image processing apparatus, control method therefor, and program - Google Patents

Description

  The present invention relates to an image processing apparatus, a control method, and a program, and more particularly to preprocessing in document image processing.

  In recent years, due to the spread of networks, the opportunity for electronic distribution of documents has increased, and along with this, a technology that enables paper documents to be electronically distributed by scanning has become widespread. However, there are subjects (documents) that are difficult to scan, such as posted posters, whiteboards used in meetings, and large-size imitation paper. In view of this, techniques for electronically documenting images taken with a camera have been developed. Here, since a trapezoidal distortion occurs in the image obtained by the positional relationship between the camera and the document, a technique for correcting the distortion is required.

  For example, there is a technique of acquiring an edge from a color difference, detecting a line segment having a certain length or more as a document frame, and correcting distortion (see Patent Document 1). In addition, there is a technique for obtaining an adjacent side from a relative position on a photographed image plane of a detected line segment candidate when a document on a pedestal is photographed (see Patent Document 2, etc.).

JP 2003-058877 A JP 2007-58634 A

  When a rectangular object such as a document or a whiteboard is photographed by a camera, it is difficult to accurately face the object and the camera. Therefore, the document in the photographed image has a three-dimensional inclination. Shape distortion occurs. Therefore, in order to extract a document (whiteboard) from a captured image in a form that is easy to read, it is necessary to accurately extract a document frame (whiteboard frame). As a method of extracting a document frame, there is a method of detecting a straight line component using Hough transform or the like and estimating a document frame from four straight lines. However, depending on the background of the document, many straight lines are extracted, and the number of combinations constituting the document frame increases, which makes it difficult to estimate the correct document frame.

In order to solve the above problems, the present invention has the following configuration. That is, an image processing apparatus that extracts a frame composed of four sides of the rectangular area from image data obtained by photographing a subject having a rectangular area, wherein a plurality of linear components are extracted from the input image data. A detecting means for detecting; a calculating means for calculating a gradient direction according to the level of pixel information in a direction orthogonal to the linear component detected by the detecting means; and a plurality of linear components detected by the detecting means. select four sides from in an extraction means for extracting one or a frame candidate consisting of the four sides that are selected, among the frame candidate extracted by the extraction means, gradient direction in the four sides of the frame candidate the frame Narrowing means for excluding frame candidates that are not in the same orientation with respect to either the inside or outside direction from the frame candidates extracted by the extracting means .

  Document frame candidates can be accurately narrowed down from a document image having a complicated background in which many straight lines are detected.

FIG. 3 is a diagram illustrating an example of an input image acquisition environment according to the first embodiment. FIG. 4 is a diagram illustrating an example of an input image according to the first embodiment. FIG. 3 is a diagram illustrating a configuration example of the first embodiment. FIG. 6 is a flowchart of document area extraction in the first embodiment. FIG. 3 is a block diagram showing the operation of the first embodiment. FIG. 6 is a diagram illustrating an example of straight line detection processing in the first embodiment. FIG. 6 is a diagram illustrating an example of a gradient direction calculation process in the first embodiment. FIG. 6 is a diagram illustrating an example of an opposite side candidate creation process in the first embodiment. FIG. 6 is a diagram illustrating an example of document frame coordinate calculation processing according to the first embodiment. FIG. 10 is a diagram illustrating an example of a document frame candidate extraction process result according to the first embodiment. FIG. 6 is a diagram illustrating an example of a document frame correction result according to the first embodiment. FIG. 10 is a flowchart of document area extraction in the second embodiment. FIG. 10 is a diagram illustrating an example of a vertical line / horizontal line discrimination processing result according to the second embodiment.

<Embodiment 1>
Hereinafter, preferred embodiments of the present invention will be described in detail with reference to the drawings. FIG. 1 is a diagram illustrating an environment in which an image input to an image processing apparatus according to an embodiment of the present invention is acquired. The document medium 101 is a rectangular white board (whiteboard), a poster, a paper document, or the like to be photographed. In the present invention, these are collectively referred to as a document. The photographing device 102 is a photographing device such as a digital camera that photographs the document medium 101. An image obtained by photographing with the photographing apparatus 102 becomes an image to be processed (input image). The display unit 103 is provided in the photographing apparatus 102 and displays a photographing target and the like. The operation unit 104 is provided in the photographing apparatus 102 and is used when the user operates the photographing apparatus 102.

  FIG. 2 shows an example of the input image. The image 200 is displayed on the display unit 103, for example. An image 200 photographed by the photographing apparatus 102 is composed of a document area 210 (an area where a subject document is shown) and a background area 220 which is a part other than the document area. The document area 210 includes a character area 211. In addition, when the positional relationship between the shooting position and the object is not directly opposed (that is, when shooting from an oblique direction), the document area 210 is inclined, that is, linear distortion occurs. Here, the document area 210 is actually a rectangular shape, but has a trapezoidal image due to linear distortion.

[System configuration]
FIG. 3 shows a configuration example of an image processing apparatus 300 that implements the present invention. The image processing apparatus 300 includes an image input unit 301 that inputs captured image data, a CPU 302 that executes and controls an image processing program that performs processing of the present invention on image data, a work memory that is used to execute the program, It comprises a RAM 303 used for temporary storage of data and a storage unit 304 for storing the program and data.

  Here, the configuration of the image processing apparatus 300 is an example, and may include components other than those shown here. In addition, image processing may be executed using an external general-purpose computer or the like, or image processing may be executed on an electronic circuit such as the photographing apparatus 102.

  FIG. 5 is a block diagram illustrating the entirety of this embodiment. A photographed image 501 indicates an image photographed by the photographing apparatus 102. The image determination unit 502 determines whether the captured image 501 includes a document area. The document area extraction unit 503 extracts a frame of the document area from the captured image 501 including the document area. Here, this frame is referred to as a “document frame”. The distortion correction unit 504 performs reverse perspective transformation on the document region using the coordinates on the captured image 501 of the document frame obtained by the document region extraction unit 503 as a parameter, and corrects the rectangular shape. The electronic document generation unit 505 generates an electronic document 506 that can be handled by an external program from the image corrected by the distortion correction unit 504. An electronic document 506 indicates an electronic document generated by the electronic document generation unit 505.

[Process flow]
Hereinafter, processing when the image 200 illustrated in FIG. 2 is used as an example of the captured image 501 will be described. In this processing, the CPU 302 provided in the image processing apparatus 300 in the present embodiment reads and executes a program stored in the RAM 303, the storage unit 304, or the like. The extracted straight line information and gradient information are held in the storage unit of the RAM 303 or the storage unit 304.

  The image determination unit 502 determines whether or not a document is included in the captured image 501. Regarding a method for extracting characters from image data, for example, a method such as that disclosed in Japanese Patent Application Laid-Open No. 2002-042055 can be used. As a result of the processing, a character area 211 is obtained from the image 200, and it is determined that the image includes a document image. Further, the image type may be switched by the user using a user interface.

  Detailed processing of the document area extraction unit 503 will be described with reference to the flowchart of FIG. In step S <b> 401, the document area extraction unit 503 detects a linear component from the image 200. The detection of the linear component can be performed using a known method as follows, for example. That is, pixels corresponding to the boundary portion of the document in the image 200 are emphasized by an edge enhancement method using a Sobel filter or a Laplacian filter. A straight line can be detected by using a known straight line extraction method such as the Hough transform or the minimum approximation method for the image with the edge enhanced. The detected straight line holds the coordinates of the end points and the end points in the image data, for example, and the direction of the straight line can be obtained by calculating the vector. When obtaining the vector of a straight line, for example, when the x coordinate increases from left to right and the y coordinate increases from top to bottom in the image, the value of the x coordinate of both ends of the straight line. It is also possible to obtain an end point with a small value as a start point and another point as an end point. In the case of the same x-coordinate value, the end point having a small y-coordinate value is set as the starting point.

  The result of applying the process for extracting the linear component is shown in FIG. An image 600 shown in FIG. 6A is an example in which edge enhancement is performed on the image 200 shown in FIG. An image 610 shown in FIG. 6B is a diagram showing a linear component obtained by applying the Hough transform to the image 600. As a result of this straight line extraction process, a total of eight line segments from line segment 611 to line segment 618 are obtained.

  In S402, the document area extraction unit 503 calculates the gradient direction from the change in the pixel information in the orthogonal direction with respect to all the straight line components obtained in S401. For example, in the method shown in FIG. 7A, a luminance histogram is acquired by scanning in a direction orthogonal to a straight line in a region 700 including the extracted straight line. Here, scanning in the orthogonal direction means, for example, sequentially reading out values of pixels in the orthogonal direction and extracting the values. Then, the direction of the luminance gradient is acquired from the inclination of the histogram at the intersection of the straight line and the orthogonal line. In addition, the gradient information of luminance that is obtained secondarily at the time of generating the edge image in S401 may be used.

  Specifically, as shown in FIG. 7A, as a result of scanning from left to right in a direction orthogonal to the straight line, the luminance is higher on the right side of the straight line than on the left side of the straight line. In this case, rightward gradient information is acquired assuming that the value is directed in the direction of higher value due to the level of luminance with respect to a straight line. Specifically, an orthogonal vector orthogonal to the vector is obtained from a straight vector. Then, the pixel value (luminance information in this embodiment) is scanned along the orthogonal vector to obtain a histogram. Here, if the upper part of the straight line shown in FIG. 7A is the starting point and the lower part is the ending point, the vector of the straight line is facing downward. The direction in which the pixel value is scanned is made unique based on this direction. The gradient information (luminance gradient) of the line of interest is acquired from this histogram and the orthogonal vector for the line of interest. Then, the previously extracted straight line information and gradient information are associated with each other and stored in the storage unit. The data structure that expresses the gradient information is not particularly limited. For example, a flag “1” is given if the pixel value on the right side is high toward the vector indicated by the straight line, and “0” is given if the left side is high. Good.

  An image 710 illustrated in FIG. 7B is a result of adding gradient information to the line segment illustrated in the image 600. The arrow accompanying each line segment indicates the upward direction of luminance. For example, a line segment 611 shown in FIG. 7B is indicated by an upward arrow. This indicates that the luminance of the image increases from bottom to top with respect to the position of the line segment 611, that is, the upper side of the line segment 611 is brighter. In the present embodiment, luminance information is used in the calculation of the gradient direction, but the present invention is not limited to this. For example, other information may be used as long as a change can be calculated based on pixel values in the vicinity of a straight line as a reference.

  In step S403, the document area extraction unit 503 selects one of the line segments obtained in step S401 as a processing target line. Here, it is assumed that the line segment 611 is selected as the processing target line segment. In step S404, the document area extraction unit 503 determines whether all unprocessed line segments are opposite to the processing target line segment selected in step S403 based on the gradient information calculated in step S402. The opposite side candidate list is created with all line segments determined to form opposite sides to the processing target line segment as opposite side candidates.

  A method for determining the opposite side will be described with reference to a line segment 801 and a line segment 802 in FIG. First, “inside” is obtained from the coordinates of the line segments 801 and 802. Here, “inside” refers to drawing a quadrangle 803 having four vertices at the four ends of the line segment 801 and at both ends of the line segment 802, and the inner direction of the quadrangle is “inside”. Next, in the gradient information of the line segments 801 and 802, if both the gradient directions are inward or outward, the opposite side candidate is determined. In other words, the gradient information has a value in the opposite direction in the line segment on the opposite side. For example, since the arrows 804 and 805 representing the upward direction of the luminance are directed inward, the line segment 801 and the line segment 802 are determined as opposite sides candidates. For convenience, the opposite-side candidate formed by the two line segments that increases in luminance in the inner direction is referred to as “mountain opposite side”, and the opposite-side candidate that decreases in luminance in the inner direction is referred to as “valley-type opposite side”. For example, the opposite side in the image 800 is a mountain-type opposite side, and the opposite side in the image 810 is a valley-type opposite side.

  An image 820 shown in FIG. 8B is a diagram showing an unprocessed line segment, that is, a straight line that can be the opposite side from all straight lines other than the line segment 611 with respect to the line segment 611 to be processed. As the opposite side candidates, line segments 612, 614, 616, 618 were all obtained as valley-type opposite sides. Using the naming convention of the opposite side type (line segment a, line segment b), these are respectively the valley type opposite side (611, 612), the valley type opposite side (611, 614), the valley type opposite side (611, 616), the valley This is represented as the opposite side of the mold (611, 618). Note that the actual data structure is stored in the storage means in association with the information of each straight line. Alternatively, a table may be created and information between corresponding straight lines may be held.

  In step S405, the document area extraction unit 503 determines whether the processing in step S404 has been performed on all line segments obtained in step S401. If not, the process returns to S403, and the other line segments are also processed. If the processing has been completed for all the line segments, the process proceeds to S406. That is, for the remaining unprocessed straight lines 612 to 618, the opposite side candidate list is similarly created in S404. As a result of this processing, the valley type opposite side (611, 612), the valley type opposite side (611, 614), the valley type opposite side (611, 616), the valley type opposite side (611, 618), the mountain type opposite side (612, 613), Mountain type opposite side (612, 615), mountain type opposite side (612, 617), valley type opposite side (613, 614), mountain type opposite side (613, 615), mountain type opposite side (613, 617), valley type opposite side (614 , 616), valley-type opposite sides (614, 618), valley-type opposite sides (615, 616), mountain-type opposite sides (615, 617), and valley-type opposite sides (616, 618), 15 candidate lists are obtained. Then, after processing all the straight lines, the process proceeds to S406.

  In step S406, the document area extraction unit 503 selects one of the opposite side candidates obtained in step S404 as a processing target. Here, it is assumed that the valley-shaped opposite side (611, 612) is selected as the opposite side candidate to be processed.

  In step S407, the document area extraction unit 503 creates a document frame candidate list by combining the processing opposite side candidate and the Yamatani opposite side candidate from the unprocessed opposite side candidate in the opposite side candidate list obtained in step S404. In addition, the processing target opposite side candidate and the opposite side candidate having the same side are excluded because a document frame cannot be formed. A document frame candidate composed of two valley-shaped opposite sides is referred to as a “valley frame”, and a document frame candidate composed of two mountain-shaped opposite sides is referred to as a “mountain frame”. In these document frames, the gradient directions of the four sides are the same with respect to the inside and outside of the document frame. In other words, the mountain-shaped frame has all four sides of the gradient direction facing the inside of the document frame, and the valley-shaped frame has the four sides of the gradient direction all facing the outside of the document frame.

  The valley-type opposite side (611, 612) that is the opposite-side candidate to be processed is the same valley-type opposite side and is the opposite-side candidate that does not include the line segments 611 and 612. The valley-type opposite side (613, 614), valley The mold opposite sides (614, 616), the valley opposite sides (614, 618), the valley opposite sides (615, 616), and the valley opposite sides (616, 618) are obtained as valley form frame candidates.

  Using the naming conventions of the frame types (line segment a, line segment a ′, line segment b, line segment b ′), these are respectively referred to as a valley frame (611, 612, 613, 614), a valley frame (611). , 612, 614, 616), a valley frame (611, 612, 614, 618), a valley frame (611, 612, 615, 616), and a valley frame (611, 612, 616, 618).

  In step S408, the document area extraction unit 503 determines whether the processing in step S407 has been performed on all the opposite side candidates obtained in step S404. If the processing for all the opposite side candidates is not completed, the process returns to S406, and the process is applied to the unprocessed opposite side candidate. If the processing for all the opposite sides has been completed, the process proceeds to S409.

  As described above, the document frame candidate list is similarly created in S407 for the remaining unprocessed opposite sides. By this treatment, the valley form (611, 612, 613, 614), the valley form (611, 612, 614, 616), the valley form (611, 612, 614, 618), the valley form (611, 612) 615, 616), valley form (611, 612, 616, 618), valley form (611, 614, 615, 616), valley form (611, 614, 616, 618), valley form (611, 616,613,614), trough form (611,618,613,614), trough form (611,618,614,616), trough form (613,614,615,616), trough form ( 614, 618, 615, 616), mountain frame (612, 613, 615, 617), mountain frame (612, 615, 613, 617), mountain frame (612, 617, 613, 615) Document frame candidate list Obtained. Then, after processing all the straight lines, the process proceeds to S409.

  In step S409, a quadrangle that actually becomes a document frame is calculated from the four line segments of the document frame candidate. The line segment obtained by the straight line detection is usually not the same as the actual frame side due to the influence of distortion and noise of the lens. Therefore, by extending the line segment, each intersection of the four line segments is calculated and set as a vertex.

  For example, the following four line segments will be described. Here, x and y indicate the x coordinate and y coordinate of the end point of the line segment, respectively.

Line A (Ax1, Ay1)-(Ax2, Ay2)
Line segment a (ax1, ay1)-(ax2, ay2) (opposite side of line segment A)
Line segment B (Bx1, By1)-(Bx2, By2)
Line segment b (bx1, by1)-(bx2, by2) (opposite side of line segment B)
The vertices of the quadrilateral consisting of the above four line segments can be found by calculating the intersections of the adjacent line segments A and B, line segments A and b, line segments a and B, and line segments a and b. The calculation formula 900 shown in FIG. 9A is obtained by calculating the intersection coordinates (ABx, ABy) between the line segment A (Ax1, Ay1)-(Ax2, Ay2) and the line segment B (Bx1, By1)-(Bx2, By2). Is a calculation formula for obtaining.

Here, as a condition,
(-Ay1 + Ay2) * (Bx1-Bx2)-(Ax1-Ax2) * (-By1 + By2) = 0
If it is, the adjacent sides are parallel and there is no solution, so it is excluded from the document frame candidates.

  In addition, as shown in the image 910, as shown by the quadrilateral formed by the four vertices 915, 916, 917, and 918 calculated from the line segments 911, 912, 913, and 914, the line segment does not overlap each side of the quadrilateral. Exclude from the list. Of the 14 document frame candidate lists created in S408, the valley frame (611, 612, 613, 614), the valley frame (611, 612, 614, 616), the valley frame (611, 612, 614, 618) ), Trough form (611,612,615,616), trough form (611,612,616,618), trough form (611,614,615,616), trough form (611,614,616) , 618), trough form (611, 616, 613, 614), trough form (611, 618, 613, 614), trough form (613, 614, 615, 616), trough form (614, 618) , 615, 616), mountain frame (612, 613, 615, 617), and mountain frame (612, 615, 613, 617) are excluded because the four sides included do not form a document frame. . As shown in the image 1000 in FIG. 10A, the frame shape is finally narrowed down to two frame candidates of a valley frame (611, 614, 616, 618) 1001 and a mountain frame (612, 617, 613, 615) 1002, All the processes in the flowchart of FIG. 4 are completed. In this flowchart, the process in units of opposite sides is performed for the sake of explanation. However, it is also possible to comprehensively combine the four line segments from the beginning and determine the document frame candidate based on the gradient information.

  An image 1010 shown in FIG. 10B is an example in which the narrowing result of document frame candidates is displayed on the display unit 103. The original image display unit 1011 displays the document frame candidate as an overlay on the captured image. The candidate thumbnail portion 1012 displays the result of distortion correction for each document frame candidate as a thumbnail. By referring to the image displayed on the display unit 103 and performing a selection operation on the operation unit 104 by the user, a document frame candidate to be used for correction can be determined. Here, it is assumed that the mountain frame (612, 617, 613, 615) 1002 is selected by the user as the document frame used for correction. Note that the configuration of the image 1010 displayed on the display unit 103 is an example, and may be determined on a personal computer, or distortion correction processing may be applied to all one or more document frame candidates to form an electronic document. .

  The distortion correction unit 504 corrects distortion from the vertex information of the document frame obtained by the document area extraction unit 503. The distortion correction here is a so-called reverse perspective transformation calculation that corrects an unequal square area generated when a rectangular area is projected onto a two-dimensional plane with a three-dimensional angle to the original rectangular area. is there. The parameters of the transformation matrix used can be extracted by solving the simultaneous equations obtained by giving the coordinates of the four vertices to the inverse perspective transformation arithmetic expression as disclosed in, for example, Japanese Patent Laid-Open No. 2003-288588. In addition, any method may be used as long as it can be applied to the present invention.

  The electronic document generation unit 505 generates and outputs the corrected image as an electronic document 506. Here, the electronic document 506 is output in the JPEG format. The output in the JPEG format is an example, and conversion according to a format that can be handled by the electronic device or conversion into a reusable electronic document such as a word processing document or a presentation document may be performed.

  An image 1100 illustrated in FIG. 11A is an example of an electronic document 506 that is output as a result of applying the first embodiment. A portion distorted in a trapezoidal shape from the tilt with respect to the object is electronically documented in a rectangular state facing the object.

  Further, an image 1110 in FIG. 11B is an example in the case where gradient information is not used. Even a document frame such as a document frame 1111 including line segments 612, 614, 615, and 618 is detected. In this case, it is necessary to select document frame candidates from a total of 16 patterns.

  As described above, by applying the present invention, it is possible to reduce a document frame candidate while leaving a suitable document frame candidate as compared with the case where the present invention is not applied, and the processing load on the subsequent stage can be reduced.

<Embodiment 2>
In the first embodiment, the opposite side candidates are simply searched from all the straight lines. However, if a large number of straight lines are detected from the background area or the like in an actual document image, the load increases accordingly. Therefore, it is possible to further speed up the narrowing-down process by combining a method for classifying a vertical line segment and a horizontal line segment.

  FIG. 12 is a diagram illustrating a document area extraction process in the document area extraction unit 503 according to this embodiment with reference to a flowchart. Hereinafter, processing will be described using the image 200 illustrated in FIG. 2 as an example of the input captured image 501. Note that this processing flow is realized, for example, by the CPU 302 reading and executing a program or data stored in the RAM 303 or the storage unit 304. In step S1201, the document area extraction unit 503 detects a straight line component. The detailed processing method is the same as that in S401, and will be omitted. FIG. 6B shows the result of straight line detection of the image 610 shown in FIG. Here, eight straight lines from line segments 611 to 618 are detected. In step S1202, the document area extraction unit 503 calculates a straight line gradient direction. Since the detailed processing method is the same as that in S402, the description is omitted. An image 710 shown in FIG. 7B is a result of adding gradient information.

  In step S1203, the document area extraction unit 503 determines a vertical direction line and a horizontal direction line. The vertical line and the horizontal line are determined from the relative angle of the line segment to the horizontal on the image plane. Here, if it is 0 degree or more and less than 45 degree | times relative to the horizontal, if it is 135 degree or more and less than 180 degree | times, it will determine with a horizontal direction line, and if it is 45 degree | times or more and less than 135 degree | times, it will determine with a vertical direction line. In addition, the reference | standard for determining a vertical direction line or a horizontal direction line is not limited to said value, You may change as needed. Also, other methods may be used as long as the present invention is applicable. The image 1300 shown in FIG. 13 is a result of determining the vertical direction line and the horizontal direction line. Here, the vertical direction line is represented by a solid line, and the horizontal direction line is represented by a broken line.

  In step S1204, the document area extraction unit 503 selects a line segment to be processed. Here, it is assumed that the line segment 611 is selected. In step S1205, the document area extraction unit 503 performs branching processing based on whether the processing target line segment is a vertical line or a horizontal line. If it is a vertical line, the process proceeds to S1206. If it is a horizontal line, the process proceeds to S1207. Here, since the line segment 611 is a horizontal line, it progresses to S1207.

  In step S <b> 1207, the document area extraction unit 503 creates an upper / lower opposite candidate for an unprocessed horizontal line. Here, line segments 612, 617, and 618 are targeted as unprocessed horizontal lines. Since the determination of the opposite side candidate itself is the same as S404, the description is omitted. As a result, 612 and 618 were obtained as the opposite side candidates, as valley-type opposite sides in the horizontal direction. These are represented as upper and lower valley type opposite sides (611, 612) and upper and lower valley type opposite sides (611, 618), respectively. The upper and lower valley type opposite sides and the upper and lower mountain type opposite sides are “upper and lower sides”, and the left and right valley type opposite sides and the left and right mountain type opposite sides are “left and right opposite sides”.

  In accordance with the end determination in S1208, the line segments 612 to 618 are repeatedly processed. Assume that the line segment 613 (vertical line) is selected in S1204. In S1205, the vertical line is determined, and the process proceeds to S1206. In S1206, left and right side candidates are created for the line segments 614, 615, and 616 as unprocessed vertical lines. As a result of the processing, a line segment 614 is acquired as a valley-shaped opposite-side candidate, and a line segment 615 is acquired as a mountain-shaped opposite-side candidate. These are represented as left and right trough type opposite sides (613, 614) and left and right mountain type opposite sides (613, 615), respectively.

  Similarly, S1204 to S1208 are repeated for the remaining line segments. As a result of processing, the upper and lower valley type opposite sides (611, 612), the upper and lower valley type opposite sides (611, 618), the upper and lower valley type opposite sides (617, 618), the upper and lower mountain type opposite sides (612, 617), and the left and right valley type opposite sides (613) 614), left and right valley-type opposite sides (614, 616), left and right valley-type opposite sides (615, 616), and left and right mountain-type opposite sides (613, 615). After the processing is completed for all the line segments, the process proceeds to S1209.

  In step S1209, the document area extraction unit 503 selects the upper and lower valley type opposite side (611, 612) as the processing target opposite side, and the process advances to step S1210. In step S <b> 1210, the document area extraction unit 503 branches the process depending on the top / bottom side or the left / right side. Here, since the upper and lower valley type opposite sides (611, 612) are “upper and lower sides”, the process proceeds to S1211.

  In step S <b> 1211, the document area extraction unit 503 creates a document frame candidate list for unprocessed left and right opposite sides. Here, the left and right valley type opposite sides (613, 614), the left and right valley type opposite sides (614, 616), the left and right valley type opposite sides (615, 616), and the left and right mountain type opposite sides (613, 615) are targeted. Detailed processing is the same as that in S407, and will be omitted. As a result of the processing, left and right valley type opposite sides (613, 614), left and right valley type opposite sides (614, 616), and left and right valley type opposite sides (615, 616) were obtained as document frame candidates. These are represented as valley type frame candidates (611, 612, 613, 614), valley type frame candidates (611, 612, 614, 616), and valley type frame candidates (611, 612, 615, 616), respectively.

  In accordance with the end determination in S1213, processing is performed for the unprocessed opposite side. Assume that the left and right valley-type opposite sides (613, 614) are selected in S1209. In S1210, it is determined that the side is the opposite side, and the process proceeds to S1212. In S1212, a document frame candidate list is created for the unprocessed upper and lower opposite sides. Here, the upper and lower valley type opposite sides (611, 618), the upper and lower valley type opposite sides (617, 618), and the upper and lower mountain type opposite sides (612, 617) are targeted. As a result of the processing, valley shape frame candidates (611, 618, 613, 614) and valley shape frame candidates (617, 618, 613, 614) are obtained.

  Similarly, S1209 to S1213 are repeated for the remaining opposite sides. As a result, the valley form frame candidate (611, 612, 613, 614), the valley form frame candidate (611, 612, 614, 616), the valley form frame candidate (611, 612, 615, 616), the valley form frame candidate (611) , 618, 613, 614), valley-shaped frame candidates (611, 618, 614, 616), valley-shaped frame candidates (611, 618, 615, 616), valley-shaped frame candidates (617, 618, 613, 614), There are ten document frame candidate lists of valley type frame candidates (617, 618, 614, 616), valley type frame candidates (617, 618, 615, 616), and mountain type frame candidates (612, 617, 613, 615). Created. After the processing is completed for all the line segments, the process proceeds to S1209.

  In step S1214, the document area extraction unit 503 calculates the vertex of the document frame from the four line segments. Detailed processing is the same as in step S409, and is therefore omitted. As a result of the processing, the two frame candidates of the valley frame (611, 614, 616, 618) 1001 and the mountain frame (612, 617, 613, 615) 1002 are finally narrowed down, and the entire processing of the flowchart of FIG. Is completed. Since the processes of the distortion correction unit 504 and the electronic document generation unit 505 are the same as those in the first embodiment, a description thereof will be omitted.

  As described above, compared with the first embodiment, the number of opposite side candidates can be reduced from 15 to 8, and the number of candidate document frames can be reduced from 15 to 10. In the opposite side candidate creation processing, in the first embodiment, the search is performed in 28 ways that are combinations of two line segments out of eight, whereas in the second embodiment, a combination of two out of four vertical lines and a horizontal direction A total of 12 searches with combinations of 2 out of 4 lines are sufficient. Thus, in addition to the effects of the first embodiment, the processing cost can be further reduced.

<Other embodiments>
The present invention can also be realized by executing the following processing. That is, software (program) that realizes the functions of the above-described embodiments is supplied to a system or apparatus via a network or various storage media, and a computer (or CPU, MPU, or the like) of the system or apparatus reads the program. It is a process to be executed.

Claims (6)

An image processing apparatus for extracting a frame composed of four sides of the rectangular area from image data obtained by photographing a subject having a rectangular area ,
Detecting means for detecting a plurality of linear components from the input image data;
Calculating means for calculating a gradient direction according to the level of pixel information in a direction orthogonal to the linear component detected by the detecting means;
An extraction unit that selects four sides from the plurality of linear components detected by the detection unit, and extracts one or more frame candidates including the selected four sides;
Among the frame candidate extracted by the extraction means, a frame candidate gradient direction of four sides of the frame candidate is not the same orientation with respect to either direction of the inside or outside of the frame, by the extraction means An image processing apparatus comprising: a narrowing-down unit that excludes from the extracted frame candidates. The extraction means includes
Classifying means for classifying the linear component into a horizontal line as upper and lower sides and a vertical line as left and right sides in a frame from a relative inclination with respect to the input image data;
Selection means for selecting a certain vertical direction line and another vertical direction line having a reverse gradient direction as left and right opposite sides, and selecting a certain horizontal direction line and another horizontal direction line having a reverse gradient direction as upper and lower opposite sides Further comprising
The image processing apparatus according to claim 1, wherein the frame candidate is extracted from the four sides by combining the upper and lower opposite sides and the left and right opposite sides.   The image processing apparatus according to claim 1, wherein the calculation unit calculates a gradient direction using a luminance value as the pixel information.   The image processing apparatus according to claim 1, wherein the frame is a frame of a whiteboard, a poster, or a paper document. A control method of an image processing apparatus for extracting a frame composed of four sides of the rectangular area from image data obtained by photographing a subject having a rectangular area ,
A detecting step in which the detecting means of the image processing device detects a plurality of linear components from the input image data;
A calculation step in which the calculation means of the image processing device calculates a gradient direction according to the level of pixel information in a direction orthogonal to the linear component detected in the detection step;
An extraction step in which the extraction means of the image processing device selects four sides from the plurality of linear components detected in the detection step, and extracts one or more frame candidates composed of the selected four sides;
Narrowing means of said image processing apparatus, of the frame candidates extracted in the extraction step, not the same orientation with respect to either direction of the inside or outside of the gradient direction in the four sides of the frame candidate the frame And a narrowing-down step of removing frame candidates from the frame candidates extracted in the extraction step . Computer
Detecting means for detecting a plurality of linear components from the input image data;
Calculating means for calculating a gradient direction according to the level of pixel information in a direction orthogonal to the linear component detected by the detecting means;
An extraction unit that selects four sides from the plurality of linear components detected by the detection unit and extracts one or more frame candidates including the selected four sides;
Among the frame candidate extracted by the extraction means, a frame candidate gradient direction of four sides of the frame candidate is not the same orientation with respect to either direction of the inside or outside of the frame, by the extraction means A program for functioning as a narrowing-down means to be excluded from the extracted frame candidates.

Images