JP2013235418A - Image processing device and manuscript reading system - Google Patents

Abstract

PROBLEM TO BE SOLVED: To make it possible to, in a simple configuration, prevent an inappropriate tilt correction from being performed when it is difficult to detect an effective straight line component from a photographing image.SOLUTION: A manuscript reading system 1 is configured to comprise: a line segment extraction part 33 that extracts a plurality of line segments on the basis of an edge pixel extracted from a photographing image; a cross line segment percentage calculation part 34 that determines a cross state of the line segments and thereby calculates a cross line segment percentage of the plurality of line segments; a tilt correction angle setting part 35 that performs statistical processing to tilt angles of the plurality of line segments and thereby estimates a tilt angle of a photographing image; and an image rotation part 36 that rotates a photographing image on the basis of an estimated tilt angle and thereby performs a tilt correction of the photographing image. When the cross line segment percentage is equal to or more than a threshold value, the image rotation part is configured so as not to perform the tilt correction.

Description

  The present invention relates to an image processing apparatus that processes an image obtained by reading a document such as a book, and a document reading system including the image processing apparatus.

  2. Description of the Related Art Document cameras (book scanners) that can take a picture of a page and read an image of the page while the page of the book is naturally opened are widely used (see Patent Document 1). When such a document camera is used, the images on the page can be read one after another while turning the page, so that the work of digitizing the book can be performed efficiently. Further, in the document reading system using the document camera, even when the document is set in an inclined state with respect to a normal reading position (that is, a reference direction such as horizontal or vertical), an image of the read document ( The inclination of the image can be automatically corrected by estimating the inclination angle based on the document binding portion and the position of the document edge.

  With respect to the technique for estimating the inclination angle of an image, for example, a Hough transforming unit that generates a parameter chart by performing Hough transform processing (linear component extraction) on binarized image data after edge processing that emphasizes the pixel density of an edge portion , A histogram generating means for generating a histogram by integrating the frequency of coordinates in the parameter chart for each angle θ, and an inclination angle detecting means for estimating the inclination angle by specifying the angle θ having the maximum frequency of coordinates from the histogram Is known (see Patent Document 2).

JP 2001-103240 A JP 11-328408 A

  By the way, in order to accurately estimate the inclination angle of the image as described above, effective linear components detected from the outer contour, the character string, the figure frame, and the like along the reference direction such as the horizontal / vertical direction of the document are used. It will be necessary. On the other hand, for example, if there are photographs, complex figures, etc. over a wide area of the original, the linear component that becomes noise is mixed in the estimation of the tilt angle, making it difficult to detect effective linear components. However, there is a problem that a large error occurs in the estimation of the tilt angle of the image.

  However, since the prior art described in Patent Document 2 does not take into account the contents of the page, it has been difficult to cope with the above problems. Therefore, with such conventional techniques, when images that are difficult to detect effective linear components are corrected for tilt and displayed or recorded continuously, the orientation of the image after a series of tilt correction changes unnaturally. It becomes an image that is difficult to see.

  The present invention has been devised in view of such problems of the prior art, and with a simple configuration, inappropriate tilt correction is performed when it is difficult to detect an effective linear component from a captured image. It is a main object of the present invention to provide an image processing apparatus that can prevent such a situation and a document reading system including the same.

  An image processing apparatus according to the present invention includes an edge extraction unit that extracts a plurality of edge pixels in a captured image obtained by sequentially photographing a paper surface of a document, a straight line extraction unit that extracts a plurality of straight line components based on the edge pixels, and the straight line component An intersection line segment calculation unit that calculates an intersection line segment ratio that is a ratio of linear components intersecting each other among the plurality of linear components, and an inclination angle of the plurality of linear components A tilt angle estimation unit that estimates the tilt angle of the captured image by performing statistical processing; and the tilt correction of the captured image by rotating the captured image based on the tilt angle estimated by the tilt angle estimation unit A first operation mode in which the image rotation unit performs the tilt correction, and a second operation mode in which the image rotation unit does not perform the tilt correction. And, the intersecting line segment ratio is equal to or greater than a predetermined threshold value, and executes the second operation mode.

  As described above, according to the present invention, it is possible to prevent an inappropriate inclination correction from being performed when it is difficult to detect an effective linear component from a captured image with a simple configuration.

1 is an overall configuration diagram showing a document reading system 1 according to the present invention. The top view which shows the state which set the book B in the regular reading position in the original reading system 1 shown in FIG. 1 is a block diagram showing a schematic configuration of the document camera 2 and the PC 3 in FIG. FIG. 1 is a flowchart showing a main part of an image display procedure by the document reading system 1 shown in FIG. Explanatory drawing which shows the example of the intersection line segment acquired by cross line segment acquisition (ST105) in FIG. FIG. 4 is a flowchart showing details of the intersection line segment acquisition (ST105) in FIG. Explanatory drawing which shows the specific example of cross line segment acquisition (ST105) in FIG. 4, and the decision | availability determination (ST106) of deskew implementation. State transition diagram showing an outline of the determination process of deskew execution in FIG. FIG. 4 is a flowchart showing details of the deskew execution feasibility determination (ST106) in FIG. Flow chart showing details of line segment information statistical processing (ST107) in FIG.

  According to a first aspect of the present invention, there is provided an edge extracting unit that extracts a plurality of edge pixels in a photographed image obtained by sequentially photographing a paper surface of a document, and a straight line that extracts a plurality of linear components based on the edge pixels. An intersecting line segment calculating unit that calculates an intersecting line segment ratio that is a ratio of linear components intersecting each other among the plurality of linear components by determining an intersecting state of the linear components; A tilt angle estimator for estimating the tilt angle of the captured image by performing statistical processing on the tilt angle of the linear component, and rotating the captured image based on the tilt angle estimated by the tilt angle estimator. An image rotation unit that performs tilt correction of the captured image, wherein the image rotation unit performs the tilt correction, and the image rotation unit does not perform the tilt correction. And a second operating mode, the cross-line rate is equal to or greater than a predetermined threshold, and configured to execute the second operation mode.

  According to this, with a simple configuration, it is possible to prevent inappropriate inclination correction from being performed when it is difficult to detect an effective linear component from a captured image. In particular, since the intersection line segment ratio of the straight line component is used as an index of the effective straight line component ratio (in other words, the noise component ratio), the type of image (photo, graphic, character, etc.) is determined. There is an advantage that whether or not tilt correction can be performed can be determined at high speed and with high accuracy, while eliminating the need for simple processing.

  In a second aspect based on the first aspect, the threshold value includes a first value and a second value that is larger than the first value. When the intersecting line segment ratio in the photographed image decreases from the intersecting line segment ratio of the previous photographed image that is larger than the first value and falls below the first value, the second operation mode is changed to the first operation mode. When switching to the first operation mode, the intersecting line segment ratio in the current photographed image increases from the intersecting line segment ratio in the previous photographed image smaller than the second value and exceeds the second value. The first operation mode is switched to the second operation mode.

  According to this, since the two operation modes are switched by providing hysteresis with two threshold values, when the captured image is sequentially tilt-corrected, some destabilization factor (illumination) is used even though the document is stationary. ), Slight fluctuations in the paper surface, image sensor noise, randomness of stochastic Hough transform, etc. Can do.

  In a third aspect based on the first aspect or the second aspect, the inclination angle estimating unit intersects one or more other line segments with a straight line component to be determined by determining the intersection state. In this case, the straight line component that is the determination target is excluded from the statistical processing target.

  According to this, when the straight line component that is the determination target intersects with one or more other linear components, the straight line component that is the determination target is excluded from the statistical processing target. Noise components (that is, inappropriate linear components that do not match the reference direction of the document) can be effectively removed from the target, and as a result, the tilt angle of the captured image can be estimated with high accuracy.

  According to a fourth aspect of the present invention, there is provided a document reading system including the image processing apparatus according to any one of the first to third aspects of the present invention and an image input apparatus having a camera unit that generates the captured image.

  Hereinafter, embodiments of the present invention will be described with reference to the drawings.

  FIG. 1 is an overall configuration diagram showing a document reading system 1 according to the present invention, and FIG. 2 is a plan view showing a state where a book B is set at a regular reading position in the document reading system 1. The document reading system 1 reads a paper image of a book (original) B to acquire image data of the paper surface, and a document camera (image input device) 2 that shoots the paper surface and converts it into a video signal. The document camera 2 and the PC 3 connected to be communicable with each other.

  The document camera 2 includes a camera unit 4 having a photographing function and a stand unit 5 that holds the camera unit 4. The camera unit 4 includes an image sensor made up of a CCD, a CMOS, etc., and an illumination light source (both not shown) made up of an LED, a fluorescent lamp, or the like. The stand unit 5 includes a leg 7 that is mounted on a mounting surface 6 such as a desk surface and that is open in a substantially V shape, and an arm 8 that is supported by the leg 7. On the upper surface of the leg 7, a pair of guide members 9 for defining the photographing position of the book B are projected. The arm 8 is provided so as to be capable of extending and contracting obliquely upward from the leg 7, while holding the camera unit 4 so as to be rotatable by a hinge unit 8 a, whereby the angle of view and the optical axis direction of the camera unit 4 (photographing direction). ) Can be adjusted. The camera unit 4 also has a known zoom function that continuously changes the angle of view and magnification.

  The PC 3 functions as an input / output device for the user to set various operating conditions of the document camera 2 and for the user to check a captured image captured by the document camera 2, and to perform image processing and recording of the captured image. It also functions as an image processing apparatus that performs the above.

  The apparatus used together with the document camera 2 in the document reading system 1 is not limited to a PC (Personal Computer), and any information processing apparatus capable of realizing the same function can be used. It is also possible to add a part of the functions of the PC 3 to the document camera 2 or to configure the PC 3 and the document camera 2 integrally. Further, the document camera 2 and the PC 3 do not need to be directly connected, and may be connected via a network (not shown), for example. In such a configuration, the document camera 2 generates a trigger signal for imaging (specifically, for example, a switch or button for instructing image capture), and pushes the remote PC 3 (for example, a server). Data transmission.

  In photographing with the document camera 2, the user places the book B in an open state on the placement surface 6 immediately below (in the optical axis direction) of the camera unit 4, so that two pages of the book B and the periphery of the book B are placed. A captured image (moving image or still image) including a part of the mounting surface 6 is obtained. At this time, as shown in FIG. 2, the user sets the book B at the reading position with high accuracy by adjusting the position and inclination of the book B by contacting the upper edge Ba of the book B with the guide member 9 extending in the left-right direction. Is possible. The captured image is appropriately transmitted to the PC 3, where necessary image processing is performed, and then stored in a predetermined recording medium and displayed on the display for the user.

  The document read by the document reading system 1 is not limited to a book, but may be any information transmission medium including characters, drawings, photographs, and similar information. Further, the information transmission medium may be a transmission original such as a positive film or a negative film. In this case, for example, light is emitted from the back side by a light source unit in which a light source is arranged on the side of a transparent light guide plate, and transmitted light is transmitted. May be configured to take an image.

  FIG. 3 is a block diagram showing a schematic configuration of the document camera 2 and the PC 3 in FIG.

  The document camera 2 includes an imaging processing unit 11 having a camera unit 4, an operation instruction unit 12 that causes the imaging processing unit 11 to perform a required operation based on an operation condition set by a user, and a PC 3. And an external interface 13 conforming to the USB standard or the like.

  The PC 3 includes an external interface 21 compliant with the USB standard for connection with the document camera 2, an image data input unit 22 to which photographed image data from the document camera 2 is input, and recording and display of photographed images. An image processing unit 23 that performs necessary image processing, an operation system control unit 25 that transmits operation conditions set by the user to the document camera 2 in an input operation unit 24 such as a keyboard, and a captured image after image processing. It has a display data generation unit 27 that generates data to be displayed on a display unit 26 such as an LCD or a projector, and a data storage unit 28 that stores captured image data after image processing. The image processing unit 23 includes an image tilt correction unit 30 that performs tilt correction of a captured image. The captured image data processed by the image inclination correction unit 30 is read from the image data storage unit 37. In addition, information on various parameters used for calculation of intersection line segments and statistical processing in the image inclination correction unit 30 described later is read from the parameter storage unit 38.

  The processing functions of the image data input unit 22, the image processing unit 23, the operation system control unit 25, the display data generation unit 27, and the like in the PC 3 can be realized by software processing in which a program such as an image processing application is executed by the CPU. Of course, the PC 3 may be interpreted as an image processing apparatus, and may be configured to include hardware that executes specific processing at high speed. The image data storage unit 37 and the parameter storage unit 38 are composed of general-purpose memories.

  In the PC 3, the image data input unit 22 sequentially stores the captured image data transmitted from the document camera 2 in the image data storage unit 37 of the image processing unit 23, and the captured image data is stored in the image inclination correction unit 30 as necessary. Are output sequentially. Then, the image tilt correction unit 30 estimates the tilt angle of the captured image during image processing of the captured image data, and corrects the tilt of the captured image based on the estimated tilt angle. The captured image data whose inclination is corrected is stored in the data storage unit 28 and is also sent to the display data generation unit 27 and displayed on the display unit 26.

  The image inclination correction unit 30 is a grayscale conversion unit 31 that performs grayscale conversion or binarization of a captured image, and an edge extraction unit that extracts a plurality of edge pixels from the grayscale converted captured image (performs edge detection). 32, a line segment extraction unit (straight line extraction unit) 33 that extracts a plurality of line segments (straight line components) connecting the edge pixels based on the extracted plurality of edge pixels, and a mutual intersection of each extracted line segment An intersection line segment calculation unit 34 that determines a state and calculates an intersection line segment ratio that is a ratio of all line segments that intersect each other with other line segments; and a plurality of extracted line segments The inclination angle of the captured image is estimated by performing statistical processing on the inclination angle of the image, and an inclination correction angle setting unit (inclination angle estimation unit) 35 that sets the inclination correction angle of the captured image based on the inclination angle; And an image rotating unit 36 for rotating the captured image based on the correction angle.

  In the PC 3, the user can appropriately input operating conditions such as the resolution of the image captured by the document camera 2, the frame rate, the shutter speed, and the amount of light emitted from the illumination light source by operating the input operation unit 24. it can. This operating condition is transmitted as a control signal from the operation system control unit 25 to the document camera 2, and the document camera 2 captures the imaging processing unit 11 in accordance with a processing command sent from the operation instruction unit 12 based on the control signal from the PC 3. Performs the shooting operation.

  FIG. 4 is a flowchart showing the main part of the image display procedure by the document reading system 1 shown in FIG. 1, and FIG. 5 shows the intersection line segment ratio acquired in the intersection line segment acquisition (ST105) in FIG. It is explanatory drawing which shows an example.

  A user who uses the document reading system 1 first sets the book B in an opened state at a reading position (position where the document camera 2 can be photographed), activates the document camera 2, and activates a required application on the PC 3. . Thereafter, imaging by the document camera 2 is started, and input of an image from the document camera 2 is detected in the image data input unit 22 of the PC 3 (ST101). The input of the captured image is performed at a predetermined frame rate, and the following processing is sequentially performed on each captured image.

  Next, in the PC 3, after the captured image data received from the document camera 2 is stored in the image data storage unit 37 by the image data input unit 22, the gray scale conversion unit 31 converts the captured image from an RGB color image to a monochrome image. (ST102). This gray scale conversion process can be performed using a known method such as an intermediate value method. If the document camera 2 is configured to directly output the signal after YC separation, the Y (luminance) signal may be used as it is.

  Subsequently, the edge extraction unit 32 extracts, as edge pixels, a portion where the luminance changes abruptly in the grey-scale converted captured image (ST103). This edge extraction process can be performed using a known method such as the Canny method. Further, the line segment extraction unit 33 extracts line segments (straight line components) from the plurality of acquired edge pixels (ST104). This line segment extraction process can be performed using a known method such as probabilistic Hough transform. Probabilistic Hough transform is used when detecting a line segment having an end point from an image, and the coordinate values of the start point and end point of the detected line segment can be acquired.

  Information of each line segment such as the coordinate value of the end point extracted here (hereinafter referred to as “line segment information”) is stored in the parameter storage unit 38. This line segment information includes not only data such as the coordinates of both end points in each line segment, but also a flag indicating whether or not it is an intersection line segment to be described later (hereinafter referred to as “intersection line flag”; two line segments). The crossing line flag is “false” when does not intersect, and “true” is assumed when it intersects.) A sign is included. Note that the linear component extracted from the captured image is not limited to the line segment shown in the present embodiment as long as it can estimate the degree of inclination of the book B by statistical processing of at least those inclination angles. It is not something.

  Next, the intersection line segment calculation unit 34 determines the intersection state of each line segment, and calculates the intersection line segment ratio for the input captured image (ST105). If we define “intersection line” as a line segment that has an intersection with another line segment and “non-intersection line segment” as a line segment that does not have an intersection with another line segment, the intersection line segment = intersection line segment / (Number of intersecting line segments + number of non-intersecting line segments). It should be noted that the number of intersecting line segments + the number of non-intersecting line segments = the number of all line segments. In addition, the “intersection line segment” may include one whose own end point coincides with the end point of another line segment, or one whose own end point is located on another line segment.

  Here, as shown in FIG. 5, the intersection line segment ratio of the captured image 41 varies greatly depending on the content of the paper surface that is the spread page (that is, the type of the image). For example, in the textbook B1 shown in FIG. 5A, a text area 43 where characters, symbols, and the like are displayed occupies most of the paper surface, and a graphic area 44 including a table, a drawing, a photograph, and the like exists in part. Since many non-intersecting line segments (straight line components effective in estimating the inclination angle of the image) are extracted from the outline of the textbook B1, the frame area of the text area 43, and the graphic area 44, the intersection here The line segment is about 20-30%. Further, in the animal picture book B2 shown in FIG. 5 (B), the graphic area 44 such as a photograph from which a large number of intersecting line segments (noise components) are extracted occupies more on the paper than the text area 43. The intersection line segment is about 60 to 70%. Furthermore, in the image taken by enlarging a part (photograph part) of the animal picture book B2 shown in FIG. 5C, the graphic area 44 occupies the entire photographed image 41, and the outline of the book does not exist. Therefore, the intersection line segment here is approximately 100%. Here, as shown in FIG. 5A, an example in which the reference direction (see line C1) of the document (textbook B1) is inclined by α ° with respect to the reference direction (see line C2) of the photographed image is shown. Yes. In FIG. 5C, for convenience of illustration, the photographed image 41 is shown in a reduced scale with respect to the animal picture book B2 shown in FIG.

  Referring to FIG. 4 again, next, the inclination correction angle setting unit 35 determines whether or not deskew can be performed on the current captured image based on the intersection line segment ratio calculated in ST105 (whether or not inclination correction by the image rotation unit is possible). Is determined (ST106). Although details will be described later, in this deskew execution determination, if the intersection line segment ratio is equal to or less than a predetermined threshold, it is determined that deskew is ON (ST106: YES), and the PC3 executes the first operation mode in which the inclination is corrected. Is done. On the other hand, when the intersection line segment ratio is equal to or greater than the predetermined threshold, it is determined that deskew is OFF (ST106: NO), and thereby the second operation mode in which the PC 3 does not perform inclination correction is executed.

  When the inclination correction is executed (ST106: YES), the inclination correction angle setting unit 35 performs statistical processing (hereinafter, line segment information statistical processing) on information regarding the inclination angles of the plurality of line segments detected by the line segment extraction unit 33. The inclination angle of the photographed image is estimated to set the inclination correction angle (ST107). In this line segment information statistical process, histogram information relating to the inclination angle of the line segment is generated.

  Further, based on the tilt correction angle, the image rotation unit 36 rotates the rectangular captured image and corrects the tilt of the captured image (ST108). Thereby, it is possible to generate a captured image equivalent to the case where the paper surface of the book B is read at the normal reading position (that is, without tilting). Note that the captured image whose inclination has been corrected is sent from the image inclination correction unit 30 to the display data generation unit 27 and the data storage unit 28. Then, the display data generation unit 27 extracts, for example, an area in which the content is described from the entire page, and corrects the curvature caused by the height of the book B with respect to the photographed image after the tilt correction, thereby correcting the flat image. Is generated and displayed on the display unit 26 (ST109). On the other hand, when the tilt correction is not executed (ST106: NO), a captured image that is not tilt corrected is displayed on the display unit 26.

  In the PC 3, the series of processes ST <b> 101 to ST <b> 109 are repeatedly executed, so that a series of images (video) whose inclination is corrected is sequentially displayed on the display unit 26. When it is determined in ST106 that deskew execution is not possible, ST107 and ST108 are omitted. However, the present invention is not limited to this. For example, the same determination process as ST106 is performed regardless of the inclination correction angle set in ST107. May be configured to be executed by the image rotation unit 36 in ST108.

  By executing such processing, the original reading system 1 has a simple configuration and the proportion of effective line segments detected from the captured image is low, that is, it is difficult to detect effective linear components from the captured image. However, it is possible to prevent inappropriate inclination correction from being executed. In particular, since the intersection line segment ratio is used as an index of the effective line segment ratio (in other words, the ratio of the noise component), complicated processing such as determining the type of image (photo, graphic, character, etc.) is performed. There is an advantage that whether or not inclination correction can be performed can be determined at high speed and with high accuracy while making it unnecessary.

  FIG. 6 is a flowchart showing details of the intersection line segment acquisition (ST105) in FIG. First, the intersection line segment calculation unit 34 initializes an intersection line segment counter i indicating the number of line segments intersecting with other line segments to set i = 0 (ST201). Therefore, one line segment that has not been determined for the intersection state is extracted as a determination target line segment from all the line segments acquired as line segment information (ST202).

  When an undetermined line segment is extracted in ST202 (ST203: YES), an intersection line segment flag indicating the intersection state of the line segment is initialized to “false” (no intersection) (ST204). One line segment other than the determination target line segment is extracted as a comparison target line segment (another line segment) from all the line segments acquired as information (ST205). In ST205, another line segment that has not been compared with the current determination target line segment in the line segment information is sequentially selected as a comparison target line segment.

  When the comparison target line segment is extracted in ST205 (ST206: YES), the intersection line segment ratio calculation unit 34 determines whether or not the determination target line segment extracted in ST202 intersects the comparison target line segment extracted in step ST205. This determination is executed (ST207). The intersection state of each line segment can be determined using a known method based on the coordinates of both end points of the line segment. Therefore, when the determination target line segment intersects with the comparison target line segment (ST208: YES), the intersection line segment ratio calculation unit 34 changes the intersection line segment flag to “true” (with intersection) (ST209), The intersection line counter i is counted up to i = i + 1 (ST210). Thereafter, the intersection line segment calculation unit 34 excludes the current determination target line segment from the extraction candidates of ST202 (ST211), and further resets the comparison target line segment information to be extracted in ST205 (ST212). Thereafter, the process returns to ST202 and the same processing as described above is performed.

  On the other hand, if it is determined in ST208 that they do not intersect (NO), the intersection line segment calculation unit 34 excludes the current comparison target segment from the extraction candidates of ST205 (ST213), and returns to ST205 for the next comparison target. A line segment is extracted and the same processing as described above is performed. When the current determination target line segment has been compared with all the comparison target line segments and no new comparison target line segment is extracted in ST205 (ST206: NO), the determination target line segment Since neither line segment intersects, the processing of ST209 and ST210 is omitted and the process proceeds to ST211.

  When the determination of the intersection state is finally completed for all the line segments (ST203: NO), the intersection line segment calculation unit 34 divides the value of the intersection line segment counter i by the total number of line segments acquired as the line segment information. Thus, the intersection line segment ratio (%) is calculated (ST214).

  Note that the value (false or true) of the intersection line segment flag assigned to each line segment in the process of acquiring the intersection line segment ratio is referred to in the histogram frequency count generated to estimate the inclination angle of the document. .

  FIG. 7 is an explanatory diagram showing a specific example of crossing line segment acquisition (ST105) and deskew execution feasibility determination (ST106) in FIG. 4, and FIG. 8 shows an outline of the deskew execution feasibility judgment process in FIG. FIG.

  As shown in FIG. 7, here, a threshold TL1 (first value) and a threshold TL2 (second value) larger than the threshold TL1 are used as the threshold of the intersection line segment ratio for determining whether or not deskew can be performed. Is used. Here, the threshold value TL1 = 70% and the threshold value TL2 = 80%, but these values can be appropriately changed according to the content of the document to be photographed. Further, in FIG. 7, the result of the determination on whether or not deskew can be performed (ST106) is switched (deskew ON → OFF or deskew OFF → ON) is displayed as a white circle and other points (black circle) It is distinguished from.

  First, at time T1 to T2, a page having a relatively large text area 43 as shown in FIG. All the intersection line segments calculated during this period are equal to or less than the threshold TL1, and it is determined that the deskew is ON in ST106 of FIG. 4, and this state is “Non-Threshold Status” 51 in FIG. Since the same page is photographed at time T1 to T2, the same intersection line segment ratio should be calculated ideally. However, in reality, a change in illumination, a slight fluctuation of the paper surface, an image sensor The crossing line segment varies due to noise, randomness of probabilistic Hough transform, and the like (see fluctuation range D1 in FIG. 8). Note that the point P1 at the time T1 does not indicate the intersection line segment immediately after the start of imaging, but is in the middle of a series of imaging.

  Next, at time T2 to T3, the user turns the page (changes the shooting paper). At this time, the intersection line segment gradually increases from point P7 at time T2 to point P10 at time T3. Here, when the point P8 is shifted (increased) to the point P9, the crossing line segment ratio exceeds the threshold value TL1, but is still less than the threshold value TL2, and the process shifts to “Threshold1 Status” 52 in FIG. The state continues. When the point P9 further shifts (increases) from the point P9 to the point P10, the intersection line segment ratio exceeds the threshold TL2, thereby determining that the deskew is OFF (see also the deskew OFF 53 in FIG. 8) in ST106 of FIG. Then, the process returns to “Non-Threshold Status” 51 in FIG.

  Next, at time T3 to T4, a page (same page) in which the graphic area 44 as shown in FIG. During this time, as in the case of the times T1 to T2, the crossing line segment ratio fluctuates (see the fluctuation range D2 in FIG. 8), and the crossing line segment shifts from the point P13 to the point P14. The rate is less than the threshold value TL2, but still exceeds the threshold value TL1, and the process proceeds to “Threshold2 Status” 54 in FIG. 8 to continue the deskew OFF state. When the transition from point P14 to point P15 is performed (increased) immediately after that, the intersection line segment ratio becomes equal to or greater than the threshold value TL2, and the processing returns to “Non-Threshold Status” 51 in FIG. 8 again.

  Next, at time T4 to T5, the page is turned by the user as in the case of time T2 to T3. At this time, the crossing line segment gradually decreases from the point P17 at time T4 to the point P19 at time T5. Here, when the point P17 is shifted (lowered) from the point P17 to the point P18, the intersection line segment ratio is less than the threshold value TL2, but still exceeds the threshold value TL1, and the process proceeds to “Threshold2 Status” 54 in FIG. The OFF state is continued. When the point P18 further shifts (lowers) from the point P18, the intersection line segment ratio becomes equal to or less than the threshold value TL1, thereby determining that the deskew is ON (see also the deskew ON 55 in FIG. 8) in ST106 of FIG. Then, the process returns to “Non-Threshold Status” 51 in FIG.

  Next, at time T5 to T6, as in the case of time T1 to T2, the cross line segment ratio fluctuates (see fluctuation width D3 in FIG. 8), and shifts (rises) from point P23 to point P24. In this case, the intersection line segment ratio exceeds the threshold value TL1, but is still less than the threshold value TL2, and the process proceeds to “Threshold1 Status” 52 in FIG. 8 to continue the deskew ON state. Then, immediately after that, when the point P24 is shifted (lowered) from the point P24, the intersection line segment ratio becomes equal to or less than the threshold value TL1, and the processing returns to the “Non-Threshold Status” 51 in FIG. 8 again.

  As described above, in the determination of whether or not deskew can be performed (ST106), in a series of captured images, the crossing line segment ratio in the current captured image is decreased from the cross line segment ratio of the previous captured image larger than the threshold value TL1, and the threshold value is reached. When it falls below TL1, the second operation mode (deskew ON) is switched to the first operation mode (deskew OFF), while the intersection of the previous captured images in which the intersecting line segment ratio in the current captured image is smaller than the threshold value TL2 When increasing from the line segment ratio and passing the threshold TL2, the first operation mode is switched to the second operation mode. In other words, since the two thresholds TL1 and TL2 are provided with hysteresis to switch between the two operation modes (deskew ON / OFF), when the captured image is sequentially tilt corrected, the original is in a stationary state. It is possible to prevent the switching of the operation mode (deskew ON / OFF) from being improperly performed due to fluctuations in the intersection line segment of the linear component due to some destabilizing factor.

  Here, the two threshold values TL1 and TL2 are used. However, the present invention is not limited to this. For example, the first operation mode is executed when only the threshold value TL2 is used and the intersection line segment ratio is less than the threshold value TL2. A configuration in which the second operation mode is executed when the intersection line segment ratio is equal to or greater than the threshold TL2 is also possible.

  FIG. 9 is a flowchart showing details of the deskew execution feasibility determination (ST106) in FIG. First, the inclination correction angle setting unit 35 sets “Non-Threshold Status” 51 in FIG. 8 as an initial state and sets the deskew OFF (ST301). Subsequently, when the intersection line segment ratio is acquired in ST105 (ST302: YES), it is determined whether or not it is “Non-Threshold Status” (ST303).

  Therefore, when the threshold value is passed in “Non-Threshold Status” (ST304: YES), when the passed threshold value is TL1 (ST305: TL1), it is determined that “Threshold1 Status” 52 in FIG. ST306). On the other hand, when the threshold value passed is TL2 (ST304: TL2), it is determined that it is in “Threshold2 Status” 54 in FIG. 8 (ST307). After that, it returns to ST302 again.

  In ST303, when it is not in “Non-Threshold Status”, it is subsequently determined whether or not it is “Threshold1 Status” (ST308). Therefore, if the threshold value is passed in “Threshold1 Status” (ST309: YES), if the passed threshold value is TL1 (ST310: TL1), it is determined that “Non-Threshold Status” exists (ST311). On the other hand, if the threshold value passed is TL2 (ST310: TL2), after deskew is turned off (see reference numeral 53 in FIG. 8) (ST312), it is determined that “Non-Threshold Status” is set (ST311). After that, it returns to ST302 again.

  If it is not in “Threshold1 Status” in ST308, it is subsequently determined whether it is “Threshold2 Status” (ST313). Therefore, if the threshold value is passed in “Threshold2 Status” (ST314: YES), if the passed threshold value is TL2 (ST315: TL2), it is determined that “Non-Threshold Status” exists (ST316). On the other hand, when the threshold value passed is TL1 (ST315: TL1), after deskew is turned on (see reference numeral 55 in FIG. 8) (ST316), it is determined that the state is “Non-Threshold Status” (ST317). . After that, it returns to ST302 again.

  FIG. 10 is a flowchart showing details of the line segment information statistical processing (ST107) in FIG. First, the inclination correction angle setting unit 35 acquires line segment information about a captured image that is an object of image processing from the parameter storage unit 38 (ST401). This line segment information includes the intersection determination result of each line segment in ST207 of FIG.

  Next, the inclination correction angle setting unit 35 extracts one line segment (undecided line segment) as a determination target line segment from all the line segments in the line segment information in order to determine whether or not each line segment has an intersection. Then (ST402), the presence or absence of intersection is determined with reference to the intersection line segment flag of the determination target line segment (ST404). Note that at least at the start of the line segment information statistical process, there is a determination target line segment for which comparison has not been completed. Therefore, it is determined No in the determination of whether or not the intersection determination of all line segments has been completed (ST403).

  Next, when the inclination correction angle setting unit 35 determines that there is no intersection (intersection line flag = false) for the determination target line segment (ST404: NO), the inclination correction angle setting unit 35 determines the inclination based on the coordinate data of the determination target line segment. The angle θ is calculated (ST405), and the calculation result is added to the histogram information (that is, extracted as an object of statistical processing) (ST406). That is, the frequency corresponding to θ is counted up with the horizontal axis as the tilt angle θ and the vertical axis as the frequency. Thereafter, the determination target line segment is excluded from the extraction candidates in ST402 (ST407), and the process returns to ST402 again.

  On the other hand, when the inclination correction angle setting unit 35 determines that there is an intersection for the determination target line segment (intersection line flag = true) (ST404: YES), ST405 and ST406 are not executed (added to the histogram information). No) The determination target line segment is excluded from the extraction candidates in ST402 (ST407), and the process returns to ST402 again.

  When the determination of the intersection state of all the determination target line segments is finished (ST403: Yes), the inclination correction angle setting unit 35 generates a histogram based on the histogram information added in ST406, and the inclination angle of this histogram The maximum frequency in the distribution is searched (ST408). Then, the tilt correction angle setting unit 35 estimates the retrieved maximum frequency value (for example, the intermediate value of the class interval) as the tilt angle of the captured image (that is, the tilt correction angle) (ST409).

  In this way, when the line segment that is the determination target intersects with other line segments in ST407, the line segment that is the determination target is excluded from the statistical processing target. Noise components (that is, inappropriate line segments that do not match the reference direction of the document) can be effectively removed, and as a result, the tilt angle of the captured image can be accurately estimated. Further, the determination of the presence / absence of the intersection of each line segment can be simplified because the process result of the intersection line segment rate acquisition in ST105 is used.

  Although the present invention has been described based on specific embodiments, these embodiments are merely examples, and the present invention is not limited to these embodiments. For example, in the above-described embodiment, the line segment that intersects with another line segment is excluded from the target of statistical processing, but is excluded when the line segment intersects with a predetermined number of other line segments of 2 or more. It is good. Note that not all of the components of the image processing apparatus according to the present invention and the document reading system including the image processing apparatus according to the present invention shown in the above-described embodiments are necessarily selected as long as they do not depart from the scope of the present invention. It is possible.

  The image processing apparatus according to the present invention and the document reading system including the image processing apparatus can prevent an inappropriate inclination correction from being performed when it is difficult to detect an effective linear component from a captured image with a simple configuration. The present invention is useful as an image processing apparatus that processes an image obtained by reading a document such as a book, and a document reading system including the image processing apparatus.

1 Document Reading System 2 Document Camera (Image Input Device)
3 PC (image processing device)
26 Display Unit 27 Display Data Generation Unit 28 Data Storage Unit 30 Image Inclination Correction Unit 32 Edge Extraction Unit 33 Line Segment Extraction Unit (Linear Extraction Unit)
34 Crossing line ratio calculation unit 35 Inclination correction angle setting unit (inclination angle estimation unit)
36 Image Rotating Unit 41 Photographed Image B Book (Original)

Claims (4)

An edge extraction unit that extracts a plurality of edge pixels in a photographed image obtained by sequentially photographing the paper surface of the document;
A line extraction unit that extracts a plurality of line components based on the edge pixels;
By determining the crossing state of the straight line components, a cross line segment calculating unit that calculates a cross line segment ratio that is a ratio of straight line components crossing each other among the plurality of straight line components;
A tilt angle estimating unit that estimates the tilt angle of the captured image by performing statistical processing on the tilt angles of the plurality of linear components;
An image rotation unit that performs tilt correction of the captured image by rotating the captured image based on the tilt angle estimated by the tilt angle estimation unit;
The image rotation unit has a first operation mode in which the tilt correction is performed and the image rotation unit has a second operation mode in which the tilt correction is not performed, and the intersection line segment ratio is equal to or greater than a predetermined threshold value An image processing apparatus that executes the second operation mode. The threshold is composed of a first value and a second value that is greater than the first value,
In a series of the captured images, when the intersecting line segment ratio in the current captured image decreases from the intersecting line segment ratio of the previous captured image larger than the first value and falls below the first value, While the second operation mode is switched to the first operation mode, the intersecting line segment ratio in the current photographed image is increased from the intersecting line segment ratio in the previous photographed image smaller than the second value. The image processing apparatus according to claim 1, wherein when the second value is exceeded, the first operation mode is switched to the second operation mode.   When the straight line component that is the determination target intersects with one or more other line segments based on the determination of the intersecting state, the tilt angle estimation unit excludes the straight line component that is the determination target from the statistical processing target The image processing apparatus according to claim 1, wherein the image processing apparatus is an image processing apparatus.   4. A document reading system comprising: the image processing apparatus according to claim 1; and an image input apparatus having a camera unit that generates the captured image.

Images