JP6077873B2 - Image processing apparatus and image processing method - Google Patents

Description

  The present invention relates to a technique for reading a document.

  Many reading devices that are currently popular use a scanner to read a document placed downward on a document table, but in recent years, an image is placed on the document table and mounted on the top of the document table. An apparatus for reading a document in a non-contact manner has been proposed. Since the document is facing upward and the image pickup apparatus is non-contact, it is possible to continuously read a multi-split document such as a document bound by a stapler or the like while turning and repositioning the document without separating the document. Patent Document 1 discloses a technique for extracting a straight line from a photographed image and estimating the four sides of the document based on color gradient information in the vertical direction of the straight line.

JP 2012-14430 A

  However, the conventional method cannot estimate a plurality of documents existing in a captured image. Therefore, it is necessary to photograph a plurality of originals printed on both sides one by one. The present invention has been made in view of such a problem, and an object of the present invention is to provide a technique capable of capturing a double-sided page of a document bound by a stapler or the like with a smaller number of times of photographing than before.

In order to achieve the object of the present invention, for example, an image processing apparatus of the present invention includes an acquisition unit that acquires a captured image obtained by capturing two spread pages;
Reference area acquisition means for acquiring a reference area on the captured image;
Among the two spread pages in the captured image, the image in the first page corresponding to the reference area is registered as an image of the rear page in the two spread pages, and among the two spread pages in the captured image The image processing apparatus includes registration means for registering an image in a second page different from the first page as an image of a front page in the two spread pages .

  According to the configuration of the present invention, it is possible to capture a double-sided page of a document bound by a stapler or the like with a smaller number of times of shooting than in the past.

2A and 2B illustrate an example of appearance and a configuration of a system. FIG. 3 is a block diagram showing an example of a hardware configuration of the control apparatus 101. The flowchart of the process which the control apparatus 101 performs. The flowchart which shows the detail of the process in step S312. The figure which shows an example of a picked-up image. The figure explaining the process which estimates the frame of a document area. The figure explaining the process which estimates the frame of a document area. The figure explaining the process which estimates the frame of a document area. The figure explaining the process which estimates the frame of a document area. The figure which shows the image after correction | amendment. The figure which shows the example of a projection. The flowchart which shows the detail of the process in step S312. The figure which shows an example of a picked-up image. The figure explaining the process which estimates the frame of a document area. The figure explaining the process which estimates the frame of a document area. The figure which shows the image after correction | amendment. The figure explaining a coincidence area and a mismatch area.

  Preferred embodiments of the present invention will be described below with reference to the accompanying drawings. The embodiment described below shows an example when the present invention is specifically implemented, and is one of the specific examples of the configurations described in the claims.

[First Embodiment]
First, an example of the appearance of a system according to the present embodiment will be described with reference to FIG. In the present embodiment, a scanned image of a page printed on the front and back of each original is obtained by capturing an image of a multi-paged original placed on the original table each time the original is turned. Generate / acquire.

  The image capturing apparatus 102 is provided to capture an image of a plurality of spelled documents placed on a document table, and the captured image is input to the control apparatus 101. The projection device 103 is configured by a projector or the like. The photographing device 102 and the projection device 103 are connected to the control device 101, and the operation is controlled by the control device 101. The control device 101 controls the operations of the image capturing device 102 and the projection device 103 and generates a scan image of each page of a plurality of spelled documents placed on the document table from the captured images captured by the image capturing device 102. .

  A functional configuration example of the control device 101 is shown as a block diagram in FIG. The configuration illustrated in FIG. 1A is a main configuration used in the following description, and is not necessarily limited to this configuration, and various modifications can be considered for the configuration. For example, the photographing apparatus 102 and the projection apparatus 103 may be housed in the control apparatus 101, or the data storage unit 107 may be externally connected to the control apparatus 101 as an external memory of the control apparatus 101. The description relating to the function of each functional unit shown in FIG. 1 will be made in the description using the flowchart of FIG.

  Further, each of the functional units constituting the control device 101 shown in FIG. 1A may be configured by hardware, but each functional unit excluding the data storage unit 107 may be configured by software. In this case, a device having the data storage unit 107 as a memory and capable of executing this software can be applied to the control device 101. In the present embodiment, each unit excluding the data storage unit 107 is configured by software, and a hardware configuration example of the control device 101 in that case will be described with reference to the block diagram of FIG. The configuration illustrated in FIG. 2 is a main configuration used in the following description, and is not necessarily limited to this configuration, and various modifications can be considered for the configuration.

  The CPU 204 executes processing using computer programs and data stored in the memory 205, thereby controlling the operation of the entire control apparatus 101 and executing each process described later as what the control apparatus 101 performs.

  The memory 205 has an area for temporarily storing computer programs and data loaded from the hard disk 203 and captured images of each frame sequentially transmitted from the imaging apparatus 102. Furthermore, the memory 205 also has a work area used when the CPU 204 executes various processes. That is, the memory 205 can provide various areas as appropriate. The memory 205 also functions as the data storage unit 107 in FIG.

  The hard disk 203 stores computer programs and data for causing the CPU 204 to execute processes described later as those performed by the OS (operating system) and the control device 101. Computer programs and data stored in the hard disk 203 are appropriately loaded into the memory 205 under the control of the CPU 204 and are processed by the CPU 204. Note that part or all of what has been described as being stored in the memory 205 may be stored in the hard disk 203. The hard disk 203 also functions as the data storage unit 107 in FIG.

  Each of the CPU 204, the memory 205, and the hard disk 203 is connected to a common bus 206, and data communication can be performed via this bus 206. The bus 206 is also connected with the above-described photographing apparatus 102 and projection apparatus 103.

  Next, a process performed by the control apparatus 101 for acquiring a scanned image of each page in a document with a plurality of sheets will be described with reference to FIG. 3 showing a flowchart of the process. Note that computer programs and data for causing the CPU 204 to execute processing according to the flowchart of FIG. 3 are stored in the hard disk 203. However, the CPU 204 loads the computer program and data from the hard disk 203 to the memory 205 and executes processing using the computer program and data. As a result, the CPU 204 executes processing according to the flowchart of FIG.

  In step S301, the CPU 204 initializes a variable N used in the following processing to 1. This variable N is a variable used as a page counter indicating the number of the scanned page.

  In step S <b> 302, the CPU 204 functions as the photographing control unit 104 and causes the photographing apparatus 102 to photograph a document placed on the document table. As a result, a photographed image obtained by photographing the document table by the photographing apparatus 102 is input to the memory 205.

In step S <b> 303, the CPU 204 functions as the shooting control unit 104. That is, the CPU 204 captures two frames of the captured image (current frame captured image) stored in the memory 205 in step S302 and the captured image one frame before the current frame (already stored in the memory 205). The following two points are judged.
(1) Whether or not the original is captured in the captured image of the current frame (2) Whether or not the image content has changed between the two frames (whether or not the original is stationary)
Various methods can be considered as a method for determining whether or not a document is included in a photographed image of the current frame. For example, if a “captured image on a document table with no document placed” is acquired in advance and the difference between the captured image and the captured image of the current frame is equal to or greater than a specified amount, the current frame is captured. It is determined that the document is included in the image. Of course, image recognition may be performed on the captured image of the current frame, and it may be determined whether or not a document is included in the captured image. Of course, it may be estimated whether a document exists by actually applying a document extraction process described later.

  Various methods can be considered as a method for determining whether or not the image content is changed between the two frames. For example, if the distance in the RGB color space between the RGB value of each pixel in the captured image of the current frame and the RGB value of each pixel in the captured image one frame before the current frame is greater than or equal to the threshold value, It is determined that there is a change in the image content between the two frames. This may be performed between a plurality of frames. In this case, it is determined that there is a change if the average distance between the plurality of frames is equal to or greater than a threshold value. Note that pixel values other than RGB values may be used. Whether the document is stationary on the document table may be determined using a dedicated sensor such as an infrared sensor.

  As a result of the determination in step S303, if it is determined that the document is still in the captured image of the current frame, the captured image of the current frame is set as the processing target image, and the process proceeds to step S304. On the other hand, if it is determined that the original is not captured in the captured image of the current frame, the process returns to step S302.

  The fact that the document is still in the captured image can be determined not to be in the middle of turning the document but to be after the document has been turned. In the present embodiment, every time a document is turned, the document is photographed to generate a scanned image of a facing page. Therefore, the state of “after turning over the document” is reflected in the photographed image. Required.

  In step S304 and subsequent steps, the captured image of the current frame, that is, the captured image in which the document is stationary is the processing target. However, as long as it is possible to acquire a captured image in which the document is stationary, “a captured image in which the document is stationary” may be acquired by another method. For example, an interface that can be operated by the user of the apparatus and a display screen on which the captured images of the frames sequentially transmitted from the imaging apparatus 102 can be viewed are provided. When the user determines that the “photographed image taken with the document stationary” appears on the display screen, the user operates the interface to input a photographing instruction. When the CPU 204 detects the input of the photographing instruction, the CPU 204 sets the photographed image captured from the photographing apparatus 102 at this time as a processing target image.

  In step S304, the CPU 204 functions as the image processing unit 105, and performs preprocessing for the processing to be described later on the captured image set as the processing target image in step S303. Specifically, after this photographed image is converted into a grayscale image, a known edge enhancement process is performed. Known edge enhancement techniques include techniques such as Sobel and Laplacian filters.

  In step S305, the CPU 204 functions as the image processing unit 105, and extracts a line segment group (edge line group) from the image subjected to edge enhancement in step S304 by a known method such as Hough transform or a minimum approximation method. In the present embodiment, based on mathematical terms, a straight line having both ends is referred to as a line segment, and a straight line having no ends is referred to as a straight line. When a line segment given the same description number is described as a straight line, it refers to a straight line that includes the line segment, that is, shares the slope and intercept with the line segment.

  In step S306, the CPU 204 determines whether or not the value of the variable N is 1. If N = 1 as a result of the current determination, the CPU 204 determines that “the cover page of the document (one page of document) is included in the processing target image”, and the process proceeds to step S307. On the other hand, if N ≠ 1, the CPU 204 determines that “the original document in the state after turning one or more sheets is reflected in the processing target image”, and the process proceeds to step S312.

  In step S307, the CPU 204 functions as the document determination unit 106, and estimates the frame of the document area in the processing target image. As disclosed in JP 2012-14430, a known document frame estimation method classifies line segments into a vertical direction and a horizontal direction, obtains a combination candidate of four sides of a document, and determines a luminance difference between the background and the document. There is a method for estimating the four sides of an original in an image using the fact that it is constant.

  In step S308, the CPU 204 functions as the document determination unit 106, and information indicating the frame estimated in step S307 (for example, image coordinates on the images at the four corners of the frame) is used for the frame of the first page of the document in the captured image. Information is stored in the memory 205.

  In step S309, the CPU 204 functions as the image processing unit 105, extracts the image within the frame estimated in step S307, and stores the extracted image in the memory 205 as a scanned image of the first page of the document.

  In step S310, the CPU 204 functions as the document determination unit 106, and stores in the memory 205 information (for example, image coordinates on the images at the four corners of the frame) that defines the area in the frame estimated in step S307 as the reference document position.

  In the present embodiment, a state where the original is bound, that is, a state where the cover of the original bundle is photographed is referred to as a reference original. From this state, the position of the back side of the document, that is, the document that becomes even pages when the cover is counted as the first page is called a back document.

  In step S311, the CPU 204 increments the value of the variable N by one. Then, the process proceeds to step S317. On the other hand, in step S <b> 312, the CPU 204 functions as the document determination unit 106. Then, the CPU 204 specifies the areas of the two originals in the state where the bound original is opened from the processing target image, and the image in each of the specified areas is the scan image of the Nth page, (N + 1) page The scanned image of the eye is stored in the memory 205. However, there are cases where only the last page is shown in the processing target image. In this case, the area of the last page is specified from the processing target image, and the image in the specified area is stored in the memory 205 as the scan image of the Nth page. To do. Details of the processing in step S312 will be described later with reference to the flowchart of FIG.

  In step S313, the CPU 204 functions as the document determination unit 106 and determines whether or not the document obtained in step S312 is one. If there is one, the process proceeds to step S320, and if not, the process proceeds to step S314.

  In step S314, the CPU 204 functions as the document determination unit 106, and updates the reference document position stored in the memory 205 to the reference document position obtained in step S312.

  In step S <b> 315, the CPU 204 functions as the document determination unit 106, and stores the back side document position obtained in step S <b> 312 in the memory 205 when the back side document position is not yet stored in the memory 205. On the other hand, if the back side document position is already stored in the memory 205, the already stored back side document position is updated to the back side document position obtained in step S312.

  In step S316, the CPU 204 increments the value of the variable N by two. In step S317, the CPU 204 functions as the image processing unit 105, and corrects an image stored as a scanned image in the memory 205 (that is, an unequal square area) into an original rectangular area image, a so-called inverse perspective transformation calculation ( Perform trapezoidal distortion correction). The parameters of the transformation matrix used can be extracted by solving the simultaneous equations obtained by giving the coordinates of the four vertices to the inverse perspective transformation formula as disclosed in Japanese Patent Laid-Open No. 2003-288588. .

  In step S318, the CPU 204 functions as the image processing unit 105, and determines the document direction for the corrected scanned image obtained in step S317. The method of direction determination processing is a known technique, and this is not essential for the present embodiment. Therefore, although detailed explanation is omitted, several characters are recognized by a known character recognition method in a total of four directions with the image rotated to 0 °, 90 °, 180 °, and 270 °, respectively. A method of determining the direction with the highest recognition score at that time as the correct direction is common.

  In step S319, the CPU 204 functions as the image processing unit 105, and rotates the corrected scanned image obtained in step S317 so that the document direction determined in step S318 becomes a specified direction. Thereby, it is possible to obtain a scanned image in which the document direction is a prescribed direction. Then, the CPU 204 stores the rotated scanned image in the memory 205.

  On the other hand, in step S320, the CPU 204 functions as the image processing unit 105, and performs the trapezoidal distortion correction on the image stored as the scan image in the memory 205 in the same manner as in step S317.

  In step S321, the CPU 204 functions as the image processing unit 105, and determines the document direction of the corrected scanned image obtained in step S320 in the same manner as in step S318.

  In step S322, the CPU 204 functions as the image processing unit 105, and rotates the corrected scanned image obtained in step S320 so that the document direction determined in step S321 becomes a specified direction, similarly to step S319. Then, the CPU 204 stores the rotated scanned image in the memory 205.

  In step S323, the CPU 204 functions as the electronic document generation unit 108, and converts the scanned image of each page obtained by the processing so far into one electronic document data reflecting the front / back relationship.

  In step S324, the CPU 204 functions as the electronic document generation unit 108, and stores the electronic document data generated in step S323 in the memory 205 or the hard disk 203. In the present embodiment, an electronic document refers to a file format having a plurality of pages using layout information such as images, characters, and graphics as page data.

  Next, details of the processing in step S312 will be described using the flowchart of FIG. In step S <b> 401, the CPU 204 reads out the reference document position stored in step S <b> 310 or updated in step S <b> 314 from the memory 205.

  In step S402, the CPU 204 estimates the frame of the document area in the processing target image using the reference document position read in step S401. Details of the processing in this step will be described below with a specific example.

  In step S403, the CPU 204 determines whether or not the document area frame has been successfully estimated in step S402. If the determination result is successful, the process proceeds to step S404. If the determination result is not successful, the process proceeds to step S406.

  In step S404, the CPU 204 stores information indicating the frame estimated in step S402 (for example, image coordinates on the image at the four corners of the frame) as information on the frame of the (N + 1) th page of the document in the processing target image. Stored in 205.

  In step S405, the CPU 204 extracts the image within the frame estimated in step S402, and stores the extracted image in the memory 205 as a scanned image of the (N + 1) th page document.

  In step S <b> 406, the CPU 204 determines whether the back side document position is stored in the memory 205. As a result of this determination, if it is stored, the process proceeds to step S407, and if it is not stored, the process proceeds to step S408.

  In step S <b> 407, the CPU 204 estimates the frame of another document area (turned back page) in the processing target image using the back document position stored in the memory 205.

  On the other hand, in step S408, the CPU 204 uses the information indicating the frame of the (N + 1) th page of the document stored in the memory 205 in step S404 to use another document area (turned back page) in the processing target image. ) Is estimated. Details of the processes in steps S407 and S408 will be described below with a specific example.

  In step S409, the CPU 204 uses information indicating the frame estimated in step S407 or step S4082 (for example, image coordinates on the image at the four corners of the frame) as information on the frame of the Nth page of the document in the processing target image. Store in the memory 205.

  In step S410, the CPU 204 extracts the image within the frame estimated in step S407 or step S4082, and stores the extracted image in the memory 205 as a scanned image of the Nth page document.

  Next, each process described with reference to FIGS. 3 and 4 will be described with specific examples. In the following specific example, the cover (first page), the second page (the back of the cover), the third page (the second document), the fourth page (3) A case where each scanned image on the back side of the page is acquired is taken as an example. In the following description, the photographed image to be processed first in step S304 is an image photographed by placing the document cover face up on the document table as shown in FIG. . As shown in FIG. 5B, the photographed image to be processed for the second time in step S304 turns one original from the state shown in FIG. 5A, and the second and third pages are spread pages. It is the image image | photographed in the open state. As shown in FIG. 5C, the photographed image to be processed for the third time in step S304 is an image obtained by turning one document from the state shown in FIG. 5B and photographing the fourth page as the final page. It is.

  First, in step S301, the variable N is initialized to 1, and in step S302, the document in the state shown in FIG. As a result, a photographed image as shown in FIG. 5A is obtained. In steps S304 and S305, preprocessing and Hough transform are performed on the photographed image shown in FIG. Assume that an image as shown in FIG. 6A is obtained as a result of the Hough transform, and line segments (edge lines) 601, 602, 603, and 604 are extracted.

  In step S306, since N = 1, it is determined that the first page, that is, only the cover is a one-page document, and the process proceeds to step S307. In step S307, a document frame (a quadrangle with line segments as four sides) in the photographed image shown in FIG. 5A is extracted. In the method described in Patent Document 2, the vertical component and the horizontal component are classified based on the slope of the straight line in the image. Line segments 601 and 603 are classified as vertical line segments, and line segments 602 and 604 are classified as horizontal line segments. Then, two vertical line segments and two horizontal line segments are combined (actually, four straight lines obtained by extending these four line segments are combined) to form a quadrangle. FIG. 6B shows an example of a quadrangle composed of straight lines. In this case, since only the quadrangle 605 (shaded portion) formed from the vertical line segments 601 and 603 and the horizontal line segments 602 and 604 is obtained as a combination, the quadrangle 605 is set as a document area (document square). When there are a plurality of combinations of rectangles obtained from vertical lines and horizontal lines, in the present embodiment, the combination that maximizes the area of the rectangle is estimated as a document rectangle.

  In step S308, information defining the rectangle 605 is stored in the memory 205 as a document frame. In step S309, an image within the rectangle 605 in the photographed image, that is, an image of an area where the cover of the document is shown in the photographed image is extracted, and the extracted image is stored in the memory 205 as a scan image of the first page of the document. Store. In step S310, information defining the rectangle 605 is stored in the memory 205 as a reference document position. In step S311, the value of variable N is incremented by one.

  In step S317, the reverse perspective transformation calculation (trapezoidal distortion correction) is performed on the image stored in the memory 205 in step S309, that is, the image of the area where the cover of the document is shown in the photographed image. As a result, the image within the quadrangle 605 in the captured image is converted into a rectangular image 1001 as shown in FIG. This conversion matrix can be fixed if the position, focal length, and optical axis direction of the photographing apparatus 102 with respect to the document are known. If the position of the photographing apparatus 102 is fixed, the conversion matrix is obtained in advance. You can leave it.

  In step S318, the document orientation of the corrected scanned image obtained in step S317 is determined. Here, it is assumed that 0 ° clockwise rotation, that is, no rotation is determined as the correct direction. In step S319, the corrected scanned image is stored in the memory 205 as the scanned image of the first page of the document without rotating the corrected scanned image.

  The scanned image of the cover (first page) of the document is obtained so far, and the user then turns the cover of the document and opens the second and third pages as shown in FIG. Put it in a state. If it is determined in step S303 that the document is still in the captured image of the current frame (the operation for opening the cover has been completed), the process proceeds to step S304.

  In steps S304 and S305, preprocessing and Hough transform are performed on the captured image (FIG. 5B) obtained in step S302. Assume that an image as shown in FIG. 7A is obtained as a result of the Hough transform, and line segments (edge lines) 701 to 717 are extracted.

  Since N = 2 in step S306, the process proceeds to step S312. In step S312, the Nth page image and the (N + 1) th image are extracted from the photographed image of FIG. In step S312, as described above, the processing of steps S401 to S410 in FIG. 4 is executed.

  In step S <b> 401, the reference document position (information defining the rectangle 605 and information defining the cover frame) stored in step S <b> 310 is read from the memory 205.

  In step S402, the frame of the document area of the third page in the processing target image is estimated using the reference document position read in step S401. Specifically, as shown in FIG. 7B, a line segment is selected from a range of an area 720 that is equal to or less than a distance threshold T from each of the four sides of the frame defined by the reference document position. If the four line segments do not exist in the region 720, the neighborhood range is expanded until the four line segments are found while gradually increasing the value of T. For example, the threshold value T may be increased by 1% with an initial value of 1% of the length of the short side of the frame defined by the reference document position. Here, since four line segments 709, 710, 711, and 717 were found inside the region 720, an unequal quadrangle 721 obtained by combining four straight lines including these four line segments as four sides (FIG. 7 (c) )) Is estimated as the frame of the document area of the third page. Note that the method of finding four line segments using the region 720 is an example, and a method of calculating the degree of proximity of the position with respect to each side of the reference original of each line segment and combining them in order from the highest score is used. May be.

  In step S403, since the document area frame has been successfully estimated in step S402, the process proceeds to step S404.

  In step S <b> 404, information indicating the frame of the document area of the third page (for example, image coordinates on the image at the four corners of the frame) is stored in the memory 205.

  In step S405, an image within the frame (rectangle 605) of the document area estimated in step S402 is extracted (image within the frame of the document area on the third page), and the extracted image is scanned on the document on the third page. The image is stored in the memory 205 as an image.

  In step S406, since the back side document position is not yet stored in the memory 205 at the time of N = 2, the process proceeds to step S408.

  In step S408, the frame of the document area of the second page is estimated using the information indicating the frame of the document of the third page stored in the memory 205 in step S404. The document area frame estimation method in step S408 will be described with reference to FIG.

  First, two straight lines passing through the internal angles 801, 802, 803, and 804 of the document area on the third page are searched from the straight lines including the detected line segments. Since the corners do not overlap completely depending on the bound position, a distance from the corner to the radius R is allowed. If not found, the radius R is gradually increased. The radius R is, for example, 1% of the length of the short side of the reference document and may be widened in increments of 1%. As a result, two straight lines 702 and 708 were found as candidates for two of the four sides. Further, this search method is an example, and a method of narrowing down combinations in ascending order using the shortest distance between the straight line and the corner as a score may be used. Next, the document area is estimated together with the remaining two-side candidate line segments from the area 810 sandwiched between the two sides. In the area 810, the document frame may be estimated by the known document estimation technique described above. The area 810 includes eight line segments 701, 702, 703, 704, 705, 706, 707, and 708. From these, a combination of unequal-sided rectangles including 702 and 708 as two sides is used as a document. Guess the area. Here, a method is used in which the four sides having the maximum area are estimated as the frame of the document area. First, fifteen combinations combining six lines to two lines are all the frame candidates for the document area. Among them, there are three types including a combination of 701 and 705, a combination of 701 and 708, and a combination of 703 and 706 that include line segments on the four sides of the unequal side rectangle. As the largest area, an unequal side quadrangle 820 in FIG. 8C formed from a combination of 701 and 705 was estimated as a frame of the document area.

  In step S409, information indicating the unequal side rectangle 820 (frame of the document area of the second page) (for example, image coordinates on the image at the four corners of the frame) is used as information of the frame of the document of the second page in the processing target image. Is stored in the memory 205.

  In step S410, the CPU 204 extracts an image within the unequal side rectangle 820, and stores the extracted image in the memory 205 as a scan image of the second page of the document.

  Returning to FIG. 3, since the second page and the third page are detected in step S313, the process proceeds to step S314. In step S314, the reference document position is updated to an unequal side rectangle 721 (frame of the document area on the third page). In step S315, the back side document position is updated to an unequal side rectangle 820 (frame of the document area of the second page). In step S316, the value of the variable N is incremented by two. As a result, N = 4.

  In step S317, the above-described inverse perspective transformation calculation (trapezoidal distortion correction) is performed on the scan image on the second page and the scan image on the third page, so that each scan image is an image 1002, FIG. The image is converted into an image 1003 in FIG.

  In step S318, the image 1002 is determined to be rotated in the clockwise direction by 270 degrees, and the image 1003 is determined to be rotated in the clockwise direction by 0 degrees. In step S319, the image 1002 is rotated 270 degrees counterclockwise, and the rotated image is stored in the memory 205 as a scanned image of the second page of the document. In step S319, the image 1003 is stored in the memory 205 as a scanned image of the third page without being rotated.

  Up to this point, since the scanned images of the second and third pages of the document have been obtained, the user then turns the second document and, as shown in FIG. 5C, the last page faces upward on the document table. To the state where it is mounted. If it is determined in step S303 that the original is still in the captured image of the current frame (the operation of turning the second original has been completed), the process proceeds to step S304.

  In steps S304 and S305, preprocessing and Hough transform are performed on the captured image (FIG. 5C) obtained in step S302. As a result of the Hough transform, it is assumed that line segments (edge lines) 901 to 904 are extracted as shown in FIG.

  Since N = 4 in step S306, the process proceeds to step S312. In step S312, an Nth page image is extracted from the captured image of FIG. In step S312, as described above, the processing of steps S401 to S410 in FIG. 4 is executed.

  In step S <b> 401, the reference document position updated in step S <b> 314 (information defining the frame of the unequal rectangle 721) is read from the memory 205.

  In step S402, the frame of the document area of the fourth page in the processing target image is estimated using the reference document position read in step S401. As shown in FIG. 9A, since there are no four line segments in the area 910 that is equal to or smaller than the distance threshold T from each of the four sides of the frame defined by the reference document position, the frame of the document area cannot be estimated.

  In step S403, since the frame of the document area has not been successfully estimated in step S402, the process proceeds to step S406. In step S406, since the back side document position is stored in the memory 205, the process proceeds to step S407.

  In step S407, the frame of the fourth page of the document area in the processing target image is estimated using the back side document position. The estimation method may be the same method as in step S402. As shown in FIG. 9B, four line segments 901 to 904 are selected from the range of the area 920 that is equal to or less than the distance threshold T from each of the four sides of the frame defined by the back side document position. Then, an unequal side rectangle 930 (FIG. 9C) obtained by combining four straight lines including these four line segments as four sides is estimated as the frame of the document area of the fourth page.

  In step S409, information indicating an unequal side rectangle 930 (frame of the document area of the fourth page) (for example, image coordinates on the image at the four corners of the frame) is used as information of the frame of the document of the fourth page in the processing target image. Is stored in the memory 205.

  In step S410, an image within the unequal side rectangle 930 (frame of the document area of the fourth page) is extracted, and the extracted image is stored in the memory 205 as a scan image of the document of the fourth page.

  Returning to FIG. 3, in step S313, since only the fourth page is detected, the process proceeds to step S320. In step S320, in the same manner as in step S317, the above-mentioned trapezoidal distortion correction is performed on the “scan image of the fourth page of the document” stored as a scan image in the memory 205, and as a result, as shown in FIG. A corrected scan image 1004 is obtained.

  In step S321, in the same manner as in step S318, the document orientation is determined for the corrected scanned image 1004 obtained in step S320. As a result, since it is determined that it is only necessary to rotate 270 degrees clockwise, in step S322, the corrected scan image 1004 is rotated 270 degrees clockwise, and the rotated scan image is stored in the memory 205.

  In step S323, the scanned images for four pages obtained by the processing so far are processed in the order of the first page scanned image, the second page scanned image, the third page scanned image, and the fourth page scanned image. Convert to a single electronic document data. In step S324, the electronic document data generated in step S323 is stored in the memory 205 or the hard disk 203.

  FIG. 11 shows an example in which the scanned images for four pages obtained in this way are projected onto an appropriate surface using the projection device 103. In the area 1101 on the projection screen, thumbnail images of the scanned images for four pages obtained by the above processing are displayed in a list. Of these four pages of scan images, the scan image selected by the user using an operation unit (not shown) is enlarged and displayed in the area 1102, and the user can view the scan image or edit the operation image by appropriately operating the operation unit. You can do it. The display method and display form are not limited to the display method and display form shown in FIG.

  As described above, according to the present embodiment, an original document bound by a stapler or the like is placed on an original table and the original can be easily captured as an electronic document only by turning the page.

  In the present embodiment, the flow of processing has been described with a specific example, but this is only an example of a basic configuration described below. According to the basic configuration, an imaging unit provided for imaging a multi-page document placed on a document table acquires a captured image of the document that is captured each time one document is turned. . Here, a captured image captured by the imaging unit p (p is a natural number of 2 or more) times is defined as a captured image p. At this time, for the captured image p, an area is set at the same position as the original area of the (2p-3) -th page specified from the captured image (p-1), and from the set area (2p-1) Specify the document area of the page. Further, an image in the specified document area is acquired as a document image of the (2p-1) th page (first acquisition).

  If the document area of the (2p-4) th page from the captured image (p-1) is not specified, the document area from the (2p-1) th page of the captured image p in the captured image p is displayed. (2p-2) The document area of the page is specified (first specification).

  On the other hand, when the document area of the (2p-4) th page from the captured image (p-1) is specified, the area at the same position as the document area of the (2p-4) page with respect to the captured image p. And the document area of the (2p-2) th page is specified from the set area (second specification).

  Then, the image in the document area of the (2p-2) th page specified by the first specification or the second specification is acquired as the image of the (2p-2) page (second acquisition). However, various modified examples can be considered as long as it results in such a basic configuration or a configuration equivalent to this.

[Second Embodiment]
In the first embodiment, the document bundles are all processed with the same document size, but by comparing the difference between the previous and next pages, it is possible to process even if documents of different sizes are mixed in the document bundle. . Hereinafter, only differences from the first embodiment will be described, and unless otherwise noted, the same as the first embodiment. However, also in this embodiment, the system shown in FIGS.

  Also in this embodiment, the control apparatus 101 executes the process according to the flowchart of FIG. 3, but in step S312, the process according to the flowchart of FIG. 12 is executed instead of the process according to the flowchart of FIG. .

  In FIG. 12, steps S1201, S1202, S1208, S1210, S1211, and S1216 are the same processes as steps S401, S402, S405, S408, S407, and S410, respectively. Therefore, the description regarding each of step S1201, S1202, S1208, S1210, S1211, and S1216 is abbreviate | omitted.

  In step S1203, the CPU 204 functions as the document determination unit 106 to determine whether a document frame has been extracted at the reference document position. If it has been extracted, the process proceeds to step S1204. If not, the process proceeds to step S1203. The process proceeds to S1209.

  In step S1204, the CPU 204 functions as the image processing unit 105, and performs distortion correction by reverse perspective transformation on the image in the document frame extracted in step S1203 for comparison of differences. It is not necessary to correct the trapezoidally corrected image in the subsequent step S317.

  In step S1205, the CPU 204 functions as the image processing unit 105, obtains a difference between the image subjected to the distortion correction processing in step S1204 and the scanned image two pages before, and determines a matching area and a non-matching area in each image. Identify. In addition, foregrounds such as an image area, a character area, and a graphic area may be extracted by a known document analysis technique, and a matching area and a mismatching area may be determined between two images from the position shape of each object. As a known image analysis technique, for example, there is a method disclosed in US Pat. No. 5,680,478.

  In step S1206, the CPU 204 functions as the document determination unit 106, and determines whether or not there is a matching portion (matching region) between the two images from the difference obtained in step S1205. As a result of this determination, if there is a matching area, the process proceeds to step S1207, and if there is no matching area, the process proceeds to step S1208.

  In step S <b> 1207, the CPU 204 functions as the document determination unit 106, and stores the remaining area image obtained by removing the matching area from the scanned image two pages before in the memory 205 as a new “scanned image two pages before”. .

  In step S1209, the CPU 204 determines whether or not the back side document position is already stored in the memory 205. As a result of this determination, if it is stored, the process proceeds to step S1211. If it is not stored, the process proceeds to step S1210.

  In step S <b> 1212, the CPU 204 functions as the image processing unit 105, and performs distortion correction by reverse perspective transformation for the N-th page scan image obtained based on the back side document position for the difference comparison. It is not necessary to correct the trapezoidally corrected image in the subsequent step S317.

  In step S1213, the CPU 204 functions as the image processing unit 105, obtains a difference between the image subjected to the distortion correction processing in step S1212 and the scanned image two pages before, and a matching area and a non-matching area in each image. Is identified.

  In step S1214, the CPU 204 functions as the document determination unit 106, and determines whether or not there is a matching portion (matching region) between the two images from the difference obtained in step S1213. As a result of the determination, if there is a matching area, the process proceeds to step S1215. If there is no matching area, the process proceeds to step S1216.

In step S1215, the CPU 204 functions as the document determination unit 106, and specifies a frame of the remaining area obtained by removing the matching area from the image subjected to the distortion correction process in step S1212.

  Below, the process according to the flowchart of FIG.3, 12 is demonstrated, giving a specific example. First, the photographed image of FIG. 5A is obtained, and then the second photographed image is photographed with the second page and the third page smaller in size than the second page opened. Assume that a captured image (FIG. 13A) is obtained. Then, as the third photographed image, a photographed image photographed in a state where the fourth page, which is the back surface of the third page, and the fifth page having a size larger than the fourth page are opened (FIG. 13). Suppose that (b)) is obtained. Finally, assume that the captured image of FIG. 5C is obtained as the fourth captured image. A process according to the flowcharts of FIGS. 3 and 12 when four photographed images are obtained in this order will be described.

  First, in step S301, the variable N is initialized to 1, and in step S302, the captured image shown in FIG. 5A is obtained. The processing in the case of N = 1 (S303 to S311, S317 to S319) is as described in the first embodiment. However, when the processing in the case of N = 1 (S303 to S311, S317 to S319) is completed, the square 605 of FIG. 6B is obtained as the reference document position, and the scanned image of the first page is shown in FIG. ) Image 1001 is obtained.

  When N = 2, the captured image obtained in step S302 is the image shown in FIG. Thereafter, when the processing of steps S304 and S305 is performed on this image, line segments (edge lines) 1401 to 1410 are obtained as shown in FIG. Since N = 2, the process proceeds to step S312 via step S306. In step S312, processing according to the flowchart of FIG. 12 is performed.

  In steps S1201 and S1202, the frame of the document area is estimated from the photographed image shown in FIG. 13A using the reference document position (frame of rectangle 605). As a result, as shown in FIG. 14B, the frame of the area 1411 is obtained as the frame of the document area. Since the frame of the document area is obtained, the process proceeds to step S1204 via step S1203. In step S1204, an image within the frame of the document area is extracted, and the reverse perspective transformation is performed on the extracted image to obtain the image shown in FIG.

  In step S1205, the RGB value difference between the corresponding pixel positions in the image of FIG. 16A and the image of FIG. Then, an area occupied by pixel positions where the obtained difference is equal to or greater than the threshold (mismatch area) and an area occupied by pixel positions where the obtained difference is less than the threshold (matching area) are specified. Here, since there is no matching area in each image, the process proceeds to step S1208 via step S1206.

  In step S1208, the image of FIG. 16A is stored in the memory 205 as the scanned image of the (N + 1) th page (third page). Since the back side document position is not yet stored in the memory 205, the process proceeds to step S1210 via step S1209.

  In step S1210, a straight line passing through the corner of the area 1411 in FIG. 14B is detected, and the frame of the document area is estimated from the area in between, so that the frame of the area 1412 shown in FIG. Obtained as a frame of the region.

  In step S1216, an image in the area 1412 is extracted. The extracted image is stored in the memory 205 as an Nth page (second page) scanned image.

  Returning to FIG. 3, since the number of detected documents is two, the process proceeds to step S314 via step S313. In step S314, information indicating the frame of the area 1411 is stored in the memory 205 as the reference document position of the (N + 1) th page (third page), thereby updating the reference document position in the memory 205. In step S 315, information indicating the frame of the area 1412 is stored in the memory 205 as the N-th page (second page) back side document position in the memory 205. In step S316, since the value of the variable N is incremented by 2, N = 4.

  In step S317, the image shown in FIG. 16B is obtained by performing reverse perspective transformation on the scanned image of the second page stored in the memory 205 in step S1216. Since the third page has already been corrected in step S1204, it is excluded from the processing target in step S317. In step S318, the direction of the corrected scanned image of the second page is determined, and 0 degree rotation is determined as the correct direction. The direction determination is similarly performed on the scanned image of the third page. In step S319, the respective images are appropriately rotated in accordance with the direction discrimination result, whereby the image of FIG. 16A is obtained as the final scan image of the third page, and the final scan image of the second page is shown in FIG. The image (b) is registered in the memory 205.

  When N = 4, the captured image obtained in step S302 is the image shown in FIG. Thereafter, when the processing of steps S304 and S305 is performed on this image, line segments (edge lines) 1501 to 1510 are obtained as shown in FIG. Since N = 4, the process proceeds to step S312 via step S306. In step S312, processing according to the flowchart of FIG. 12 is performed.

  In steps S1201 and S1202, the frame of the document area is estimated from the photographed image shown in FIG. 13B using the frame of the area 1411 indicated by the reference document position. As a result, as shown in FIG. 15B, the frame of the area 1511 is obtained as the frame of the document area. Since the frame of the document area is obtained, the process proceeds to step S1204 via step S1203.

  In step S1204, an image within the frame of the document area is extracted, and the reverse perspective transformation is performed on the extracted image to obtain the image shown in FIG.

  In step S1205, the RGB value difference between the corresponding pixel positions in the image of FIG. 16C and the image of FIG. Then, an area occupied by pixel positions where the obtained difference is equal to or greater than the threshold (mismatch area) and an area occupied by pixel positions where the obtained difference is less than the threshold (matching area) are specified. Here, as shown in FIG. 17A, a region 1701 as a matching region and a region 1702 as a non-matching region are specified for the image in FIG. However, the process proceeds to step S1207 via step S1206.

  In step S1207, an image of the remaining area (FIG. 16 (e)) obtained by removing the matching area from the image of FIG. 16 (a) is generated, and the image already stored in the memory 205 as the scanned image of the third page is generated. Replace with the generated image.

  In step S1208, the image of FIG. 16C is stored in the memory 205 as the scanned image of the (N + 1) th page (fifth page). Since the back side document position is already stored in the memory 205, the process proceeds to step S1211 via step S1209.

  In step S1211, the frame of the document area is estimated from the photographed image shown in FIG. 13B using the frame of the area 1412 indicated by the back side document position. As a result, as shown in FIG. 15C, the frame of the area 1512 is obtained as the frame of the document area. In step S1212, an image within the frame of the document area is extracted, and the reverse perspective transformation is performed on the extracted image to obtain the image shown in FIG.

  In step S1213, the RGB value difference between the corresponding pixel positions in the image of FIG. 16D and the image of FIG. Then, an area occupied by pixel positions where the obtained difference is equal to or greater than the threshold (mismatch area) and an area occupied by pixel positions where the obtained difference is less than the threshold (matching area) are specified. Here, as shown in FIG. 17B, a region 1703 as a matching region and a region 1704 as a non-matching region are specified for the image of FIG. 16D. However, the process proceeds to step S1215 via step S1214.

  In step S1215, the remaining area frame (the image frame in FIG. 16F, that is, the mismatch area frame) obtained by removing the matching area from the image in FIG. 16D is specified. In step S1216, the image within the frame specified in step S1215 (FIG. 16F) is extracted, and the extracted image is stored in the memory 205 as the scan image of the Nth page (fourth page).

  Returning to FIG. 3, since the number of detected documents is two, the process proceeds to step S314 via step S313. In step S314, information indicating the frame of the area 1511 is stored in the memory 205 as the reference document position of the (N + 1) th page (fifth page), thereby updating the reference document position in the memory 205. In step S315, information indicating the frame of the area 1512 is stored in the memory 205 as the back side document position of the Nth page (fourth page). In step S316, since the value of the variable N is incremented by two, N = 6.

  Since the scanned image for page 4 and the scanned image for page 5 are both corrected, they are excluded from the processing target in step S317. In step S318, direction discrimination is performed on each scanned image. In step S319, each image is appropriately rotated according to the direction discrimination result. As a result, the image shown in FIG. 16F is registered in the memory 205 as the final scan image of the fourth page, and the image shown in FIG. 16C is registered as the final scan image of the fifth page. It will be.

  When N = 6, the captured image obtained in step S302 is the image shown in FIG. In steps S1212 and S1213 of the flowchart of FIG. 12, the matching region and the mismatching region are specified by obtaining the RGB value difference between the pixel positions as described above in the image of FIG. 10D and the image of FIG. As a result, it is determined that there is no matching area. The other processes are as described in the first embodiment, and a description thereof will be omitted. Then, an image obtained by rotating the image of FIG. 10D clockwise by 270 degrees is registered in the memory 205 as the scan image of the sixth page. In step S324, an electronic document is generated in which FIG. 10A, FIG. 16B, FIG. 16E, FIG. 16F, FIG. 16C, and FIG. The

  As described above, according to the present embodiment, it is possible to easily convert documents with different page sizes into electronic documents as well. Note that the first embodiment and the second embodiment may be used by switching appropriately. For example, the user may select and execute either the electronic document generation process described in the first embodiment or the electronic document generation process described in the second embodiment using an operation unit (not illustrated). I do not care.

(Other examples)
The present invention can also be realized by executing the following processing. That is, software (program) that realizes the functions of the above-described embodiments is supplied to a system or apparatus via a network or various storage media, and a computer (or CPU, MPU, or the like) of the system or apparatus reads the program. It is a process to be executed.

Claims (19)

An acquisition means for acquiring a captured image obtained by capturing two spread pages;
Reference area acquisition means for acquiring a reference area on the captured image;
Among the two spread pages in the captured image, the image in the first page corresponding to the reference area is registered as an image of the rear page in the two spread pages, and among the two spread pages in the captured image Registration means for registering an image in a second page different from the first page as an image of a front page in the two spread pages;
An image processing apparatus comprising: The reference area acquisition unit acquires a region of a page on the rear side as the reference region in two spread pages in the preceding captured image acquired by the acquisition unit prior to the captured image. An image processing apparatus according to 1. The said reference area acquisition means acquires the area | region of this 1 page as the said reference area in the preceding picked-up image of 1 page acquired before the said captured image by the said acquisition means. Image processing device. The registering unit registers an image in the first page of the two spread pages in the captured image as an odd page image, and sets the image in the second page of the two spread pages as an even page. The image processing apparatus according to claim 1, wherein the image processing apparatus is registered as an image. The image processing apparatus according to claim 1, wherein the registration unit performs registration after correcting the shape and orientation of an image to be registered. Furthermore,
The image processing apparatus according to claim 1, further comprising a holding unit that holds an image registered by the registration unit. Furthermore,
The image processing apparatus according to claim 6, further comprising a generation unit that generates an electronic document in which images held by the holding unit are arranged in page order. The image processing apparatus according to claim 1, wherein the acquisition unit acquires a captured image captured in a state where the two facing pages are stationary on a document table. An image processing method performed by an image processing apparatus,
An acquisition step in which the acquisition unit of the image processing apparatus acquires a captured image obtained by capturing two spread pages;
A reference area acquisition step of acquiring a reference area on the captured image by a reference area acquisition unit of the image processing apparatus;
The registration means of the image processing device registers an image in the first page corresponding to the reference region among the two spread pages in the captured image as an image of a page on the rear side in the two spread pages, A registration step of registering an image in a second page different from the first page among the two spread pages in the captured image as an image of a front page in the two spread pages;
An image processing method comprising: Means for obtaining an image of the original image captured by an imaging unit provided for imaging a multi-page original placed on the original table each time one original is turned;
The captured image captured by the imaging unit p (p is a natural number equal to or greater than 2) is defined as a captured image p, and the (2p-3) th page specified from the captured image (p-1) with respect to the captured image p An area is set at the same position as the original area, and the (2p-1) th page original area is specified from the set area, and the image in the specified original area is the (2p-1) th page original. First acquisition means for acquiring an image;
If the document area of the (2p-4) th page from the captured image (p-1) is not specified, the (2p-1) th page of the document area in the captured image p is (2p) in the captured image p. -2) first specifying means for specifying the document area of the page;
When the document area of the (2p-4) th page from the captured image (p-1) is specified, an area is set at the same position as the document area of the (2p-4) page with respect to the captured image p. A second specifying unit for specifying the document area of the (2p-2) th page from the set area;
Second acquisition means for acquiring an image in the original area of the (2p-2) th page specified by the first specifying means or the second specifying means as an image of the (2p-2) page; An image processing apparatus comprising: Furthermore,
Selecting four edge lines constituting a quadrangular area from the edge line group in the captured image first captured by the imaging unit, and acquiring the area composed of the selected edge lines as a document area of the first page ; 11. The image processing apparatus according to claim 10 , further comprising means for extracting the acquired image in the document area as an image of the first page . Furthermore,
The image processing apparatus according to claim 10 or 11, characterized in that it comprises a correcting means for performing keystone correction for the first acquisition means and the second acquisition means acquired image. Furthermore,
13. The image processing apparatus according to claim 12 , further comprising means for discriminating a document direction in the image corrected by the correcting unit and registering the image in the memory after rotating the image according to the determined document direction. . Furthermore,
14. The image processing apparatus according to claim 13 , further comprising means for projecting and displaying a list of images registered in the memory. Furthermore,
An area in which the pixel value difference between the (2p-1) page image and the (2p-3) page image is smaller than a threshold value is specified, and the specified area from the (2p-3) page image the image processing apparatus according to any one of claims 10 to 14, characterized in that it comprises a means for removing. Furthermore,
An area in which the difference in pixel value between the (2p-2) page image and the (2p-4) page image is smaller than a threshold value is specified, and the specified area from the (2p-2) page image the image processing apparatus according to any one of claims 10 to 15, characterized in that it comprises a means for removing. An image processing method performed by an image processing apparatus,
The means for acquiring a captured image of the image processing apparatus captures an image captured each time a document is turned by an imaging unit provided to capture a plurality of spelled documents placed on a document table. Obtaining a captured image of the document;
The first acquisition unit of the image processing apparatus uses the captured image captured by the imaging unit p (p is a natural number of 2 or more) as the captured image p, and the captured image (p−1) with respect to the captured image p. An area is set at the same position as the original area of the (2p-3) th page specified from (2p-3), and the original area of the (2p-1) th page is specified from the set area, and within the specified original area A first acquisition step of acquiring the image of (2p-1) as a document image of page (2p-1);
When the first specifying unit of the image processing apparatus has not specified the document area of the (2p-4) th page from the captured image (p-1), the (2p-1) page in the captured image p. A first specifying step of specifying an original area of the (2p-2) th page in the captured image p from the original area of the eye;
When the second specifying unit of the image processing apparatus specifies the document area of the (2p-4) th page from the captured image (p-1) to the captured image p, (2p-4) A second specifying step of setting an area at the same position as the original area of the page and specifying the original area of the (2p-2) page from the set area;
The second acquisition unit of the image processing apparatus selects the image in the document area of the (2p-2) th page specified in the first specifying step or the second specifying step as the (2p-2) th page. A second acquisition step of acquiring the first image as an image of the first image processing method. A computer program for causing a computer to function as each unit of the image processing apparatus according to any one of claims 1 to 8 and 10 to 16 . A computer-readable storage medium storing the computer program according to claim 18.

Images