JP6426815B2 - Image processing apparatus and image processing method - Google Patents

Description

  The present invention relates to a captured image processing system that outputs an image captured by an imaging device by an image output device.

  Along with the progress of the Internet technology, opportunities for storing images captured using mobile terminal devices such as mobile phones are increasing. In addition to simply taking a landscape, a person, etc. as subjects, the opportunity to photograph slides of explanatory diagrams and explanatory texts exhibited at various shows and the like, or academic conferences and the like is also increasing. When saving an image captured by such a portable terminal device, it is preferable to use a file name that is automatically added based on information such as the shooting date, or to create a file name and save it yourself. It is common.

  Further, Patent Document 1 discloses a technique for obtaining an outline of an image from image data captured into a storage unit by imaging, and obtaining a shape of an imaging object from the obtained outline. Then, projection parameters between the determined shape of the imaging object and the actual imaging object are determined, and image conversion is performed using the projection parameters.

Unexamined-Japanese-Patent No. 2005-122320 (May 12, 2005 publication)

  However, the technique disclosed in Patent Document 1 is to obtain the shape of the object to be imaged from the image data taken into the storage unit by imaging, and when the desired shape can not be extracted, it is necessary to perform imaging again . Moreover, even if a desired shape can be extracted, when a part of the imaging target object exists outside the imaging range, a part of the target object can not be imaged.

  The present invention has been made to solve the above problems, and it is possible to capture image data into a storage unit after easily confirming what position the object is in within the imaging range. An object is to provide an apparatus, a captured image processing system, a program and a recording medium.

  In order to solve the above-mentioned subject, the image processing device of the present invention is provided with the display means which displays the picturized picture which picturized the subject, the above-mentioned subject is a rectangle, and the above-mentioned pictorial image determines the output subject picture And the second captured image determined as the output target image, and before the output target image is determined, the target of the first captured image is determined. The first correction processed image in which the object range is geometrically corrected to a predetermined rectangular shape is displayed on the display means, and the output object image is determined, and then the object range of the second captured image is specified. And a display processing unit configured to display the image after the second correction processing geometrically corrected in the rectangular shape on the display unit.

  According to the above configuration, the image after the first correction processing is displayed on the display unit before the output target image is determined, and the image after the second correction processing is displayed on the display unit after the output target image is determined. Display. Therefore, it is possible to display (preview display) an image after correction processing when geometric correction is performed on the captured image before and after the output target image is determined.

  Further, in the image processing apparatus according to the present invention, the display processing unit displays the first captured image and the image after the first correction processing in the same screen, and the second captured image and the second correction processing It is preferable to display the image on the same screen.

  According to the above configuration, the captured image and the image after correction processing are displayed in the same screen of the display means. Therefore, it is possible to display (preview display) an image after correction processing when geometric correction is performed on the captured image.

  Furthermore, in the image processing apparatus according to the present invention, the image pickup apparatus further includes a detection unit that detects an edge pixel group connected in a line segment in the captured image, and the display processing unit detects the edge pixel group detected by the detection unit. A rectangular outline indicating the outline of the object surrounded by a straight line corresponding to each of the four line segments indicated by the four edge pixel groups detected by the detection unit is superimposed on the captured image. It is preferable to display.

  According to the above configuration, an outline indicating the outline of the object is superimposed and displayed on the edge pixel group detected by the detection unit. Here, the edge pixel group detected by the detection unit is continuous in a line segment, and the probability of showing the boundary between the rectangular object and the background is very high. Therefore, the contour line indicating the contour of the object is displayed superimposed on the edge pixel group, so that the user can easily confirm the position of the object in the captured image. . As a result, the user can easily capture image data into the storage unit after easily confirming what position the object is in the imaging range.

  Furthermore, in the image processing apparatus according to the present invention, the display processing unit receives an instruction to edit the coordinates of the vertex of the contour, and the image of the area surrounded by the contour connects the coordinates of the edited vertex. Preferably, the image after the first correction processing and the image after the second correction processing are generated.

  According to the above configuration, the user can output the image after the first correction processing and the image after the second correction processing based on the area surrounded by the desired outline.

  Furthermore, it is preferable that the image processing apparatus of the present invention includes a switching unit that switches whether the display function of the outline processing by the display processing unit is enabled or disabled.

  According to the above configuration, the user effectively switches the display function of the outline when the object is a rectangle such as a document, and the display function of the outline when it is not necessary to display the outline. Can be switched off.

  Furthermore, in the image processing apparatus according to the present invention, when the switching unit effectively sets the display function of the outline, the display processing unit performs the geometrically corrected image after the first correction processing and the first correction processing. Preferably, the image after the 2 correction processing is displayed on the display means.

  According to the above configuration, by operating the switching unit, it is possible to display an image after correction processing when geometric correction is performed on a captured image.

  The image processing method according to the present invention further includes display means for displaying a captured image obtained by capturing an object, wherein the object is rectangular, and the captured image is displayed before the output object image is determined. An image processing method of an image processing apparatus including a first captured image displayed on a means and a second captured image determined as the output target image, wherein: (a) the above-mentioned output target image is determined After the first corrected image is displayed on the display means, in which the range of the object of the first captured image is geometrically corrected to a predetermined rectangular shape, and after the output target image is determined, the second captured image And displaying the second correction-processed image on which the range of the object is geometrically corrected to a predetermined rectangular shape on the display means.

  According to the above method, the first corrected image is displayed on the display unit before the output target image is determined, and the second corrected processing image is displayed on the display unit after the output target image is determined. Display. Therefore, it is possible to display (preview display) an image after correction processing when geometric correction is performed on the captured image before and after the output target image is determined.

  Further, in the image processing apparatus according to the present invention, the display processing unit displays the first captured image and the image after the first correction processing in the same screen, and the second captured image and the second correction processing It is preferable to display the image on the same screen.

  Further, in the image processing method of the present invention, in the step (a), the first captured image and the image after the first correction processing are displayed in the same screen, and the second captured image and the second correction processing Display the post-image on the same screen.

  According to the above method, the captured image and the image after correction processing are displayed on the same screen of the display means. Therefore, it is possible to display (preview display) an image after correction processing when geometric correction is performed on the captured image.

  Furthermore, the image processing method according to the present invention further includes the step (b) of detecting a group of edge pixels connected in a line segment in the captured image, and the step (a) detects the edge pixel group. On the edge pixel group, a rectangular outline indicating the outline of the object surrounded by a straight line corresponding to each of the four line segments indicated by the four edge pixel groups detected in the step (b) Is superimposed on the captured image and displayed.

  According to the above-described method, the user can easily find out what position the object is in the captured image by displaying an outline indicating the outline of the object overlapping on the edge pixel group. It can be confirmed. As a result, the user can easily capture image data into the storage unit after easily confirming what position the object is in the imaging range.

  Furthermore, in the image processing method of the present invention, in the step (a), an instruction to edit the coordinates of the vertex of the contour is received, and the image of the area surrounded by the contour connecting the coordinates of the edited vertex is The image after the first correction processing and the image after the second correction processing are generated.

  According to the above method, the user can output the image after correction processing based on the area surrounded by the desired outline.

  Furthermore, the image processing method of the present invention further includes a step (c) of switching whether the display function of the outline in the step (a) is enabled or disabled.

  According to the above method, the user effectively switches the display function of the outline when the object is a rectangle such as a document, and the display function of the outline when it is not necessary to display the outline. Can be switched off.

  Furthermore, in the image processing method of the present invention, when the display function of the outline is set to be effective in the step (c) in the step (a), the geometrically corrected image after the first correction processing And displaying the image after the second correction processing on the display means.

  According to the above method, by switching the display function of the outline, it is possible to display an image after correction processing when geometrical correction is performed on the captured image.

  The present invention has the effect of being able to realize an imaging device capable of capturing image data into the storage unit after the user easily confirms what position the object is in the imaging range. .

FIG. 1 is a diagram showing an entire configuration of a captured image processing system according to an embodiment of the present invention. It is a block diagram which shows the structure of the portable terminal device which concerns on one Embodiment of this invention. It is a block diagram which shows the structure of the imaging range determination part with which a portable terminal device is provided. It is a flowchart which shows an example of a process of an imaging range determination part. It is a figure which shows the example of a production | generation of an edge image. It is a figure which shows the method of labeling of a connection edge area | region. It is a flow chart which shows an example of extraction processing of a feature field which is a candidate of a connection edge field which serves as a boundary of an imaging subject and a background. It is a figure which shows a mode that a feature area | region is extracted from the edge image containing many connection edge area | regions. It is a flowchart which shows an example of a straight-line extraction process which specifies the approximate straight line of the edge pixel group used as the upper side of the boundary of a rectangular imaging target object and a background. It is a flow chart which shows an example of straight line extraction processing which specifies an approximate straight line of an edge pixel group which serves as a left side of a boundary between a rectangular imaging target and a background. It is a figure which shows the example which extracts the edge pixel group which continues in a line form which comprises the upper side of the square which becomes a boundary of a rectangular imaging target object and a background, a left side, a right side, and a lower side among feature areas. It is a figure which shows the example which calculates | requires the intersection of the approximate straight line of an edge pixel group. It is a flowchart which shows the process example of a display process part. It is a figure which shows the example of a screen where the information which shows that an imaging target object is in an imaging range was displayed. It is a figure showing an example of a screen where the 1st lack information was displayed. It is a figure showing an example of a screen where the 2nd lack information was displayed. It is a figure which shows the example of a detection of geometric distortion of the image by a geometric arrangement detection part. It is a figure which shows the example of an edge detection process of the imaging target object in an image. It is a figure which shows the example of edge detection of the raster direction of an image. FIG. 1 is a block diagram showing the configuration of an image forming apparatus according to an embodiment of the present invention. FIG. 2 is a block diagram showing a configuration of an image processing unit included in the image forming apparatus. FIG. 7 is a diagram showing an example of a look-up table created at the time of color balance adjustment of an image. It is a figure which shows the example of correction | amendment of the lens distortion of an image. It is a figure which shows the example of correction | amendment of geometric distortion and inclination of an image. It is a flow chart which shows an example of processing of high resolution amendment. The reference pixel and the interpolation pixel in high resolution image data are shown. It is a figure which shows an example of a first order differential filter. It is a figure which shows the determination method of the pixel value of an interpolation pixel. 5 is a flowchart showing the entire processing flow of the image forming apparatus. It is a figure which shows the example of a display of 1st missing information. It is a figure which shows the example of a display of 2nd missing information. It is a figure which shows the example of a screen in case the auto-shutter function is effective. It is a block diagram which shows the structure of the portable terminal device which concerns on a modification. It is a figure which shows the example of an inclination detection of an image. It is a figure which shows inclination-angle (theta) in the example of inclination detection of FIG. 34, and its tangent value. It is a figure which shows the example of determination of the reconstruction pixel value of an image. It is a flowchart which shows the modification of feature field extraction processing (S3). It is a figure which shows a mode that a feature area | region is extracted from a captured image by the feature area | region extraction process shown in FIG. It is a flowchart which shows the determination process of the detection method applied with respect to a captured image. It is a figure which shows the example which determines a detection method by the flowchart of FIG. It is a flowchart which shows the example of a process performed in S611 shown in FIG. FIG. 14 is a flowchart illustrating an example of processing performed in the case of No in S612, No in S614, or No in S615 in FIG. 13. It is a figure which shows the example of a screen of a display part when a display process part operate | moves according to the process of FIG. 41 and FIG. It is a block diagram which shows the modification of an imaging range judgment part. FIG. 8 is a diagram showing an example of a screen including a check box for switching document correction mode on / off. The example of a screen where the image after correction processing was displayed is shown.

  Hereinafter, embodiments of the present invention will be described in detail.

(1) Overall Configuration of Captured Image Processing System FIG. 1 is a view showing the overall configuration of a captured image processing system according to the present invention. As shown in FIG. 1, the captured image processing system is an image forming apparatus (image output such as a mobile terminal (imaging apparatus) 100 equipped with an imaging unit such as a camera-equipped mobile phone, a digital still camera, etc. Apparatus) 200, and an image display apparatus (image output apparatus) 300 such as an information display or an electronic blackboard.

  The portable terminal device 100 is carried by a user. The user can capture an object with the mobile terminal device 100 in various situations.

  In the present embodiment, the portable terminal device 100 captures a rectangular image such as a sheet on which a document image is printed, a poster, or a display screen on which the document image is displayed (for example, a display screen or a screen projected by a projector). It has a function of a document imaging mode for imaging an object and outputting an image obtained by the imaging from the image forming apparatus 200 or the image display apparatus 300. That is, the portable terminal device 100 obtains image data (hereinafter referred to as output target image data) which is obtained by imaging in the document imaging mode and is a target to be output by the image forming device 200 or the image display device 300. Or it transmits to the image display device 300.

  Then, the image forming apparatus 200 performs predetermined image processing on the received output target image data, and indicates the output target image data after image processing (hereinafter referred to as corrected image data) or corrected image data. Output an image. Further, the image display apparatus 300 performs display processing of output target image data.

  Here, the user may not always be able to capture an image from the front with respect to a rectangular imaging object, a sheet on which a document image is printed, a poster, a display screen on which the document image is displayed, or the like. That is, the user may pick up an image of the imaging object from an oblique direction in a state in which the normal direction of the plane on which the document image is formed in the imaging object does not match the imaging direction of the imaging means. In this case, an image may be captured in a state in which a part of the imaging target protrudes from the imaging range of the imaging unit, that is, in a state in which a part of the imaging target is missing. In such a case, imaging is performed in a state where part of the information desired by the user is missing. In the present embodiment, when the document imaging mode is selected, the mobile terminal device 100 has a function of preventing imaging in a state where a part of such an imaging target is out of the imaging range. .

  Note that, as output processing executed by the image forming apparatus 200, printing processing that prints and outputs an image indicated by the corrected image data, filing processing that stores output target image data to a storage device such as a server or a USB memory , Email attachment processing for attaching corrected image data to an email, and the like. The output process performed by the image display device 300 is a display process of output target image data.

  The portable terminal device 100 and the image forming apparatus 200 can communicate with each other, and the portable terminal device 100 transmits output target image data to the image forming apparatus 200 as described above. As a communication method between the portable terminal device 100 and the image forming apparatus 200, there is a method shown by a symbol A or B in FIG. The system indicated by the code A is a wireless communication system based on any of infrared communication standards such as IrSimple. In the method indicated by the code B, output target image data is temporarily sent from the mobile terminal device 100 to the image display device 300 by non-contact wireless communication such as Felica (registered trademark), and then, for example, Bluetooth (registered trademark) The data is transferred from the image display apparatus 300 to the image forming apparatus 200 using wireless communication. In the present embodiment, the user operates the mobile terminal 100 after coming to the front of the image forming apparatus 200, and uses the short distance wireless communication method such as infrared communication to transfer the image from the mobile terminal 100 to the image forming apparatus 200. It shall transmit data.

  Here, the image display device 300 is, for example, an information display, an electronic blackboard, or the like including a liquid crystal display, a plasma display, an organic electroluminescence display, and the like. Such an image display device 300 performs display processing for displaying an image indicated by output target image data. This display process is one of the output process of the output target image data. That is, it can be said that the image display device 300 is an image output device that performs output processing of output target image data. After displaying on the image display apparatus 300, printing is performed on the image forming apparatus 200, or e-mail transmission is performed to another address, or stored in a computer or a server connected via a network. You can also

  The communication method between the portable terminal device 100 and the image forming device 200 and the image display device 300 is not limited to these, and any known communication method can be applied. For example, output target image data may be attached to an e-mail and transmitted to the image forming apparatus 200 or the image display apparatus 300.

(2) Configuration of Mobile Terminal Device First, the mobile terminal device 100 according to the present embodiment will be described based on FIG. FIG. 2 is a block diagram showing the configuration of the mobile terminal device 100. As shown in FIG. As illustrated in FIG. 2, the mobile terminal device 100 includes an imaging unit 101, an imaging range determination unit 110, an image processing unit 103, a communication unit (transmission unit) 104, a display unit 105, an input unit 106, and a recording medium access unit 107. , Storage unit 108, and control unit 109.

  The imaging unit 101 performs imaging of an imaging target object using a CCD sensor or a CMOS sensor, and causes the display unit 105 to display a captured image obtained by imaging. Note that the imaging unit 101 captures an image of an imaging target at a preset resolution. A range to be imaged by the imaging unit 101 (hereinafter referred to as an imaging range) is determined in accordance with a preset enlargement / reduction rate. Here, it is assumed that the imaging range is a rectangle having a width Xmax and a height Ymax.

  When the document imaging mode is selected by the user, the imaging range determination unit 110 sets the rectangular imaging target within the imaging range based on the captured image captured by the imaging unit 101 and displayed on the display unit 105. It is determined whether or not it is contained. Further, the imaging range determination unit 110 stores image data indicating a captured image displayed on the display unit 105 at a designated timing in the storage unit 108 as output target image data. Details of the imaging range determination unit 110 will be described later.

  The image processing unit 103 performs at least A / D conversion processing on output target image data stored in the storage unit 108 by the imaging unit 101.

  The communication unit 104 has functions of serial transfer / parallel transfer and wireless data communication based on the USB (Universal Serial Bus) 1.1 or USB 2.0 standard. The communication unit 104 transmits output target image data to the image forming apparatus 200 or the image display apparatus 300 in accordance with a transmission instruction input by the user.

  The display unit 105 is configured of, for example, a liquid crystal display. The input unit 106 also has a plurality of buttons, and is used by the user to input data.

  The recording medium access unit 107 reads a program from a recording medium in which a program for performing each process of the mobile terminal device 100 is recorded.

  Further, the storage unit 108 stores a program for performing each process of the mobile terminal device 100, model information of the mobile terminal device 100, user information, and data necessary for performing the process. The user information is information for identifying the user of the mobile terminal device 100, and includes, for example, a user ID and a password. In addition, the storage unit 108 stores output target image data obtained by imaging in the document imaging mode and its attached information (output processing information to be described later, a file name, and the like).

  The control unit 109 controls each unit of the mobile terminal device 100. When an instruction to select a document imaging mode is input to the input unit 106, the control unit 109 instructs selection of the type of output processing (print processing, filing processing, mail transmission processing, etc.) in the image forming apparatus 200, In addition, the display unit 105 displays a screen prompting input of setting conditions (a printing condition such as the number of printed sheets, an address of a filing destination server, an e-mail transmission destination address, etc.) for executing the selected output process. Then, the control unit 109 acquires, from the input unit 106, output processing information indicating the type of output processing and the setting conditions of the output processing.

  The control unit 109 adds a file name and output processing information to the output target image data stored in the storage unit 108.

  In addition, when a transmission instruction is input to input unit 106, control unit 109 causes communication unit 104 to transmit the output target image data stored in storage unit 108 to image forming apparatus 200 or image display apparatus 300. Make it run. At this time, the communication unit 104 combines the output target image data, the file name and output processing information associated with the output target image data, and the model information and user information stored in the storage unit 108. It is transmitted to the image forming apparatus 200 or the image display apparatus 300.

(3) Regarding Imaging Range Determination Unit (3-1) Configuration of Imaging Range Determination Unit Next, a detailed configuration of the imaging range determination unit 110 of the mobile terminal device 100 will be described. FIG. 3 is a block diagram showing an internal configuration of the imaging range determination unit 110. As shown in FIG. As shown in FIG. 3, the imaging range determination unit 110 includes a geometric arrangement detection unit 111, a display processing unit 112, and an output target image determination unit 113.

  The geometric arrangement detection unit 111 assumes that the imaging object is rectangular, and extracts the edge pixel group that is the boundary between the imaging object and the background, thereby the geometrical arrangement of the imaging object. (Geometric distortion) is detected.

  The display processing unit 112 superimposes and displays an outline indicating an outline of the object to be imaged on the edge pixel group detected by the geometric arrangement detection unit 111 in the captured image displayed on the display unit 105. It is. Thus, the user can easily confirm whether the imaging target is within the imaging range by confirming the outline.

  In addition, the display processing unit 112 determines whether or not the imaging target object is included in the imaging range based on the geometrical arrangement of the imaging target object, and displays the determination result on the display unit 105. As a result, the user can more easily confirm whether the imaging target is within the imaging range by confirming the determination result. Then, when the position does not fit, the user can fit the imaging target object within the imaging range by changing the orientation and the position of the mobile terminal device 100.

  The output target image determination unit 113 determines, as output target image data, image data indicating a captured image displayed on the display unit 105 at a designated timing, and stores the output target image data in the storage unit 108. is there.

(3-2) Processing of Imaging Range Determination Unit Next, an example of specific processing of the imaging range determination unit 110 will be described. FIG. 4 is a flowchart illustrating an example of processing of the imaging range determination unit 110.

(Step 1 (S1))
First, the geometric arrangement detection unit 111 extracts edge pixels from the captured image captured by the imaging unit 101 and displayed on the display unit 105. Then, the geometric arrangement detection unit 111 generates an edge image in which the edge pixel is “1” and the non-edge pixel is “0”.

  For extraction of edge pixels, for example, a Canny filter is applied to a luminance image captured by the imaging unit 101 and extracted. The Canny filter is a filter that detects a thinned edge using a Gaussian filter and a Sobel filter. At this time, the image size may be reduced to speed up the processing. Further, in order to enhance the detection accuracy of the edge pixel, for example, morphological conversion such as expansion and contraction may be performed after the smoothing or the filter processing before the filter processing.

  FIG. 5 is a view showing an edge image extracted from a luminance image. In FIG. 5, the upper part shows a luminance image, and the lower part shows an edge image extracted.

(Step 2 (S2))
Next, the geometric arrangement detection unit 111 performs a labeling process in which different labels are attached to each of the connected edge pixel areas (connected edge areas).

The geometrical arrangement detection unit 111 performs labeling, for example, using the following method as shown in FIG.
(I) In the case where the pixel of interest is an edge pixel, if the pixel above and adjacent to the pixel of interest is an edge pixel and is already labeled, the pixel of interest is also labeled the same (FIG. 6A).
(II) If the pixel adjacent to the left is also an edge pixel and is labeled differently from the pixel adjacent to the upper pixel, the same label as that attached to the upper pixel is attached (FIG. 6 (b)).
(III) When the pixel on the upper side is a non-edge pixel and the pixel on the left is an edge pixel, the target pixel is labeled the same as on the left (Fig. 6 (c)).
(IV) In the case where neither the upper adjacent nor the left adjacent is a non-edge pixel, a new label is attached to the pixel of interest (FIG. 6 (d)).
(V) Label all edge pixels.
(VI) If multiple labels are attached, unify the labels based on the above rules.

(Step 3 (S3))
Next, the geometric arrangement detection unit 111 extracts a candidate of a region including the boundary between the object to be imaged and the background (hereinafter referred to as a feature region) from among the connected edge regions labeled (S3). .

  It is usual that the object to be imaged is imaged with its center near the center of the imaging range and occupying most of the imaging range. Therefore, the boundary between the object to be imaged and the background has its center located near the center of the imaging range, and the length in the horizontal direction (width direction) and the length in the vertical direction (height direction) of the imaging range become longer. Therefore, the geometric arrangement detection unit 111 extracts a connected edge region that satisfies the following condition A as a feature candidate.

  Condition A: In the imaging range, the upper left corner is the origin, the right direction (width direction) is the x axis, the downward direction (height direction) is the y axis, the x coordinate of the right end of the imaging range is Xmax, y at the lower end of the imaging range Coordinates are assumed to be Ymax. At this time, the length in the width direction of the connection edge area is 1/4 or more of the width of the imaging range (that is, Xmax), and the length in the height direction is 1/4 or more of the height of the imaging range (that is, Ymax) And the center x coordinate of the connected edge area is not less than Xmax / 4 and not more than 3 × Xmax / 4, and the center y coordinate of the area is not less than Ymax / 4 and not more than 3 × Ymax / 4.

  FIG. 7 is a flowchart showing a detailed process example of the feature area extraction process (S3). As shown in FIG. 7, first, the geometric arrangement detection unit 111 selects one of the connected edge areas labeled S2 (S301). Then, the geometric arrangement detection unit 111 specifies the length in the width direction, the length in the height direction, the center x coordinate, and the center y coordinate for the selected connected edge area. Specifically, the length in the width direction is obtained by subtracting the smallest x coordinate from the largest x coordinate among edge pixels constituting the connected edge area, and the smallest y coordinate is subtracted from the largest y coordinate. The length in the height direction is determined, the average value of the maximum x coordinate and the minimum x coordinate is determined as a center x coordinate, and the average value of the maximum y coordinate and the minimum y coordinate is determined as a center y coordinate.

  Next, the geometric arrangement detection unit 111 determines whether the center x coordinate is not less than Xmax / 4 and not more than 3 × Xmax / 4 (S302).

  In the case of Yes in S302, the geometric arrangement detection unit 111 determines whether the center y coordinate is Ymax / 4 or more and 3 × Ymax / 4 or less (S303).

  In the case of Yes in S303, the geometric arrangement detection unit 111 determines whether the length in the width direction of the connection edge area is Xmax / 4 or more (S304).

  In the case of Yes in S304, the geometric arrangement detection unit 111 determines whether the length in the height direction of the connection edge area is Ymax / 4 or more (S305).

  In the case of Yes in S305, the selected connected edge area is extracted as a feature area (S306). Thereafter, the geometric arrangement detection unit 111 confirms whether there is an unselected connected edge region, and in some cases, selects an unselected connected edge region (S308).

  On the other hand, if the result is No in any of S302 to S305, the process proceeds to S308 without extracting the selected connected edge area as a feature area.

  Thereafter, the processing of step S302 and subsequent steps is executed for the connected edge area selected in step S308. Thus, it is possible to extract a connected edge region satisfying the above condition A as a feature region. Then, the geometric arrangement detection unit 111 generates feature area image data in which only pixels belonging to the extracted feature area are edge pixels and the remaining pixels are non-edge pixels.

  FIG. 8 is a diagram showing how a feature area is extracted from an edge image including a large number of connected edge areas. In FIG. 8, the left column is an edge image. The middle row is an image showing a state in which the connected edge area whose center coordinate is located on the end side of the imaging range is removed by the processing of S302 and S303. The right row is an image showing only the extracted feature regions, from which the connected edge regions having a short length in the width direction or the length in the height direction are further removed from the middle row by the processing of S304 and S305. That is, the right column is an image represented by feature area image data.

  As shown in FIG. 8, in the process of S3, a feature area which is a candidate for an edge pixel that is a boundary between a rectangular imaging target and a background is extracted. However, as shown in the lower part of FIG. 8, depending on the image, edge pixels other than the boundary between the object to be imaged and the background may be extracted as a feature region.

(Step 4 (S4))
Therefore, the geometric arrangement detection unit 111 is a group of edge pixel groups connected in a line shape that forms the upper side, the left side, the right side, and the lower side of the quadrangle serving as the boundary between the rectangular imaging target and the background. Are extracted, and processing (straight line extraction processing) for specifying an approximate straight line of the edge pixel group is performed (S4).

  Here, the upper side is located in the upper half of the captured image (that is, the y coordinate is in the range of 0 to Ymax / 2), and the probability of being parallel to the width direction of the imaging range is high. Also, the left side is located in the left half of the captured image (that is, the x coordinate is in the range of 0 to Xmax / 2), and the probability of being parallel to the height direction of the imaging range is high. The right side is located in the right half of the captured image (that is, the x coordinate is in the range of Xmax / 2 to Xmax) and has a high probability of being parallel to the height direction of the imaging range. The lower side is located in the lower half of the captured image (that is, the y coordinate is in the range of Ymax / 2 to Ymax) and has a high probability of being parallel to the width direction of the imaging range.

  Therefore, in the range where the existing probability is high, the number of edge pixels connected in a specific direction in the feature area image data is the largest, and a line-shaped edge pixel group having a predetermined length or more It extracts as an edge pixel group which continues in a line segment shape which becomes a boundary of and the background. Then, the approximate straight line of the extracted edge pixel group is specified.

  FIG. 9 is a flowchart showing the flow of the process of identifying the approximate straight line of the edge pixel group constituting the upper side in the process of S4 shown in FIG.

  First, the geometric arrangement detection unit 111 sets a variable x = 0, a variable count = 0, and a variable pre_y = -1 (S401). Next, the geometric arrangement detection unit 111 sets a variable y = 0 (S402).

  Then, the geometric arrangement detection unit 111 confirms whether the pixel indicated by the coordinates (x, y) in the feature area image data is an edge pixel (S403). If No in S403, the variable y is changed to y + 1 (S404). Subsequently, it is checked whether the changed variable y is larger than half of the height of the imaging range (that is, Ymax) (S405). If No in S405, the process returns to S403.

By repeating the processing of S403 to S405, in the feature area image data, searching from the (0, 0) coordinate to the coordinate of 1/2 the height of the imaging range in the lower direction of the y axis is performed. When is found, it becomes Yes in S403. If Yes in S403, the geometric arrangement detection unit 111
pre_y-1 ≦ y ≦ pre_y + 1
It is confirmed whether the condition is satisfied (S407).

  If S407 is not satisfied, the geometric arrangement detection unit 111 stores the value of the variable count in the coordinate group previously set (S408). Thereafter, the geometrical arrangement detection unit 111 sets a new coordinate group [x] and sets the variable count to 0 (S409). On the other hand, if S407 is satisfied, the geometric arrangement detection unit 111 adds 1 to the variable count (S410), and stores the coordinates (x, y) in the most recently set coordinate group (S411). Then, after S409 or S411, the geometric arrangement detection unit 111 sets the variable pre_y to y (S412).

  On the other hand, when searching in the lower y-axis direction and no edge pixel is found (S405: No), the geometric arrangement detection unit 111 sets the variable count = 0 and the variable pre_y = -1 (S406). .

  After S406 or S412, the geometric arrangement detection unit 111 adds 1 to the variable x (S413). Subsequently, it is checked whether the changed variable x is larger than the width of the imaging range (that is, Xmax) (S414). If No in S414, the process returns to S402.

  For example, at x = 0, a search is made downward in the y axis, and the coordinates (0, y0) of the edge pixel found first are recorded. Next, the y axis is searched downward from the (1, 0) coordinates, and the coordinates (1, y 1) at the time when the edge pixel is first found are recorded. At this time, if y0-1 ≦ y1 ≦ y0 + 1, it is determined that these two points are continuous along the specific direction, and the variable count is increased by 1 in S409 and the coordinate (1, y1) is set to the coordinate group [0] Store in If y0-1 ≦ y1 ≦ y0 + 1 is satisfied, it means that two points are continuous in a specific direction within a predetermined angle range from the width direction of the imaging range.

  In this way, after storing coordinates (k, yk) in coordinate group [0], the coordinates (k + 1, y (y + 1) of the edge pixel first found searching from the (k + 1, 0) coordinates downward along the y axis It is assumed that k + 1) does not satisfy yk-1 ≦ y (k + 1) ≦ yk + 1. In this case, the value k of the variable count is stored in the coordinate group [0] in S408. In S409, a new coordinate group [k + 1] is set, and the variable count is reset to 0.

  The above process is repeated until x becomes the width of the imaging range (that is, Xmax). Thereafter, the geometric arrangement detection unit 111 selects a coordinate group having the largest number of coordinates, that is, a coordinate group having the largest value of count (S415). Then, the geometric arrangement detection unit 111 checks whether the number of coordinates (that is, the count value) included in the selected coordinate group is Xmax / 4 or more (S416). In the case of No in S416, since the upper side which is the boundary between the object to be imaged and the background is short, the geometric arrangement detection unit 111 can not extract the straight line constituting the upper side (extraction impossible information) Are generated (S417).

  On the other hand, in the case of Yes in S416, the geometric arrangement detection unit 111 obtains an equation of an approximate straight line using the least squares method based on a plurality of coordinates included in the selected coordinate group (S418). In the case of Yes in S416, the coordinate group with the largest number of coordinates is an edge pixel group connected in a line segment of Xmax / 4 or more in a direction within a predetermined angle range from the width direction of the imaging range . Therefore, the edge pixel group has a very high probability of indicating the upper side of the boundary between the object to be imaged and the background.

  For example, if (6, 120), (7, 120), (8, 121), (9, 122), (10, 121) are stored in the coordinate group with the largest number of coordinates, geometrical arrangement The detection unit 111 can obtain the equation y = 0.4x + 117.6 by the least squares method.

The same process is performed on the lower side, the right side, and the left side to extract straight lines of four sides.
In the case of the lower side, the variable y = Ymax is set in S402 of FIG. In S404, 1 is subtracted from the variable y, and in S405, it is checked whether the changed variable y is smaller than half the height of the imaging range (that is, Ymax).

  Further, in the case of the left side, the process of FIG. 10 may be performed. FIG. 10 shows a process in which x and y are interchanged in FIG. This is because the left side is located in the left half of the imaging range and has a high probability of being parallel to the height direction of the imaging range. According to FIG. 10, in S418, a straight line expression is generated based on a group of edge pixels connected in a line segment having a length of Ymax / 4 or more in a direction within a predetermined angle range from the height direction of the imaging range. Ask for The edge pixel group has a very high probability of indicating the left side of the boundary between the object to be imaged and the background.

  In the case of the right side, the variable x = Xmax is set in S402 'of FIG. In addition, 1 is subtracted from the variable x in S404 ', and it is checked in S405' whether the changed variable x is smaller than 1/2 of the width of the imaging range (that is, Xmax).

  FIG. 11 illustrates an example of extracting (detecting) a group of edge pixel groups connected in a line shape, which constitute the upper side, the left side, the right side, and the lower side of a quadrangle serving as the border between the rectangular imaging target and the background. FIG. (A) of FIG. 11 is an image showing all the feature regions. (B) shows a group of edge pixels extracted as the upper side of a quadrangle that is the boundary between the rectangular imaging target and the background, as a solid line. (C) shows a group of edge pixels extracted as the left side of a quadrangle serving as the border between the rectangular imaging target and the background as a solid line. In (d), an edge pixel group extracted as a lower side of a quadrangle that is a boundary between a rectangular imaging target and a background is indicated by a solid line. (E) shows a group of edge pixels extracted as the right side of a quadrangle, which is a boundary between the rectangular imaging target and the background, by a solid line.

  Thus, the geometric arrangement detection unit 111 extracts the approximate straight line of the edge pixel group extracted as the upper side as the upper side straight line, and the approximate straight line of the edge pixel group extracted as the left side as the left side straight line, the edge extracted as the right side The approximate straight line of the pixel group is taken as the right side straight line, and the approximate straight line of the edge pixel group extracted as the lower side is taken as the lower side straight line, and the equation of each straight line is generated.

(Step 5 (S5))
When the straight line extraction processing of the four sides in S4 is completed, the geometrical arrangement detection unit 111 obtains intersection coordinates based on the equation of the straight line obtained in S4 (S5).

  When the straight line equation corresponding to the four sides is obtained in S4, the geometric arrangement detection unit 111 can easily obtain the intersection coordinates of the two straight lines. That is, the geometrical arrangement detection unit 111 determines the intersection coordinates of the left side straight line and the upper side straight line as the upper left apex coordinate, the intersection coordinates of the upper side straight line and the right side straight line as the upper right apex coordinate, and the intersection coordinates of the right side straight line and the lower side straight line as the lower right apex coordinate , The intersection coordinates of the lower side straight line and the left side straight line are determined as the lower left vertex coordinates. Then, the geometric arrangement detection unit 111 outputs extraction result information including the four vertex coordinates to the display processing unit 112.

  FIG. 12 is a diagram showing an example of obtaining four vertex coordinates. In FIG. 12, upper left vertex coordinates (X1, Y1), upper right vertex coordinates (X2, Y2), lower right vertex coordinates (X3, Y3), and lower left vertex coordinates (X4, Y4) are obtained.

  When a straight line equation corresponding to only three sides is determined in S4, the geometric arrangement detection unit 111 determines the straight line equation with the straight line corresponding to the other side as the straight line at the end of the imaging range. That is, x = 0 if the left side can not be extracted, x = Xmax if the right side can not be extracted, y = 0 if the upper side can not be extracted, y = Ymax if the lower side can not be extracted Let it be the corresponding straight line equation. Then, the geometric arrangement detection unit 111 obtains four vertex coordinates using the linear expression.

  However, the point of intersection with the straight line at the end of the imaging range is determined as temporary vertex coordinates. For example, when the right side can not be extracted, the upper right vertex coordinates and the lower right vertex coordinates are obtained as temporary vertex coordinates.

  Then, the geometric arrangement detection unit 111 generates extraction result information including four vertex coordinates, information indicating that only three sides could be extracted, and extraction impossible information indicating sides that could not be extracted, It is output to the display processing unit 112. In addition, about temporary vertex coordinate, the information which shows that it is temporary vertex coordinate is attached.

  In addition, in S4, when the equation of the straight line corresponding to the three sides or the four sides is not obtained, the geometric arrangement detection unit 111 extracts that indicates that the boundary between the imaging object and the background could not be properly extracted. Result information is generated and output to the display processing unit 112.

(Step 6 (S6))
Subsequently, the display processing unit 112 performs display processing based on the extraction result information. Specifically, it is as follows.

  The display processing unit 112 superimposes and displays a quadrilateral line connecting four vertex coordinates indicated by the extraction result information as an outline of the imaging target on the captured image.

  In addition, when the extraction result information includes four vertex coordinates and does not include the extraction disabling information, the display processing unit 112 determines whether the four vertex coordinates are within the imaging range. If all the vertex coordinates are within the imaging range, the display processing unit 112 causes the display unit 105 to display information (for example, “OK”) indicating that the imaging target object is within the imaging range.

  In addition, when the three vertex coordinates are within the imaging range and one vertex coordinate is outside the imaging range, the display processing unit 112 is the first missing information indicating that a part (one corner) of the imaging object can not be imaged. Is displayed on the display unit 105.

  Furthermore, when the extraction result information includes four vertex coordinates and includes extraction impossible information, the display processing unit 112 determines whether the four vertex coordinates are within the imaging range. If all the vertex coordinates are within the imaging range, the display processing unit 112 causes the display unit 105 to display second dropout information indicating that a part (one side) of the imaging target object can not be imaged.

  In the case other than the above, the display processing unit 112 may leave the screen of the display unit 105 as it is, or information prompting the user to change the orientation of the imaging unit 101 (for example, Adjust the orientation of the camera so as to fit, etc.) may be displayed on the display unit 105.

FIG. 13 is a flowchart showing the details of the process of S6.
First, the display processing unit 112 determines whether all four sides serving as the boundary between the rectangular imaging target and the background have been extracted (S601). Here, the display processing unit 112 determines that all the four sides have been extracted when the extraction result information includes four vertex coordinates and does not include the extraction disabling information.

  If all four sides are extracted (Yes in S601), the display processing unit 112 determines whether the upper left vertex coordinates (X1, Y1) indicated by the extraction result information are within the imaging range (S602). Specifically, the display processing unit 112 determines whether both 0 ≦ X1 ≦ Xmax and 0 ≦ Y1 ≦ Ymax are satisfied, and when both are satisfied, the upper left vertex coordinate is within the imaging range I will judge.

  If the upper left vertex coordinate is within the imaging range, the display processing unit 112 generates a flag indicating that the upper left vertex coordinate (X1, Y1) is within the imaging range and stores the flag (S603). On the other hand, if the upper left vertex coordinate is out of the imaging range, the process proceeds to the next process.

  The display processing unit 112 performs the same processing as in S602 and S603 for upper right vertex coordinates (X2, Y2), lower right vertex coordinates (X3, Y3), and lower left vertex coordinates (X4, Y4) (S604 to S609) .

  Then, the display processing unit 112 confirms whether a flag is stored for all vertex coordinates (S610). When the flag is stored for all the vertex coordinates (Yes in S610), the display processing unit 112 determines that the imaging target object falls within the imaging range without missing. Then, the display processing unit 112 superimposes and displays a quadrilateral line connecting four vertex coordinates indicated by the extraction result information as an outline of the imaging target on the captured image. Here, the vertex coordinates are the intersections of the approximate straight lines with the line-shaped edge pixel group detected by the geometric arrangement detection unit 111. Therefore, the outline is displayed superimposed on the edge pixel group. In addition, the display processing unit 112 causes the display unit 105 to display information (for example, “OK”) indicating that the imaging target object is within the imaging range (S611). At this time, the display processing unit 112 also causes the display unit 105 to display a shutter button for determining an output target image.

  FIG. 14 is a view showing an example of a screen displayed on the display unit 105 in S611. As shown in FIG. 14, an outline L which is a square line connecting vertex coordinates is displayed on the boundary between the rectangular imaging target and the background. Therefore, the user can easily confirm that the imaging target is within the imaging range.

  In FIG. 14, reference numeral 10 is a shutter button, reference numeral 20 is an auto focus setting button, reference numeral 30 is an exposure adjustment bar, reference numeral 40 is an image reading button, and reference numeral 50 is It is a condition setting button. The shutter button 10 is a button for determining a captured image displayed on the display unit 105 as an output target image. The autofocus button 20 is a button for automatically focusing. The exposure adjustment bar 40 is a bar for adjusting the light exposure. The image reading button 40 is for processing an image saved in advance as a captured image. When the image reading button 40 is pressed, the stored image list screen is displayed, and when one of the images is selected by the user, the next processing (shutter is performed as an image captured by the imaging unit 101). Processing) is performed. The condition setting button 50 is a button for setting various imaging conditions.

  On the other hand, when the flag is not stored for all the vertex coordinates (No in S610), the display processing unit 112 confirms whether the flag is stored for the three vertex coordinates (S612).

  In the case of No in S612, that is, when only vertex coordinates of 2 or less are within the imaging range, the display processing unit 112 displays information indicating that the imaging object is within the imaging range (for example, "OK") Is not displayed on the display unit 105. Therefore, the user recognizes that the imaging target object is not within the imaging range, and changes the orientation of the imaging unit 101. After the direction of the imaging unit 101 is changed (S617), the process returns to S1 again. That is, based on the image data after the direction of the imaging unit 101 has been changed, the processes after S1 are performed again.

  In the case of Yes in S612, that is, when the three vertex coordinates are within the imaging range, the display processing unit 112 superimposes a quadrilateral line connecting the four vertex coordinates on the captured image as the outline of the imaging object. indicate. Furthermore, the display processing unit 112 causes the display unit 105 to display first loss information indicating that a part (one corner) of the imaging target can not be imaged (S613).

  (A) and (b) of FIG. 15 are diagrams showing an example of a screen displayed on the display unit 105 in S613. As shown in FIG. 15, an outline L which is a rectangular line connecting vertex coordinates is displayed on the boundary between the rectangular imaging target and the background. Therefore, the user can easily confirm that the imaging target is within the imaging range.

  Further, as the first missing information, a first icon B indicating that a corner is missing is displayed in the vicinity of vertex coordinates located outside the imaging range. Here, the display processing unit 112 identifies a point closest to the vertex coordinate from the imaging range based on the vertex coordinate outside the imaging range, and sets the first point within the range of a predetermined distance from the identified point. Display icon B. At this time, the display processing unit 112 displays so that the first icon B and the outline L do not overlap. Thus, the user who has confirmed the first icon B can easily recognize that the corner near the first icon B is out of the imaging range. Then, the user can easily change the orientation of the imaging unit 101 so that the imaging target object falls within the imaging range. After the direction of the imaging unit 101 is changed (S617), the process returns to S1 again. That is, based on the image data after the direction of the imaging unit 101 has been changed, the processes after S1 are performed again.

  In addition, in the case of No in S601, the display processing unit 112 determines whether only three sides out of the four sides serving as the boundary between the rectangular imaging target and the background are extracted (S614). Here, the display processing unit 112 determines that only three sides are extracted when the extraction result information includes four vertex coordinates and includes the extraction impossible information.

  In the case of Yes in S614, the display processing unit 112 determines whether two vertex coordinates excluding temporary vertex coordinates indicated by the extraction result information are within the imaging range (S615). Here, the two vertex coordinates excluding temporary vertex coordinates are the upper left vertex coordinates which are the intersections of the upper side and the left side, and the intersections of the upper side and the right side when the extracted three sides are the upper side, the left side and the right side. It is upper right vertex coordinates. When the extracted three sides are the upper side, the left side, and the lower side, the upper left vertex coordinate which is the intersection of the upper side and the left side and the lower left vertex coordinate which is the intersection of the left side and the lower side. When the extracted three sides are the left side, the lower side, and the right side, they are the lower left vertex coordinate which is the intersection of the lower side and the left side, and the lower right vertex coordinate which is the intersection of the lower side and the right side. Further, in the case where the extracted three sides are the upper side, the right side, and the lower side, they are the upper right vertex coordinate which is the intersection of the upper side and the right side and the lower right vertex coordinate which is the intersection of the lower side and the right side. The process of S615 is the same as the process of S602 to S609.

  In the case of No in S614 or S615, the display processing unit 112 does not display information (for example, “OK”) indicating that the imaging target object is within the imaging range on the display unit 105. Therefore, the user recognizes that the imaging target object is not within the imaging range, and changes the orientation of the imaging unit 101. After the direction of the imaging unit 101 is changed (S617), the process returns to S1 again.

  On the other hand, in the case of Yes in S615, the display processing unit 112 displays the line excluding the side between two temporary vertex coordinates among the quadrilateral lines connecting the four vertex coordinates as an outline of the object to be imaged superimposed on the captured image Do. Then, the display processing unit 112 causes the display unit 105 to display second loss information indicating that a part (one side) of the imaging target can not be imaged (S616).

  FIG. 16 is a view showing an example of a screen displayed on the display unit 105 in S616. As shown in FIG. 16, the outline L is displayed on the boundary between the rectangular imaging object and the background. Therefore, the user can easily confirm that the imaging target is within the imaging range.

  Further, as the second missing information, a second icon C indicating that one side of the outer frame of the imaging target is missing is displayed near the side located outside the imaging range. The display processing unit 112 displays the second icon C within a predetermined range from the side of the frame of the imaging range corresponding to the side indicated by the extraction impossible information. For example, when the extraction impossible information is the right side, the second icon C is displayed in the vicinity of the right side of the outer frame of the imaging range (that is, a line indicated by x = Xmax). Thus, the user who has confirmed the second icon C can easily recognize that the side near the second icon C is out of the imaging range. Thereby, the user can easily change the orientation of the imaging unit 101 so that the imaging target object falls within the imaging range. Then, after the orientation of the imaging unit 101 is changed (S617), the process returns to S1 again. That is, based on the image data after the direction of the imaging unit 101 has been changed, the processes after S1 are performed again.

  Preferably, the display processing unit 112 displays the first dropout information and the second dropout information of a color different from that of the outline L. Thereby, the user can easily confirm the first missing information and the second missing information.

(Step 7 (S7))
Finally, the output target image determination unit 113 determines output target image data which is image data to be output processing target. Specifically, as shown in FIG. 14, the output target image determination unit 113 displays information (for example, “OK”) indicating that the imaging target object is within the imaging range, and the shutter button 10 as a display unit. In the state displayed on the screen 105, the timing at which the shutter button 10 is operated is detected. Then, the output target image determination unit 113 determines, as the output target image data, the image data indicating the captured image displayed on the display unit 105 at the specified timing, with the timing when the shutter button 10 is operated as the specified timing. .

  The output target image determination unit 113 can accept the operation of the shutter button 10 only when the information indicating that the imaging target object is within the imaging range is displayed.

(3-3) Another Processing Example of Geometrical Arrangement Detection Unit (Part 1)
The geometric arrangement detection unit 111 omits the process of S2 described above, and performs the process shown in FIG. 37 in S3 described above, so that the candidate of the area including the boundary between the imaging object and the background (feature area ) May be extracted. FIG. 37 is a flowchart showing the feature area extraction process (S3) in the present modification. FIG. 38 is a diagram showing how a feature area is extracted by the feature area extraction process according to this modification.

  First, as shown in FIG. 38, the geometric arrangement detection unit 111 acquires the edge image of (2) extracted in S1 from the captured image of (1). Then, the geometric arrangement detection unit 111 performs a one-step expansion process on the extracted edge image (S311). The expansion process in one step is a process in which, when the pixel of interest is an edge, pixels in the vicinity 4 (pixels located above, to the left, below, and to the right of the pixel of interest) of the pixel of interest are edges. (3) of FIG. 38 shows the edge image after the expansion processing, and it can be seen that the edge portion is expanded.

  Next, the geometric arrangement detection unit 111 inverts the edge image (S312). That is, in the edge image, edge pixels are replaced with non-edge pixels, and non-edge pixels are replaced with edge pixels. (4) of FIG. 38 shows the edge image after the reverse process. As shown in (3) and (4) of FIG. 38, in the image that is expanded to increase the number of connections between the edge areas and then subjected to the inversion process, the area in contact with the image end (the imaging range end) is the background It can be considered that there is.

  Subsequently, the geometric arrangement detection unit 111 performs a labeling process for adding a different label to each area (connected area) of the connected edge pixels (S313). The said labeling process should just perform the process similar to said S2.

  The geometrical arrangement detection unit 111 selects one of the connected regions labeled in S313, and specifies a center x coordinate and a center y coordinate (S314). Specifically, the average value of the maximum x coordinate and the minimum x coordinate is determined as a center x coordinate, and the average value of the maximum y coordinate and the minimum y coordinate is determined as a center y coordinate.

  Next, the geometric arrangement detection unit 111 determines whether the center x coordinate is not less than Xmax / 4 and not more than 3 × Xmax / 4 (S315).

  In the case of Yes in S315, the geometric arrangement detection unit 111 determines whether the central y coordinate is greater than or equal to Ymax / 4 and less than or equal to 3 × Ymax / 4 (S316).

  In the case of Yes in S316, the geometric arrangement detection unit 111 determines whether the connection area is far from the image edge (S317). Specifically, whether the largest x coordinate of the selected connected area matches the x coordinate of the right end of the image (image width-1), or the smallest x coordinate matches the x coordinate of the left end of the image (0), or the largest It is determined whether the y coordinate of y coincides with the y coordinate (image height-1) at the lower end of the image or the minimum y coordinate coincides with the y coordinate (0) at the upper end of the image. Then, if even one is Yes, it is determined that the connected area is in contact with the image end (not apart from the image end), and if all are No, it is determined that the connection area is separated from the image end.

  In the case of Yes in S317, the geometric arrangement detection unit 111 extracts the selected connected area as a feature area (S318). After that, it is checked whether there is an unselected connection area (S319). On the other hand, if any one of S315, S316, and S317 is No, it is confirmed whether there is an unselected connected area without extracting the selected connected area as a feature area (S319).

  After that, when there is an unselected connected area, the geometric arrangement detection unit 111 selects one unselected connected area (S320). On the other hand, when there is no unselected connected area, the geometric arrangement detection unit 111 terminates the feature area extraction process and performs the process of S4 in FIG. 4 to indicate the boundary between the object to be imaged and the background. A line segment-shaped edge pixel group is detected.

  (5) in FIG. 38 illustrates the connection area determined as No in S315 or S316. The connection area shown in (5) of FIG. 38 is an area where the center coordinates are close to the image edge, and it is a small area drawn on the edge of the imaging object or a background portion other than the rectangular imaging object. There is a high possibility. Therefore, the connected region shown in (5) of FIG. 38 is not extracted as the feature region.

  Further, (6) in FIG. 38 shows the connected area determined No in S317. The connection area shown in (6) of FIG. 38 is an area in contact with the image end, and is highly likely to be a background portion other than the imaging target. Therefore, the connected region shown in (6) of FIG. 38 is not extracted as a feature region.

  (7) of FIG. 38 is an area obtained by removing the areas of (5) and (6) from the image shown by (4), and shows the area extracted as the feature area. As illustrated, it can be seen that the region including the boundary between the imaging target and the background is extracted as the feature region.

(3-4) Another Processing Example of Geometrical Arrangement Detection Unit (Part 2)
The geometric arrangement detection unit 111 performs the following process separately from the processes of S2 to S4 to form a straight line corresponding to a group of edge pixels connected in a line segment which is a boundary between the object to be imaged and the background. You may find the formula of

  That is, the geometric arrangement detection unit 111 detects a group of edge pixels connected in a line segment form in the captured image data (performs straight line recognition), and detects the group of detected edge pixels as the object to be imaged and the background. As the border with

  First, the geometric arrangement detection unit 111 performs raster scan on image data captured by the imaging unit 101 and displayed on the display unit 105. Here, as shown in FIG. 17, the forward direction of raster scanning is taken as the X direction, and the direction perpendicular to the X direction is taken as the Y direction. Also, in the captured image, the upper left corner is taken as the origin.

  When scanning for one line is performed and there is no edge, the geometric arrangement detection unit 111 scans for the next line shifted by a predetermined amount in the Y direction. Note that the interval between lines may be constant, and does not have to be one pixel.

  Then, in the raster scan, the geometric arrangement detection unit 111 sets the line at which the edge is first detected as L1 (first line), and as shown in FIG. 18, the point determined as the first edge in the forward direction. The coordinates are stored in a first group (first edge pixel group), and the coordinates of the point determined as the second edge on the same line are classified into a second group (second edge pixel group). Continue scanning the next line to detect edges. Then, for each line Li, a difference between X coordinate values of a point determined first as an edge of the imaging object in the forward direction and a point determined secondly as an edge of the imaging object (X coordinate The distance di) is obtained and the determination is made as follows.

  Note that the X coordinate of the first edge of the line Li is Xi1 (X coordinate classified into the first group), and the X coordinate of the second edge is Xi2 (X coordinate classified into the second group) . The detection method is as follows.

  (A) The coordinates X11 and X12 of the first line (L1) are not changed.

  (B) For the i-th line after the second line, the distances di1 (= Xi1-X (i-1) 1) and di2 (same) between coordinates are calculated. Although the subscript 1 is omitted to describe di1 hereinafter, the same applies to di2.

  (C) For the ith line after the third line, ddi = abs {(di) −di−1} is calculated. If ddi ≦ th1 (small numerical value close to ≒ 0), the coordinates Xi are classified into the same group. If not (ddi> th1), it is classified into another group (third group (third edge pixel group) or fourth group (fourth edge pixel group)).

  (D) As the initial process, only when i = 4, the process for determining the group of X2 is performed. Do as follows.

i) dd3 ≦ th1 and dd4 ≦ th1 → X2: same group
ii) dd3> th1 and dd4 ≦ th1 → X2: another group
iii) dd3 ≦ th1 and dd4> th1 → X2: same group
iv) dd3> th1 and dd4> th1 → X2: The same group Once transition to another group (the third or fourth group), it is not necessary to confirm increase or decrease.

  Such processing is performed on the entire image to extract edge pixels belonging to each group. Then, for each edge pixel group, the coordinates of the edge pixel belonging to the edge pixel group are linearly approximated by the least square method or the like, and a straight line approximated to the edge pixel belonging to the edge pixel group is determined. Here, the geometric arrangement detection unit 111 detects an edge pixel when the average of the sum of squares of the distances between the approximate straight line determined for each edge pixel group and the edge pixels included in the edge pixel group is equal to or less than a predetermined threshold. It can be determined that the groups are arranged in a line, and the following processing can be continued. Thus, the geometric arrangement detection unit 111 can detect a group of edge pixels connected in a line segment shape, which is estimated as the boundary between the object to be imaged and the background. Then, the geometric arrangement detection unit 111 may set the approximate straight lines obtained for each of the detected edge pixel groups as straight lines corresponding to the four sides serving as the boundary between the object to be imaged and the background.

  FIG. 19 is a diagram when edge points are extracted by raster scanning and classified into four edge pixel groups by the above-described processing. In the figure, circle marks indicate edge pixels belonging to the first group, square marks edge pixels belonging to the second group, triangle marks edge pixels belonging to the third group, and star marks indicate edge pixels belonging to the fourth group. An approximate straight line of edge pixels belonging to each edge pixel group, which is obtained by the square method, is indicated by a dotted line.

  Furthermore, the above classification process may be performed on an image rotated 90 degrees. By doing this, it is possible to extract an edge pixel group of a document which is ideally arranged in parallel with the horizontal and vertical directions in the image. That is, by raster scanning, vertical edges can be detected in the image before rotation. On the other hand, in the image after rotation, it is possible to detect an edge pixel group in the horizontal direction before rotation (edge in the vertical direction after rotation). Thereby, edges parallel to the vertical and horizontal directions can also be extracted. If there is a sufficient amount of information before rotation (for example, three or more intersection points in each group), only the information before rotation may be used, and if the intersection point of any group is less than one point, a straight line expression Can not be obtained, so it is sufficient to use the intersection after rotation.

  Alternatively, only the obtained intersection coordinates may be subjected to coordinate conversion again to be returned to the original, and corresponding groups may be obtained from the areas where the respective groups are distributed, and the intersection information may be integrated to obtain straight line expressions. That is, from the coordinates of the point of intersection obtained from the image before rotation and the coordinates of the point of intersection obtained by reversely rotating the point of intersection obtained from the image after rotation, an equation of a straight line You can ask for

  As a method of extracting edge pixels, pixel values within a small window of at least one or more pixel widths are compared as they are (in the case of two or more widths, the sum and average values are compared). If the difference is equal to or greater than a predetermined value, it may be determined that the pixel is an edge pixel. In order to prevent erroneous detection of the background or the edge of the text in the imaging target, only the edge pixel group having a predetermined length or more may be detected as the edge of the imaging target. In this case, the predetermined length may be, for example, about 80% of the length of the outer frame side of the imaging range. As such a detection method, for example, the technology described in Japanese Patent Application Laid-Open No. 2006-237757 can be used. Alternatively, it can be prevented by evaluating each coordinate group or performing processing (such as Hough transform) for line segment detection. Furthermore, by performing processing using a reduced image as pre-processing, it is possible to prevent erroneous detection of the edges of text and fine textures.

(3-5) Another Processing Example of Geometrical Arrangement Detection Unit (Part 3)
Each of the methods for detecting an edge pixel group continuous in the form of a line segment having a high probability of indicating the boundary between the object to be imaged and the background shown in the above (3-2) (3-3) (3-4) There is. For example, in the detection method shown in (3-2), the geometric arrangement detection unit 111 can detect not only a text original but also an illustration original or a photograph as an imaging target object. For a captured image with a pattern or an object in the periphery (a captured image in which the edge of the imaging object and the background are connected by the Canny filter), edge pixel groups indicating the boundary between the imaging object and the background are accurately detected It is difficult to do. Further, in the detection method shown in (3-3), the geometric arrangement detection unit 111 can detect the document even in the case of a captured image having a pattern or an object around the imaging target object. For a captured image such as an illustration original or a photo (a captured image whose edge is interrupted by a Canny filter), an edge pixel group indicating the boundary between the object to be captured and the background may not be detected in some cases.

  Therefore, since there is such a difference in the characteristics according to the extraction methods (3-2), (3-3), and (3-4) described above, a plurality of geometrical arrangement detection units 111 are provided for one captured image. The detection method of may be applied. For example, processing is performed on one captured image in the order of (3-2), (3-3), and (3-4), and first, three sides or four sides that become the boundary between the imaging target object and the background The extraction result information generated in the processing which can extract

  In addition, when the imaging target object and the mobile terminal device 100 are each in a stationary state, there is almost no difference in captured images (frames for preview) captured continuously by the imaging unit 101. Therefore, the geometric arrangement detection unit 111 selects one of the plurality of detection methods of (3-2), (3-3), and (3-4) for each of the captured images continuously captured. It may apply.

  FIG. 39 is a flowchart illustrating determination processing of a detection method to be applied to a captured image. A selection number indicating the selection order is attached in advance to the plurality of detection methods. First, the geometric arrangement detection unit 111 selects the detection method having the first selection number, and generates extraction result information using the selected detection method (S390). For example, the detection method shown in (3-2) is selected.

  Next, the geometric arrangement detection unit 111 confirms extraction result information for the current captured image (frame), and determines whether or not the detection of the boundary between the object to be captured and the background has succeeded (S391). The geometric arrangement detection unit 111 determines that the detection is a failure when the extraction result information indicates that the boundary between the object to be imaged and the background could not be properly extracted, and the detection is not performed otherwise. You should judge that it succeeded.

  If the detection is successful (Yes in S391), the geometric arrangement detection unit 111 turns on the previous detection success flag (S392). Thereafter, the process proceeds to the process of S398.

  On the other hand, if the detection fails (No in S391), the geometric arrangement detection unit 111 confirms whether the previous detection success flag is on (S393). If the previous detection success flag is on (Yes in S393), the geometric arrangement detection unit 111 changes the previous detection success flag to off (S394). On the other hand, when the previous detection success flag is off (No in S393), the geometric arrangement detection unit 111 confirms whether or not there is a detection method having the next selection number (S395). If the next detection method exists (Yes in S395), the detection method is selected (S396), and if the next detection method does not exist (No in S395), the first detection method is selected (S397). Thereafter, the process proceeds to the process of S398.

  Next, in S398, the geometric arrangement detection unit 111 confirms whether it is necessary to continue the display of the captured image. For example, when an instruction to end the document imaging mode is not input, it may be determined that the display continuation of the captured image is necessary.

  In the case of Yes in S398, the geometric arrangement detection unit 111 performs a detection process on the next captured image (frame) by the selected detection method (S399). Thereafter, the process returns to the process of S391.

  FIG. 40 is a diagram showing an example of determining the detection method according to the flowchart of FIG. As illustrated, with respect to the first captured image (first frame), the geometric arrangement detection unit 111 generates extraction result information using the detection method with the first selection number. It is assumed that the previous detection success flag is set to be turned off at the timing when the document imaging mode is started. Therefore, if the detection fails, the second detection method is selected as the detection method to be used for the next captured image (second frame) in S396.

  Then, the geometric arrangement detection unit 111 generates extraction result information on the second captured image (second frame) using the second detection method. Here, if the detection is successful, the previous detection success flag is set to ON in S392.

  Thereafter, when the detection is successful using the second detection method for the third captured image (the third frame), the second detection method remains selected because the detection is successful, and the fourth captured image The detection process is also performed on the (fourth frame) using the second detection method. Even if the detection failure occurs for the fourth captured image, the detection method is not changed because the previous detection success flag is on (Yes in S393). Therefore, the detection process is performed on the fifth captured image (fifth frame) using the second detection method. However, because of the detection failure, the previous detection success flag is switched off.

  After that, if the detection failure for the fifth captured image fails, the previous detection success flag is off, so the next detection method (that is, 3) is used as the detection method to be used for the next captured image (sixth frame) in S396. The second detection method is selected.

  As described above, when the extraction by the first detection method for the first frame fails, the detection process by the second detection method is executed for the next second frame. In addition, when the detection by the second detection method for the fifth frame fails, the detection processing by the third detection method is executed for the next sixth frame. That is, when the detection processing on the previous captured image of the processing target image fails, the detection processing on the captured image of the processing target is performed using a detection method different from the detection method used for the previous captured image. Do. Thereby, the optimal detection method according to the type of captured image can be selected.

  However, as shown in FIGS. 39 and 40, when detection is successful once, even if detection fails once, the detection method is not changed, and when detection fails twice in a row, the detection method is changed. Thus, the detection method is not changed even if the detection fails by any reason such as camera shake.

(3-6) Another Processing Example of Display Processing Unit In S6 of the above (3-2), the display processing unit 112 displays, for each of the captured images, four pieces of extraction result information generated from the captured image. A quadrangular line connecting vertex coordinates is displayed as an outline of the imaging object so as to be superimposed on the imaging image.

  However, when the outline of the object to be imaged is displayed for each captured image captured continuously, the extraction result information is indicated by the extraction result information due to factors such as the arrangement and contrast between the object to be imaged and the background, lighting conditions, and camera shake. Two vertex coordinates may greatly change from one captured image to another. In this case, the display processing unit 112 may perform the following processing because the outline is greatly flickered.

  The display processing unit 112 has a function of storing four vertex coordinates for display, and a function of setting a frame skip flag indicating whether detection of four vertex coordinates has succeeded for the previous captured image (frame). ing. The display processing unit 112 sets the frame skip flag to OFF at the timing when the document imaging mode is activated. Further, it is assumed that the display processing unit 112 erases the four display apex coordinates at the timing when the document imaging mode ends. Therefore, when the document imaging mode is activated, the display processing unit 112 stores the frame skip flag set to OFF and does not store the four display vertex coordinates.

  In S611 illustrated in FIG. 13, the display processing unit 112 may perform the outline display process illustrated in FIG. FIG. 41 is a flowchart showing an example of contour line display processing performed in S611 in the present modification. S611 is a process performed when all four vertex coordinates are within the imaging range (when flags of all vertex coordinates are set (Yes in S610 of FIG. 13)).

  First, the display processing unit 112 checks whether four display vertex coordinates are stored (S621). If the four display vertex coordinates are not stored (No at S621), the display processing unit 112 determines four vertex coordinates indicated by the extraction result information as the four display vertex coordinates (S622). On the other hand, when the display four vertex coordinates are stored (Yes in S621), the display processing unit 112 stores the display four vertex coordinates and the extraction result for each of the upper left, upper right, lower left, and lower right vertices. The midpoint coordinates with each vertex coordinate indicated by the information are obtained, and the four midpoint coordinates are determined as new display four vertex coordinates (S623).

  Thereafter, the display processing unit 112 stores (stores) the determined new four-display coordinates for display (S624), and sets the frame skip flag to ON (S625). Then, the display processing unit 112 displays the outline of the imaging object based on the four display top coordinates stored in S624 (S626).

  According to the process shown in FIG. 41, the four display apex coordinates generated based on the extraction result information generated for the captured image of the previous frame are stored. Then, the midpoint coordinates of the four vertex coordinates indicated by the extraction result information generated for the captured image of the current frame and the four display vertex coordinates are updated as new four vertex coordinates for display, and the contour is based on this. Display a line Therefore, in a plurality of captured images (frames) displayed continuously, it is possible to prevent the flickering of the outline even if the four vertex coordinates indicated by the extraction result information largely change for each captured image due to some factor it can.

  Further, in the case of No in S612, No in S614, or No in S615 in FIG. 13 (when the four vertex coordinates indicated by the extraction result information are not within the image range), the display processing unit 112 performs the process shown in FIG. Do.

  First, the display processing unit 112 confirms whether the frame skip flag is on (S631). If the frame skip flag is on (Yes in S631), the display processing unit 112 confirms whether four display vertex coordinates are stored (S632). If the four display vertex coordinates are stored (Yes in S632), the display processing unit 112 maintains the four display vertex coordinates as it is (S633), and the process proceeds to S635. On the other hand, when the four display vertex coordinates are not stored (No in S632), the process proceeds to S635. Then, in S635, the frame skip flag is set to off.

  On the other hand, if the frame skip flag is off (No in S631), the display processing unit 112 erases the stored four display apex coordinates (S634).

  Thereafter, in S636, the display processing unit 112 displays an outline based on the four display top coordinates. However, the outline is not displayed when the four display vertex coordinates are not stored. Then, the process returns to the process of S617 in FIG.

  FIG. 43 is a diagram showing a screen example of the display unit 105 when the display processing unit 112 operates according to the processes of FIGS. 41 and 42. The processes from S1 to S6 are performed on each captured image (frame) captured continuously. FIG. 43 illustrates a display example of captured images of the first frame (first frame) to the sixth frame (sixth frame).

  In the example of FIG. 43, since the four vertex coordinates indicated by the extraction result information for the first frame immediately after activating the document imaging mode are within the imaging range, the four vertex coordinates are for display in accordance with S622 of FIG. It is stored as 4 vertex coordinates. Therefore, an outline using the four vertex coordinates is displayed.

  Further, since the four vertex coordinates indicated by the extraction result information for the second frame are also within the imaging range, the four display vertex coordinates and the four vertexes stored at the first frame according to S623 of FIG. The middle point with the coordinates is stored as new display 4 vertex coordinates. Then, an outline using the middle point is displayed.

  Next, it is assumed that it is indicated in the extraction result information for the third frame that four vertex coordinates can not be extracted normally. In this case, the process of FIG. 42 is performed. Here, in the second frame, the frame skip flag is set to ON, and the four display vertex coordinates are stored. Therefore, in S636, the outline is displayed based on the four display top coordinates. At this time, the frame skip flag is changed to off.

  As described above, even if extraction processing of four vertex coordinates fails in one frame in the middle of the captured image being displayed continuously, the same outline as that of the previous frame is displayed on the screen. Flicker can be suppressed.

  Since the captured image is an image captured continuously, the position of the imaging target does not usually change significantly. However, depending on the illumination state, only the captured image of a certain frame does not connect the edges well, and as a result, it is determined that all the four vertex coordinates are not within the imaging range, or the positions are shifted due to camera shake or the like. May not be judged. Even in such a case, it is possible to reduce the flicker by directly displaying the four display apex coordinates displayed in the captured image of the previous frame (ignoring the extraction failure of only one frame). it can.

  Next, if the four vertex coordinates indicated by the extraction result information for the fourth frame are within the imaging range, the process proceeds to the flow of FIG. 41, and the four display vertex coordinates stored at the second frame in S623. The middle point between the four vertex coordinates is stored as new display four vertex coordinates. Then, an outline using the middle point is displayed. At this time, the frame skip flag is changed to on.

  Next, it is assumed that it is indicated in the extraction result information for the fifth frame that four vertex coordinates can not be extracted normally. In this case, display processing similar to that of the third frame is performed.

  Next, it is assumed that the extraction result information for the sixth frame indicates that the four vertex coordinates could not be extracted normally. In this case, the process of FIG. 42 is executed, and the frame skip flag is changed to OFF in the fifth frame, so the four display apex coordinates stored in S634 are erased. Therefore, the display processing unit 112 does not display the outline. As described above, when extraction processing of four vertex coordinates fails continuously for two or more frames from the state in which the outline is displayed (the stage of the fourth frame), the outline is not displayed, so the imaging target object is You can understand what you can not recognize.

  In the example shown in FIG. 42 and FIG. 43, although extraction failure of four vertex coordinates with respect to a captured image of one frame is ignored, a counter is provided to count the number of failed frames and extract a plurality of frames. Failure may be ignored.

(4) Transfer of Output Target Image Data to Image Forming Device The user holds the portable terminal device 100, approaches the image forming device 200 or the image display device 300, operates the portable terminal device 100, and performs infrared communication. The output target image data is transmitted to the image forming apparatus 200 or the image display apparatus 300 using the short distance wireless communication method as described above. Specifically, the user inputs a transmission instruction of output target image data to the input unit 106 of the mobile terminal device 100.

  When a transmission instruction of output target image data is input, the control unit 109 specifies output target image data stored in the storage unit 108. Then, the control unit 109 causes the communication unit 104 to execute transmission processing for transmitting the output target image data to the image forming apparatus 200 or the image display apparatus 300. The communication unit 104 transmits the output target image data, the file name associated with the output target image data, the output processing information, and the model information and user information stored in the storage unit 108 together. Do.

(5) Configuration of Image Forming Apparatus Next, the configuration of the image forming apparatus 200 according to the present embodiment will be described. In the present embodiment, the image forming apparatus 200 is a multifunction peripheral provided with functions such as a scanner, a printer, and a copying machine.

  FIG. 20 is a block diagram showing the configuration of the image forming apparatus 200. As shown in FIG. The image forming apparatus 200 includes an image reading unit 201, an image processing unit 202, an authentication unit 203, an image forming unit 204, a display unit 205, an input unit 206, a first communication unit 207, a second communication unit 208, and a recording medium access unit 209. , Storage unit 210, and control unit 212.

  The image reading unit 201 is for reading a document, has a scanner unit provided with a CCD (Charge Coupled Device), and is an electric signal (analog image signal) obtained by color separation of light reflected from the document into RGB. And output this electrical signal.

  The image processing unit 202 performs predetermined image processing on image data. In the present embodiment, the image processing unit 202 performs predetermined image processing on output target image data received from the mobile terminal device 100 or the image display device 300, and generates corrected image data. Details of image processing in the image processing unit 202 will be described later.

  The authentication unit 203 performs user authentication when performing output processing of output target image data received from the mobile terminal device 100 or the image display device 300. Specifically, the authentication unit 203 performs user authentication by collating the user information received from the mobile terminal device 100 with the user information (user ID and password) input to the input unit 206. The authentication unit 203 sends the authentication result to the control unit 212.

  The image forming unit 204 forms an image on a recording sheet such as paper by using, for example, an electrophotographic method or an inkjet method. That is, the image forming unit 204 executes, as one of the output processes, a printing process for printing an image indicated by the corrected image data on a recording sheet such as a recording sheet or an OHP sheet.

  The display unit 205 includes, for example, a liquid crystal display. Further, the input unit 206 is for inputting data by pressing a touch panel or a button of a liquid crystal display, for example.

  The first communication unit 207 has functions of serial transfer, parallel transfer, and wireless data communication based on the USB 1.1 or USB 2.0 standard. The first communication unit 207 receives, from the portable terminal device 100 or the image display device 300, output target image data to which a file name, model information of the portable terminal device 100, user information, and output processing information are added.

  The second communication unit 208 performs (a) data communication using a wireless technology based on any of IEEE802.11a, IEEE802.11b and IEEE802.11g, which is a standard of wireless LAN, and (b) Ethernet (registered trademark). It has the function of a communication interface used and data communication with the network via a LAN cable, (c) wireless communication standard IEEE 802.15.1 (so-called Bluetooth (registered trademark)) or infrared such as IrSimple It has a function of data communication using a wireless technology based on any of communication methods such as a communication standard and Felica (registered trademark).

  The second communication unit 208 performs, as output processing, filing processing in which the corrected image data subjected to the predetermined image processing by the image processing unit 202 is stored in the server, or a corrected image subjected to the predetermined image processing Execute e-mail sending process to send an e-mail with data attached.

  The recording medium access unit 209 reads the program from the recording medium on which the program is recorded. The storage unit 210 is for storing a program for the respective units to execute processing.

  The control unit 212 controls each unit of the image forming apparatus 200. Specifically, when the first communication unit 207 receives output target image data from the mobile terminal device 100 or the image display device 300, the control unit 212 outputs the output target image data to the image processing unit 202, and performs image processing. Run Also, the control unit 212 outputs the user information attached to the output target image data to the authentication unit 203, and causes the authentication unit 203 to execute the authentication process. When the control unit 212 receives the authentication result of the authentication success from the authentication unit 203, the control unit 212 causes the process to be executed according to the output processing information attached to the output target image data. That is, when the output processing information indicates the print processing, the control unit 212 causes the image forming unit 204 to perform printing based on the corrected image data generated by the image processing unit 202. When the output processing information indicates filing processing or mail transmission processing, the control unit 212 executes filing processing or mail transmission processing based on the corrected image data generated by the image processing unit 202 as the second communication unit 208. Make it run.

(6) Image Processing in Image Processing Unit Next, details of the image processing performed by the image processing unit 202 will be described. Although the image processing unit 202 performs image processing also on the image data read by the image reading unit 201, here, the image processing unit 202 performs image processing on output target image data received from the portable terminal device 100 or the image display device 300. The contents will be described.

  FIG. 21 is a block diagram showing an internal configuration of the image processing unit 202. As shown in FIG. As shown in FIG. 12, the image processing unit 202 includes an image quality adjustment unit 221, a geometry correction unit (mapping generation unit) 222, a lens distortion correction unit 223, a high resolution correction unit 225, and an output image processing unit 224. There is. Hereinafter, specific processing contents of each part will be described in order.

(6-1) Image Quality Adjustment Unit The image quality adjustment unit 221 corrects the color balance and contrast of the output target image data. The image quality adjustment unit 221 obtains the maximum value and the minimum value of each color channel for the received output target image data, creates a look-up table that aligns these, and applies it to each color channel. Specifically, as shown in FIG. 22, the image quality adjustment unit 221 uses MX as a lookup table, and MX as the minimum value and MN as the minimum value when the data is 8 bits, as shown in FIG. Create a table that will increase at the step of) / 255. Then, the image quality adjustment unit 221 converts each pixel value according to the created table. Thereby, the color balance is corrected.

  Further, the image quality adjustment unit 221 executes the correction of the contrast in the same manner. If it is not necessary to change the color balance in particular, the look-up table applied to each color channel may be the same.

  Other known techniques may be applied to the color balance / contrast correction method.

(6-2) Lens Distortion Correction Unit The lens distortion correction unit 223 performs lens distortion correction on output target image data.

  The lens distortion correction unit 223 sequentially detects edge pixels of the imaging target object in the captured image by raster scanning in the same manner as the processing described in (3-3) above for the output target image data. Then, the lens distortion correction unit 223 performs curve approximation on the detected edge pixels, and performs lens distortion correction from the equation of the curve.

  Specifically, the lens distortion correction unit 223 detects edge pixels of the detected imaging target object, and in the same manner as the processing described in (3-3) above, each edge pixel includes the imaging target object and the background. It is classified into four edge pixel groups corresponding to the four sides of the boundary. Then, as shown by the solid line in FIG. 23, a quadratic curve approximation is performed on the edge points belonging to each group. The quadratic curves thus obtained for the four groups correspond to the four sides of the object to be imaged. In addition, the lens distortion correction unit 223 obtains an intersection point of the four quadratic curves, which corresponds to a corner of the area surrounded by the four quadratic curves. Next, the lens distortion correction unit 223 circumscribes the quadratic curve determined for each side, and is a circumscribed quadrilateral similar to a quadrangle (indicated by a dotted line in FIG. 23) connecting four intersection points ( Determined in FIG. 23 by dashed dotted line). Then, the lens distortion correction unit 223 converts the pixel position in the area of the imaging target in the captured image so that the circumscribed quadrilateral obtained in this manner becomes the position of the edge pixel of the target after correction. This conversion may be calculated on the basis of a vector from a reference point (e.g., the center of gravity of the area of the object to be imaged). Thereby, lens distortion by the imaging unit 101 of the mobile terminal device 100 can be corrected.

  The lens distortion correction method is not limited to the above-described method, and known techniques can be used.

(6-3) Geometry Correction Unit The geometry correction unit 222 captures an image of a rectangular imaging object such as a poster or manuscript paper from a direction different from the normal direction of the plane on which the document image is formed. The distortion of the object to be imaged (i.e., the distortion of the rectangular flat surface on which the document image is formed) is corrected, and the inclination of the object to be imaged in the image data is corrected.

  Specifically, like the geometric arrangement detection unit 111, the geometry correction unit 222 corresponds to the four edge pixel groups that form the boundary between the rectangular imaging target and the background based on the output target image data. Find the equation of the straight line Then, the geometry correction unit 222 identifies a rectangular area (an area before correction) surrounded by the four straight lines, and cuts out the identified area before correction.

  Next, as shown in FIG. 24, the geometric correction unit 222 determines that the specified rectangular area before correction (indicated by an alternate long and short dash line in FIG. 24) has two upper and lower sides substantially parallel to the horizontal direction. With the aspect ratio and size, the mapping for converting into a rectangular standard area (e.g., 7:10 for A-size and B-size being used for business documents, etc .; shown as a solid line in FIG. 24) Ask. The upper and lower two sides of the standard area may not be completely parallel to the horizontal direction, and may have a slight angle within a predetermined range with respect to the horizontal direction (may be substantially parallel) . Here, the mapping is a rule fx, fy for performing mapping conversion (coordinate conversion processing) from the coordinates (x1, y1) of each pixel of the pre-correction area to the coordinates (x2, y2) of the corresponding pixel of the standard area. X2 = fx (x1, y1), y2 = fy (x1, y1). A known technique can be used as the mapping conversion. The geometry correction unit 222 may perform conversion so as to match the aspect ratio stored in advance in the storage unit 210, or may perform conversion so as to match the aspect ratio input to the input unit 206. Further, as the size of the standard area, the size input to the input unit 206 may be set, or the size may be set so as to have the same area as the area before correction.

  Next, the geometry correction unit 222 performs coordinate conversion on the pre-correction region extracted from the output target image data according to the determined mapping. This makes it possible to correct geometric distortion and inclination (hereinafter sometimes referred to as geometric correction).

  The method of geometric correction is not limited to the method described above, and known techniques can be used.

(6-4) High Resolution Correction Unit The high resolution correction unit 225 performs high resolution correction on output target image data. In the present embodiment, the high resolution correction unit 225 performs high resolution correction based on one output target image data.

  Several methods have been introduced in the Journal of the Institute of Image Information and Television Engineers Vol. 62, No. 2, pp. 181-189 (2008) regarding high-resolution image creation methods from one image data.

  Generally, the edge directionality of the image pattern is detected, interpolation is performed according to the direction, and noise removal processing is performed for the purpose of removing distortion due to interpolation and the influence of noise components present in the input image. Thus, high resolution correction can be performed. The details will be described below.

  FIG. 25 is a flowchart showing the flow of high resolution correction processing in the present embodiment. Here, an example in which double resolution conversion is performed for each of the horizontal direction and the vertical direction will be described. In the case of performing resolution conversion twice, if the number of pixels of output target image data to be corrected is n × m, the number of pixels of high resolution image data after correction is 2 n × 2 m. Such high resolution correction (double resolution conversion) uses each pixel in the output target image data as a reference pixel, generates a new pixel at the center between the reference pixels as an interpolation pixel, and the reference pixel and the interpolation pixel And image data including both of the image data and the image data as high resolution image data. FIG. 26 shows the relationship between the reference pixel and the interpolation pixel, where the pixel a indicates the reference pixel and the pixel b indicates the interpolation pixel.

  First, the high resolution correction unit 225 performs edge extraction on output target image data. For example, the high resolution correction unit 225 performs edge extraction using a first derivative filter as shown in FIG. 27, performs binarization processing, and generates binarized image data (S40). Note that if the pixel value is 1 in the binarized image data, it indicates that the pixel is highly likely to be an edge.

  Next, the high resolution correction unit 225 determines whether the target pixel in the captured image data is an edge based on the binarized image data generated in S40 (S41). Specifically, if the value of the pixel corresponding to the target pixel in the binarized image data is 1, the high resolution correction unit 225 determines that the target pixel is an edge.

  In addition, a focused pixel refers to a focused pixel when focused on each pixel in captured image data in an arbitrary order.

  If the pixel of interest is an edge (Yes in S41), the high resolution correction unit 225 detects an edge direction using a partial image of N × N (N> 1) including the pixel of interest (S42). Specifically, it is determined whether or not all the reference pixels included in the N × N partial image are edge pixels. When the upper left reference pixel and the lower right reference pixel of the pixel of interest are edge pixels, the high resolution correction unit 225 determines that the edge direction in the partial image is the upper left lower right direction. Similarly, when the left reference pixel and the right reference pixel of the pixel of interest are edge pixels, it is determined that the edge direction is the left-right direction, and the reference pixels above and below the pixel of interest are If it is an edge pixel, it is determined that the edge direction is up-down, and if the reference pixel at the upper right of the pixel of interest and the reference pixel at the lower left are edge images, it is determined that the edge direction is upper right-lower left Do.

  In FIG. 28, dotted lines indicate detected edge directions. In FIG. 28, pixels (1) to (9) are reference pixels, and pixel (5) among them is the target pixel. Then, pixels A, B and C are respectively interpolation pixels between reference pixels (1) and (5), interpolation pixels between reference pixels (2) and (5), and reference pixel (4). Interpolated pixel between (5) and (5).

  Next, the high resolution correction unit 225 calculates the pixel values of the interpolation pixel A at the upper left of the target pixel, the interpolation pixel B above the target pixel, and the interpolation pixel C at the left of the target pixel according to the edge direction detected in S42. Calculated by interpolation. At this time, the pixel value of the interpolation pixel is determined using the reference pixel along the edge direction.

When the edge direction is the upper left-lower right direction, as shown in FIG. 28A, the reference pixels (1), (5), (9) are edge pixels, and the line connecting these pixels is an edge It becomes a line. Then, for the pixel value VA of the interpolation pixel A on the edge line (notation of “V” in the figure is omitted. The same applies to the following), the reference pixel (1) (pixel value on the edge line adjacent to the interpolation pixel A). Using the pixel values of V (1) and the reference pixel (5) (pixel value V (5)), the following equation VA = (V (1) + V (5)) / 2
It asks by.

On the other hand, for the interpolation pixels B and C not on the edge line, among the reference pixels other than the reference pixels on the edge line, the reference pixel (closest reference pixel) closest to the interpolation pixel is included and the line parallel to the edge direction Interpolate using the reference pixel of For example, in FIG. 28A, the interpolation pixel B includes the reference pixel (2) which is the closest reference pixel, and a line parallel to the edge direction is a line connecting the reference pixels (2) and (6). It is. The point perpendicular to the line from the interpolation pixel B internally divides the line connecting the reference pixels (2) and (6). Therefore, the pixel value VB of the interpolation pixel B is expressed by the following equation VB = (9 × V (2) + 4 × V (6)) / 13
Find using

Similarly, the pixel value VC of the interpolation pixel C is the pixel value of the reference pixel (4) which is the closest reference pixel and the reference pixel (8) on the line parallel to the edge direction including the reference pixel (4). Using the following equation VC = (9 × V (4) + 4 × V (8)) / 13
It asks by.

When the edge direction is left-right, as shown in FIG. 28B, the reference pixels (4), (5) and (6) are edge pixels, and the line connecting these pixels is It becomes an edge line. Then, for the pixel value VC of the interpolation pixel C on the edge line, the reference pixel (4) (pixel value V (4)) and the reference pixel (5) (pixel value V (5) on the edge line adjacent to the interpolation pixel C The following equation VC = (V (4) + V (5)) / 2 using the pixel value of
It asks by.

On the other hand, for the interpolation pixels A and B not on the edge line, among the reference pixels other than the reference pixels on the edge line, the reference pixel (closest reference pixel) closest to the interpolation pixel is included and the line parallel to the edge direction Interpolate using the reference pixel of For example, in FIG. 28B, the interpolation pixel A includes the reference pixel (1) or (2) which is the closest reference pixel, and a line parallel to the edge direction corresponds to the reference pixels (1) and (2). It is a line connecting Then, a point perpendicular to the line from the interpolation pixel A exists at the center of the reference pixels (1) and (2). Therefore, the pixel value VA of the interpolation pixel A is expressed by the following equation VA = (V (1) + V (2)) / 2
Find using

  The interpolation pixel B includes the reference pixel (2) which is the closest reference pixel, and a line parallel to the edge direction is a line connecting the reference pixels (1) and (2) and (3). Then, the point vertically lowered from the interpolation pixel B to the line coincides with the reference pixel (2). Therefore, the pixel value VB of the interpolation pixel B is set to the same value as the pixel value V (2) of the reference pixel (2).

  When the edge direction is the upper right-lower left direction, as shown in FIG. 28C, the reference pixels (3), (5) and (7) are edge pixels, and the line connecting these pixels is It becomes an edge line. And all of the interpolation pixels A, B, C do not exist on the edge line.

For the interpolation pixel A, the closest reference pixel is the reference pixel (1), (2), (4). Here, the reference pixels (2) and (4) are located on the same line parallel to the edge direction, but the reference pixel (1) is not located on the line. Therefore, for the pixel value VA of the interpolation pixel A, using the pixel values of the reference pixels (1), (2) and (4) which are nearest reference pixels, the following equation VA = (V (1) + V (2) ) + V (4)) / 3
It asks by.

On the other hand, for the interpolation pixels B and C, among the reference pixels on the edge line except the reference pixel, the reference pixel (closest reference pixel) closest to the interpolation pixel is included and the reference pixel on the line parallel to the edge direction is selected. Use to interpolate. For example, in FIG. 28C, the interpolation pixel B includes the reference pixel (2) which is the closest reference pixel, and a line parallel to the edge direction is a line connecting the reference pixels (2) and (4). It is. Then, the point perpendicular to the line from the interpolation pixel B internally divides the line connecting the reference pixels (2) and (4). Therefore, the pixel value VB of the interpolation pixel B is expressed by the following equation VB = (9 × V (2) + 4 × V (4)) / 13
Find using

Similarly, the pixel value VC of the interpolation pixel C is the pixel value of the reference pixel (4) which is the nearest reference pixel and the reference pixel (2) on the line parallel to the edge direction including the reference pixel (4). Using the following equation VC = (4 x V (2) + 9 x V (4)) / 13
It asks by.

When the edge direction is up-down, as shown in FIG. 28D, the reference pixels (2), (5), (8) are edge pixels, and the line connecting these pixels is It becomes an edge line. Then, for the pixel value VB of the interpolation pixel B on the edge line, using the pixel values of the reference pixel (2) and the reference pixel (5) on the edge line adjacent to the interpolation pixel B, the following equation VC = (V (V ( 2) + V (5)) / 2
It asks by.

On the other hand, for the interpolation pixels A and C not on the edge line, among the reference pixels other than the reference pixels on the edge line, the reference pixel (closest reference pixel) closest to the interpolation pixel is included and the line parallel to the edge direction Interpolate using the reference pixel of For example, in FIG. 28D, the interpolation pixel A includes the reference pixel (1) or (4) which is the closest reference pixel, and the line parallel to the edge direction corresponds to the reference pixels (1) and (4). It is a line connecting Then, a point vertically lowered from the interpolation pixel A to the line exists at the center of the reference pixels (1) and (4). Therefore, the pixel value VA of the interpolation pixel A is expressed by the following equation VA = (V (1) + V (4)) / 2
Find using

  The interpolation pixel C includes the reference pixel (4) which is the closest reference pixel, and a line parallel to the edge direction is a line connecting the reference pixels (1), (4) and (7). Then, the point vertically lowered from the interpolation pixel C to the line coincides with the reference pixel (4). Therefore, the pixel value VC of the interpolation pixel C is set to the same value as the pixel value V (4) of the reference pixel (4).

  The storage unit 210 stores, in advance, information in which the edge direction is associated with an arithmetic expression for obtaining pixel values of the interpolation pixels A, B, and C. Then, the high resolution correction unit 225 may read the arithmetic expression corresponding to the edge direction detected in S42 from the storage unit 210, and may obtain the pixel values of the interpolation pixels A, B, and C based on the read arithmetic expression. .

  FIG. 28 shows only the case where the edge direction is linear. However, the edge may bend in the N × N partial image. For example, the edge may be bent as in the reference pixels (2)-(5)-(4), or the edge may be bent as in the reference pixels (1)-(5)-(7). Also in such a case, information associated with arithmetic expressions for obtaining pixel values of the interpolation pixels A, B, and C is stored in advance. For example, when the edge bends as in the reference pixels (2)-(5)-(4), the interpolation pixel A is similar to FIG. 28 (c) and the interpolation pixel B is similar to FIG. 28 (b). For the interpolation pixel C, the same arithmetic expression as that shown in FIG. 28D is stored. Further, when the edge is bent as in the reference pixels (1)-(5)-(7), the interpolation pixel A is the same as FIG. 28A, and the interpolation pixel B is the same as FIG. For the interpolation pixel C, the same arithmetic expression as that shown in FIG. 28D is stored. The other edge direction patterns are similarly stored.

  Thus, the high resolution correction unit 225 obtains the pixel value of the interpolation pixel located around the reference pixel determined to be an edge pixel.

  On the other hand, when the pixel of interest is not an edge (No in S41), the high resolution correction unit 225 determines that the interpolation pixel A adjacent to the upper left of the pixel of interest, the interpolation pixel B adjacent to the pixel of interest above, the left of the pixel of interest The pixel value of the interpolation pixel C is calculated by a general interpolation method (bilinear bicubic etc.) (S43).

  The high resolution correction unit 225 executes the above-described processing of S41 to S43 for all reference pixels included in one image data to generate interpolation image data including both the reference pixels and the interpolation pixels ( S44).

  Thereafter, the high resolution correction unit 225 performs an image quality enhancement process on the generated interpolated image data. For example, the high resolution correction unit 225 applies a noise removal filter, a sharpening filter, or the like to the interpolated image data to generate high resolution image data. A conventional unsharp mask or one having the center coefficient of FIG. 27 as 5 is a sharpening filter. Median filters are widely known as noise removal. As a more advanced method, a Bilateral filter [Proceedings of the 1998 IEEE International Conference on Computer Vision,] or the like may be used as a method having both the above-mentioned edge preservation and high image quality.

  The high resolution correction unit 225 is not limited to the method described above, and may use various methods as described in the Journal of The Institute of Image Information and Television Engineers Vol. 62, No. 2, pp. 181 to 189 (2008). Thus, high resolution image data may be generated from one piece of captured image data.

(6-5) Output Image Processing Unit The output image processing unit 224 executes region separation processing, color correction, black generation and under color removal processing, spatial filter processing, and halftone processing when outputting output target image data. It is. In addition, a well-known technique can be used as these processes.

(7) Flow of Processing of Image Forming Apparatus Hereinafter, the flow of processing of the image forming apparatus 200 according to the present embodiment will be described. FIG. 29 shows the process flow of the image forming apparatus 200.

  The first communication unit 207 of the image forming apparatus 200 receives the output target image data, the model information, the user information, and the output processing information from the portable terminal device 100 (S30).

  The image quality adjustment unit 221 of the image processing unit 202 corrects the color balance and the contrast of the received output target image data as described in, for example, the above (6-1) (S31). Subsequently, the lens distortion correction unit 223 performs lens distortion correction on the received output target image data as described in (6-2) above.

  Furthermore, the geometric correction unit 222 performs geometric distortion correction and inclination correction on the output target image data (S32). Specifically, the geometry correction unit 222 specifies and cuts out a rectangular area (an area before correction) surrounded by a straight line corresponding to each of the four sides corresponding to the boundary between the object to be imaged and the background. Next, the geometry correction unit 222 converts the rectangular area before correction into a rectangular standard area whose upper and lower sides are substantially parallel in the horizontal direction and which has a predetermined aspect ratio and size. Find a mapping about the coordinates for Then, using the mapping, the geometry correction unit 222 performs mapping conversion processing on the image data of the uncorrected region that has been cut out. This makes it possible to output an image free of geometric distortion.

  Next, the high resolution correction unit 225 performs high resolution correction on the image data subjected to the process of S32 (S33). The specific method of the high resolution correction is as described in (6-4).

  Next, the output image processing unit 224 performs region separation processing, color correction, black generation and under color removal processing, spatial filter processing, and intermediate filtering processing on the high resolution image data obtained by the high resolution correction in the high resolution correction unit 225. Execute various image processing such as tone processing. The output image processing unit 224 appropriately switches the contents of the various types of image processing according to the type of the output method indicated by the output processing information. Then, the output image processing unit 224 stores the corrected image data (corrected image data) in the storage unit 210 (S34). At this time, the output image processing unit 224 stores the corrected image data in association with the user information and the output processing information received together with the output target image data that is the source of the corrected image data.

  Thereafter, the control unit 212 determines whether an output instruction of the output target image data is input to the input unit 206 (S35). If the output instruction has not been input (NO in S35), the process waits until the output instruction is input.

  On the other hand, when there is an output instruction (YES in S35), authentication unit 203 causes display unit 205 to display a screen prompting input of user information (for example, user ID and password), and acquires user information from input unit 206. Then, the authentication unit 203 performs user authentication (S36). The authentication unit 203 may acquire user information from the noncontact IC card possessed by the user using a noncontact IC card reader / writer provided in the image forming apparatus 200.

  When performing user authentication, the authentication unit 203 compares the input user information with the user information received from the portable terminal device 100, and determines whether there is matching user information (S37). Then, when the user information matching the input user information is received from the portable terminal device 100 (YES in S37), the control unit 212 is stored in the storage unit 210 in S34, and the correction corresponding to the user information The output processing of the finished image data is executed according to the output processing information received from the portable terminal device 100 (S38).

  For example, when the output processing information indicates print processing, the control unit 212 causes the image forming unit 204 to print the image indicated by the corrected image data. When the output processing information indicates filing processing or mail transmission processing, the control unit 212 causes the second communication unit 208 to execute filing processing or mail transmission processing based on the corrected image data. Thereafter, the process ends.

  On the other hand, when the input user information does not match the user information received from the portable terminal device 100 (NO in S37), the authentication unit 203 determines whether the number of authentications is equal to or more than a predetermined number (S39). Then, if the number of authentications is not equal to or more than the predetermined number (NO in S39), the processes of S36 and S37 are repeated. If the number of authentications is equal to or greater than the predetermined number (YES in S39), the flow ends without output.

(8) Modified Example The captured image processing system of the present invention is not limited to the above embodiment, and various modifications are possible. Hereinafter, specific examples of the modified embodiment will be described.

(8-1) Regarding Display of First Missing Information In the above description, the display processing unit 112 specifies a point closest to the vertex coordinates from the imaging range based on the vertex coordinates outside the imaging range, and identifies The first missing information is displayed within a predetermined range. Not limited to this, the display processing unit 112 may display the first missing information in the vicinity of the vertex outside the imaging range.

  For example, as shown in FIG. 30, among vertices of a quadrangle surrounded by four straight lines detected by the geometric arrangement detection unit 111, vertices outside the imaging range are referred to as a first vertex T1 and a first vertex T1. Two adjacent vertices are taken as a second vertex T2 and a third vertex T3, and a vertex located diagonally to the first vertex T1 is taken as a fourth vertex T4. Further, an intersection point of a line connecting the first vertex T1 and the second vertex T2 with the frame of the imaging range is a first intersection point S1, an intersection point of a line connecting the first vertex T1 with the third vertex T3 and the frame of the imaging range As a second intersection point S2. At this time, the display processing unit 112 may display the first dropout information within a predetermined range from a line segment connecting the first intersection point S1 and the second intersection point S2.

(8-2) Display of Second Missing Information In the above description, the display processing unit 112 determines in advance from the side of the frame of the imaging range corresponding to the side indicated by the extraction impossible information in the extraction result information. The second missing information is displayed within the range. Not limited to this, the display processing unit 112 may display the second missing information in the vicinity of the side outside the imaging range.

  For example, as shown in FIG. 31, a straight line corresponding to each of three line segments indicated by three edge pixel groups detected by the geometrical arrangement detection unit 111 is a first straight line L1, a second straight line L2, Within the imaging range, a first intersection point T5 which is a third straight line L3 and which is an intersection point of the first straight line L1 and the second straight line L2 and a second intersection point T6 which is a crossing point of the second straight line L2 and the third straight line L3. Further, among the intersections of the first straight line L1 and the frame of the imaging range, the intersection farther from the first intersection T5 is the third intersection S3, the third straight line L3 of the intersection of the frame of the imaging range A fourth intersection point S4 is taken as the intersection point far from the second intersection point T6. At this time, the display processing unit 112 may display the second dropout information within a predetermined range from a line segment connecting the third intersection point S3 and the fourth intersection point S4T8.

(8-3) Transmission of Output Target Image Data (8-3-1) Part 1
In the above description, image data indicating the entire image captured by the imaging unit 101 is used as output target image data. At this time, the communication unit 104 may transmit a straight line equation corresponding to each of the four edge pixel groups detected by the geometric arrangement detection unit 111 together with the output target image data.

  In this case, the geometric correction unit of the image forming apparatus cuts out an area surrounded by the straight lines from the output target image data using the equation of the four straight lines added to the output target image data. Then, the geometry correction unit uses a rectangular standard area (e.g., a business document), the upper and lower two sides of which are substantially parallel in the horizontal direction and having a predetermined aspect ratio and size. If it is A size B size 7: 10 etc. (shown by a solid line in FIG. 15), find a mapping for conversion. Then, the geometry correction unit may perform geometry correction on the extracted image according to the determined mapping.

  Alternatively, the communication unit 104 may transmit, together with the output target image data, four vertex coordinates of a quadrangle surrounded by a straight line corresponding to each of the four edge pixel groups detected by the geometric arrangement detection unit 111. .

  Even in this case, the geometry correction unit 222 of the image forming apparatus 200 cuts out, from the output target image data, a quadrilateral area whose vertex is the point of the four vertex coordinates added to the output target image data. Then, the geometry correction unit obtains a mapping for converting the extracted image into a rectangular standard area. Then, the geometry correction unit may perform geometry correction in accordance with the determined mapping.

  When the quadrangle surrounded by straight lines corresponding to each of the four edge pixel groups detected by the geometric arrangement detection unit 111 is a rectangle, the communication unit 104 combines the output target image data with the quadrilateral shape. The angle (inclination) between the lower side and the lower side of the frame of the imaging range may be transmitted.

  In this case, the geometry correction unit of the image forming apparatus cuts out from the output target image data a quadrilateral area whose vertex is the point of the four vertex coordinates added to the output target image data. Then, the geometric correction unit can correct the inclination by rotating the image extracted by the angle added to the output target image data.

(8-3-2) 2
Further, the output target image determination unit 113 is a quadrangle surrounded by straight lines corresponding to each of the four edge pixel groups detected by the geometric arrangement detection unit 111 out of the entire image captured by the imaging unit 101. An area may be cut out and image data representing the cut out image may be used as output target image data. Then, the communication unit 104 transmits the determined output target image data to the image forming apparatus.

  In this case, the geometric correction unit of the image forming apparatus has a rectangular shape in which the upper and lower two sides of the image represented by the output target image data are substantially parallel in the horizontal direction and have a predetermined aspect ratio and size. A mapping for conversion into a standard area of (for example, 7:10 for A-size and B-size being used in a business document, etc. shown by a solid line in FIG. 15) is obtained. Then, the geometry correction unit may perform geometry correction in accordance with the determined mapping.

(8-3-3) The 3
As described in (8-3-2) above, the output target image determination unit 113 selects four edge pixels detected by the geometric arrangement detection unit 111 from among the entire image captured by the imaging unit 101. A rectangular area surrounded by a straight line corresponding to each of the groups is cut out, and image data representing the cut out image is set as output target image data.

  Then, the communication unit 104 may transmit a straight line equation corresponding to each of the four edge pixel groups detected by the geometric arrangement detection unit 111 together with the determined output target image data.

  In this case, the geometric correction unit of the image forming apparatus obtains a mapping for converting the area surrounded by the straight lines into a rectangular standard area using the equation of four straight lines added to the output target image data. . Then, the geometry correction unit may perform the geometry correction on the output target image data in accordance with the determined mapping.

  Alternatively, the communication unit 104 transmits, together with the determined output target image data, four vertex coordinates of a quadrangle surrounded by a straight line corresponding to each of the four edge pixel groups detected by the geometric arrangement detection unit 111. May be

  Even in this case, the geometric correction unit of the image forming apparatus obtains a mapping for converting a quadrilateral area having the four vertex coordinates added to the output target image data as the vertices into a rectangular standard area. Then, the geometry correction unit may perform the geometry correction on the output target image data in accordance with the determined mapping.

  When the quadrangle surrounded by straight lines corresponding to each of the four edge pixel groups detected by the geometric arrangement detection unit 111 is a rectangle, the communication unit 104 combines the output target image data with the quadrilateral shape. The angle (inclination) between the lower side and the lower side of the frame of the imaging range may be transmitted.

  In this case, the geometric correction unit of the image forming apparatus can correct the inclination by rotating the output target image data by an angle added to the output target image data.

(8-4) Method of Determining Output Target Image Determination Unit In the above description, the output target image determination unit 113 causes the display processing unit 112 to display information indicating that the imaging target is contained within the imaging range (for example, “ The image data indicating the captured image displayed on the display unit 105 is determined as the output target image data at the timing when the shutter button (symbol 10 in FIG. 14) is operated when the “OK” is displayed.

  However, the output target image determination unit 113 may have an auto shutter function of automatically determining output target image data instead of the operation of the shutter button 10.

  That is, when the auto shutter function is effective, the output target image determination unit 113 causes the display processing unit 112 to perform imaging target objects on a predetermined number of captured images (frame images) captured continuously by the imaging unit 101. When displaying information indicating that the image data is included in the imaging range, image data indicating one of the predetermined number of captured images (for example, the captured image captured last) is used as output target image data decide.

  More specifically, the output target image determination unit 113 outputs, from the geometric arrangement detection unit 111, a predetermined number (for example, 30) of captured images (frame images) captured continuously by the imaging unit 101. The four vertex coordinates (except temporary vertex coordinates) indicated by the extracted extraction result information are stored. Then, when the square of the difference between the vertex coordinates obtained from the predetermined number of captured images is smaller than a predetermined threshold value for each vertex, the output target image determination unit 113 is displayed on the display unit 105. Image data representing a captured image may be determined as output target image data.

Here, the predetermined threshold is, for example,
(Height of imaging range Ymax × 1/16) 2+ (Width of imaging range Xmax × 1/16) 2
Is set.

  As a result, the output target image data can be determined without the user operating the shutter button, so that it is possible to prevent the imaging target object from moving due to operating the shutter button and the imaging target object from being out of the imaging range.

  The user can switch on / off of the auto shutter function. This switching is set by operating the condition setting button 50 shown in FIG. In addition, when the auto shutter function is effective and the output target image data is determined by the auto shutter function, the output target image determining unit 113 displays a third icon D indicating that as shown in FIG. Displayed on 105.

(8-5) On / Off Switching of Display Function of Contour Line FIG. 44 is a block diagram showing a modification of the imaging range determination unit. As shown in FIG. 44, the imaging range determination unit 110a according to the present modification differs from the imaging range determination unit 110 in that the switching unit 114 is provided.

  The switching unit 114 switches on / off of the display function (document correction mode) of the outline by the geometric arrangement detection unit 111 and the display processing unit 112 according to a user input.

  As shown in FIG. 45, the switching unit 114 causes the display unit 105 to display a check box 451 for switching on / off of the document correction mode. When an instruction to switch on the document correction mode is input, the switching unit 114 operates the geometric arrangement detection unit 111 and the display processing unit 112 to execute the processes of S1 to S6 in FIG. 4. Thus, the outline of the object to be imaged is displayed superimposed on the imaged image, and whether or not the object to be imaged is within the imaging range can be easily confirmed.

  On the other hand, when an instruction to switch the document correction mode off is input, the switching unit 114 causes only the captured image to be displayed without operating the geometric arrangement detection unit 111 and the display processing unit 112.

  Note that the switching unit 114 may receive an instruction to switch on / off the document correction mode before the start of imaging, or may receive it during imaging.

  Also, the present modification may be combined with the modification described in (8-4). In this case, the auto shutter function can be set to be effective only when the document correction mode is set to on. That is, in the example shown in FIG. 45, the user checks the check box 452 for the auto shutter function only when the check is input to the check box 451 for the document correction mode (when the document correction mode is on). You can enter

(8-6) Editing view function The display processing unit 112 of the mobile terminal device 100 performs geometric correction and high image quality correction on the output target image data determined by the output target image determining unit 113 based on the outline. A function (corrected image preview function) may be provided to display a preview of the image (corrected image) after the image correction processing of (1) is performed.

  In the present modification, the display processing unit 112 has the functions of the image quality adjustment unit 221, the geometry correction unit 222, and the high resolution correction unit 225 described above. Then, when the display processing unit 112 receives an execution instruction of the correction image preview function, the display processing unit 112 reads the designated output target image data. The display processing unit 112 causes the geometric arrangement detection unit 111 to execute the processes of S1 to S5 (FIG. 4) on the captured image indicated by the output target image data. The display processing unit 112 generates an image after correction processing by performing image quality adjustment, geometric correction, and high resolution correction on the image of the area surrounded by the outline connecting the four vertex coordinates calculated in S5. And causes the display unit 105 to display a preview.

  FIG. 46 shows an example of a screen on which an image after correction processing is displayed. As shown in FIG. 46, the display processing unit 112 superimposes the outline on the captured image represented by the output target image data and displays it in the region A, and displays the image after correction processing in the region B.

  The display processing unit 112 may display the image after correction processing even before the shutter button is pressed.

  In addition, the display processing unit 112 may perform only the image quality adjustment and the geometric correction to generate an image after the correction processing.

  Furthermore, the display processing unit 112 receives an instruction to edit the four vertex coordinates calculated by the geometric arrangement detection unit 111, and based on the image of the area surrounded by the outline connecting the four vertex coordinates after editing. An image after correction processing may be generated. For example, as illustrated in FIG. 46, the display processing unit 112 causes the editing icons 461 to 464 corresponding to each of the four vertices in the area A to be displayed. Then, the display processing unit 112 receives the input of the position change instruction of each of the editing icons 461 to 464, edits the four vertex coordinates according to the position of each of the editing icons 461 to 464 after the change, and detects the position of the outline. You should decide.

  In this case, the communication unit 104 transmits the four vertex coordinates after editing together with the output target image data, as described in (8-3-1). Thereby, the user can cause the image forming apparatus 200 to output the area surrounded by the desired outline.

  Further, as shown in FIG. 46, the display processing unit 112 may cause the display unit 105 to display a check box 465 for switching whether or not to display the image after correction processing in the region B. Then, in response to the input to the check box 465, the display processing unit 112 switches the presence or absence of the display of the image after the correction processing to the area B.

  Furthermore, the display processing unit 112 may cause the display unit 105 to display a bar 466 for changing the degree of image quality correction, as shown in FIG. Then, in response to the input to the bar 466, the display processing unit 112 changes the degree of image quality adjustment (contrast correction). For example, the bar 466 can be used to change the image quality to any one of four levels of "none", "weak", "medium" and "strong".

  Furthermore, the display processing unit 112 may display the registration button 467 on the display unit 105 as shown in FIG. When the registration button 467 is pressed, the output target image data set at the pressed timing, the four vertex coordinates, and the image quality adjustment level are stored in the storage unit 108 in association with each other. Then, when transmitting the output target image data to the image forming apparatus 200, the communication unit 104 also transmits the four vertex coordinates and the image quality adjustment level associated with the output target image data. In this case, the geometry correction unit 222 of the image forming apparatus 200 cuts out a quadrilateral area whose vertex is the point of the four vertex coordinates added to the output target image data from the output target image data, and cuts the cut out image A rectangular standard area geometry correction may be performed. Further, the image quality adjustment unit 221 of the image forming apparatus 200 may perform the image quality adjustment according to the image quality adjustment level added to the output target image data. Thus, the user can set the area and the image quality of the image output from the image forming apparatus 200 while looking at the screen of the display unit 105 of the mobile terminal device 100.

  The present modification may be combined with the modification described in (8-5). In this case, the corrected image preview function is automatically turned on when the document correction mode is set to on. That is, when the output target image data is determined by pressing the shutter button 453 or the auto shutter function in FIG. 45, the display processing unit 112 automatically recognizes that the execution instruction of the correction image preview function is input. Then, the display processing unit 112 may execute the corrected image preview function and display a screen as shown in FIG. 46 on the display unit 105.

(8-7) Image Display Device The image display device 300 may include the image processing unit 202 included in the image forming device 200. Then, the image display apparatus 300 may perform, as output processing, display processing for displaying an image represented by corrected image data subjected to geometric correction and high resolution correction on output target image data.

(8-8) Mobile Terminal Device When the imaging condition of the imaging object is bad, it may be difficult to confirm the image even if the image processing is performed in the image forming device. For example, when imaging is performed in a state in which geometric distortion is relatively large, it is difficult to visually recognize even if geometric correction is performed.

  Therefore, it is determined in the mobile terminal device 100 whether or not the output target image data is captured under the condition that the image processing in the image forming device produces an effect, and the user is prompted to re-image according to the determination result. Good.

  FIG. 33 is a block diagram showing a configuration of mobile terminal apparatus 100 according to the present modification. As shown in FIG. 33, the mobile terminal device 100 according to the present modification is different from the mobile terminal device shown in FIG. 2 in that the output image determination unit 111 is provided.

  When the document imaging mode is selected, the output image determination unit 114 determines whether the output target image data determined by the imaging range determination unit 110 is captured under the condition that the image processing in the image forming apparatus produces an effect. To determine the Examples of the determination items in the output image determination unit 114 include the following.

(8-8-1) Determination of Inclination The output image determination unit 114 selects two points on the edge of the imaging target object and the background from the output target image data. For example, as shown in FIG. 34, two points 11 and 12 separated by w / 2 in the lateral direction respectively from the center of the sub-captured image data are selected. Next, the distances d1 and d2 between each of the selected two points 11 and 12 and the edge of the captured image data can be determined, and the inclination of the object to be imaged in the captured image can be determined. In the case shown in FIG. 34, tan θ = (d2−d1) / w, where θ is the inclination angle. Therefore, the output image determination unit 114 calculates the value of (d2−d1) / w, and reads the angle θ corresponding to this value from a table (see FIG. 35) or the like that has been created in advance.

  Then, the output image determination unit 114 determines whether the detected angle θ is within a predetermined range (for example, -30 ° to + 30 °), and outputs the determination result to the control unit 109. Here, it is one of the processing execution conditions that the angle θ is within a predetermined range.

(8-8-2) Determination of Geometric Distortion The output image determination unit 114 specifies a quadrangle surrounded by a straight line corresponding to the four edge pixel groups detected by the geometric arrangement detection unit 111. Then, the output image determination unit 114 calculates the ratio of the lengths of the opposite sides of the identified square. In addition, since there are two sets of opposite sides, the output image determination unit 114 obtains a length ratio for each of the two sets.

  Here, the ratio of the length of the opposite side is 1, since the imaging object in the captured image is also rectangular when imaging the rectangular imaging object from the front. On the other hand, when imaging is performed in an oblique direction, the shape of the imaging target object in the captured image is a distorted quadrilateral, and thus has a value different from 1. As the angle between the imaging direction and the normal direction of the plane on which the document image of the imaging object is formed is larger, the difference between the value of the ratio and 1 is larger. Therefore, it can be said that the ratio of the length of the opposite side is one of the feature quantities indicating the degree of geometric distortion.

  Thereafter, the output image determination unit 114 determines whether both of the two calculated ratios are within a predetermined range (for example, 0.5 to 2), and outputs the determination result to the control unit 109. Here, the predetermined range is predetermined as a range that can be corrected by the image forming apparatus 200, and is stored in the storage unit 108.

  The output image determination unit 114 may use, for example, an angle formed by two straight lines including four intersection points detected as described above, as another feature amount indicating the degree of geometric distortion.

(8-8-3) Others In addition to the above-described determination items, the output image determination unit 114 may determine, for example, brightness, contrast, color balance, blurring (strong camera shake), and the like.

  With regard to the brightness, for example, in the case of overexposure (too bright) or under (too dark), it may be considered that another image capture is required. Therefore, the output image determination unit 114 finds, for example, the maximum and minimum pixel values of the output target image data, and if the maximum value is less than or equal to a certain threshold (for example, 100 by 8 bits), the exposure is performed. If it is under and the minimum value is greater than or equal to a certain threshold (for example, 150 in 8 bits), the determination result of overexposure is output to the control unit 109.

  With regard to the contrast, the output image determination unit 114 determines that the contrast is insufficient when the maximum / minimum difference value among the pixel values of the output target image data is equal to or less than a predetermined threshold.

  In the determination of the brightness and contrast, the output image determination unit 114 may perform determination on each color channel, or the average value (R + G + B / 3) or the lightness value (0.299 × R + 0.587 × G + 0). .114 × B: NTSC compliant) may be used.

  With regard to color balance, by comparing the average value and the maximum value / minimum value of each color channel (RGB), it can be understood that excessive bias occurs in a certain channel. Therefore, for example, the output image determination unit 114 calculates the average value (Ra, Ga, and so on) of each color channel of pixel values having values near the maximum lightness value (maximum lightness to maximum lightness-5) in the output target image data Ba) is determined, and the difference between the maximum value and the minimum value of each color channel is a predetermined value or more according to the value [Max (Ra, Ga, Ba) -Min (Ra, Ga, Ba)> 0.1 × Max (Ra)] , Ga, Ba)], it is determined that the color balance is poor.

  The output image determination unit 114 creates an edge intensity image using an edge extraction filter as shown in FIG. The histogram is taken and the standard deviation (square root of the variance) is determined. Then, when the standard deviation value is equal to or less than a predetermined threshold (for example, 5), it is determined that blurring has occurred.

(8-8-4) Notification to User The control unit 109 that has received the determination result from the output image determination unit 114 causes the display unit 105 to display a message prompting re-imaging according to the determination result.

  For example, when the control unit 109 receives a determination result that the tilt angle θ is not within the predetermined range from the output image determination unit 114, the control unit 109 causes the display unit 105 to display a message prompting imaging again without the image pickup object being tilted. Display on.

  When the control unit 109 receives a determination result indicating that the feature amount indicating the degree of geometrical distortion (here, the ratio of the length of the opposite side of the imaging target in the captured image) is not within the predetermined range, the control unit 109 A message is displayed on the display unit 105 prompting imaging again from the normal direction of the plane on which the document image of the object is formed.

(8-9) High Resolution Correction In the above description, the high resolution correction unit 225 performs high resolution correction using one output target image data. However, the high resolution correction unit may perform high resolution correction using a plurality of output target image data. As the method of this high resolution correction, the method described in the Journal of the Institute of Image Information and Television Engineers Vol. 62, No. 3, pp. 337 to 342 (2008) can be used.

  In this case, the output target image determination unit 113 stores, as a data set, a predetermined number of output target image data obtained by continuously imaging a predetermined number of times (for example, 2 to 15) at a designated timing. Store in 108. Usually, images taken continuously are substantially the same image, but are shifted by a very small amount due to camera shake or the like.

  Further, in order to perform high resolution correction using a plurality of output target image data, it is necessary that the predetermined number of image data according to the resolution conversion magnification be shifted by a predetermined amount. Therefore, in the present modification, the output image determination unit 114 of the portable terminal device 100 shown in FIG. 33 is required to execute high resolution correction in the image forming apparatus 200 in the output target image data included in the data set. It is determined whether a predetermined number of sub-captured image data shifted by a predetermined amount is included.

(8-9-1) Determination of Deviation Amount of Multiple Images The deviation required for high resolution correction to improve the legibility of characters is the deviation of less than one pixel (decimal point) of the target image data Point to That is, a shift of a value below the decimal point (less than one pixel), such as 0.3 to 0.7, is important. Deviations of the integer part are not taken into account for high resolution correction. For example, in the case where a shift of less than one pixel such as 1.3 pixels or 2.3 pixels is included, high resolution correction based on a plurality of images can be performed, but one pixel such as one pixel or two pixels If it does not include less than the deviation, high resolution correction can not be performed.

  For example, when the conversion magnification is twice, the number of image data required for high resolution correction is 2, and the amount of deviation between two image data is 0.3 to 0.7 after the decimal point in pixel units. preferable. Therefore, in the storage unit 108, the resolution conversion magnification “2 ×” is associated with the number of times of imaging “2” and the processing execution condition “number of required image data: 2, shift amount: 0.3 to 0.7”. Information is stored in advance.

  When the conversion magnification is four times, the number of image data required for high resolution correction is four, and when one of the image data is used as the reference image data, the reference image data and the remaining three The amount of deviation from the image data is preferably 0.2 to 0.3, 0.4 to 0.6, or 0.7 to 0.8 after the decimal point in pixel units, respectively. Therefore, the storage unit 108 sets the resolution conversion magnification "4 times", the number of times of imaging "4", and the processing execution condition "number of necessary image data: 4, shift amount: 0.2 to 0.3, 0.4 to 0". .6, 0.7 to 0.8 "are stored.

  The control unit 109 prompts the user to select a magnification for resolution conversion, and sets processing execution conditions according to the inputted magnification. Then, the output image determination unit 114 determines whether or not the output target image data that satisfies the set processing execution condition is included in the data set.

  In the following, in order to simplify the description, a case will be described where double magnification is selected as the resolution conversion resolution.

  First, the output image determination unit 114 selects any one of the output target image data, and selects the shift detection partial area for the output target image data (hereinafter, referred to as a first captured image). Here, since the shift detection partial area is used to obtain the shift amount between the first captured image and the remaining output target image data (hereinafter referred to as the second captured image), the shift It is preferable that the change in pixel value is large (a clear pattern exists) in the detection partial region. Therefore, the output image determination unit 114 extracts the displacement detection partial area by the following method.

  The output image determination unit 114 specifies a pixel existing at the barycentric position of the area of the imaging target object, and sets the pixel as a target pixel. Then, an area of n × n pixels including the target pixel is selected. For the selected area, it is determined whether or not the following selection requirements are satisfied, and if it is satisfied, the area is used as a shift detection partial area. On the other hand, if the condition is not satisfied, the area is moved based on a predetermined offset amount, and the same determination is made for the area after movement. Thus, the shift detection partial area is extracted.

Here, as selection requirements, for example, the following two can be mentioned.
One is to use a value according to the dispersion in the region. Assuming that the pixel value is P (i) for the region of n × n pixels in the vicinity of the pixel of interest, the variance value Variance (x) of the partial region is expressed by the following equation (1). The selection requirement is that the value of the variance Variance (x) is equal to or greater than a predetermined threshold. Also, only molecules of this formula may be considered for simplification.

  As another one, an area of n × n pixels in the vicinity of the pixel of interest is subjected to an edge extraction filter such as a first order differential filter shown in FIG. The selection requirement may be that the sum total is equal to or greater than a predetermined threshold (for example, 5% or more of the number of partial area pixels).

  Next, with respect to the shift detection partial image a (n × n) of the first captured image obtained in this manner, a shift detection portion having the substantially same position in the center among the second captured image. Cut out as an image b (m × m) (m> n). In this clipping method, clipping is performed such that the coordinates of the central pixel of the displacement detection partial image a in the first captured image and the coordinates of the central pixel of the displacement detection partial image b in the second captured image coincide with each other.

  After that, in the cut-out partial image b for shift detection, a region that most fits the partial image a for shift detection is determined with sub-pixel accuracy. An example of such a method is normalized correlation pattern matching using the displacement detection partial image a as a template.

  As an example of normalized correlation pattern matching, the correlation is calculated using a known normalized correlation equation. The correlation equation between two patterns Input (I) and Target (T) generally composed of N pixels can be expressed as the following equation (2). Here, α, β and γ can be expressed respectively as follows.

  For example, in the case of n = 5 and m = 7, the correlation equation is calculated for each region (n × n) of the same size as the displacement detection partial image a in the displacement detection partial image b (m × m). As a result, 3 × 3 correlation value Map is generated. Using this correlation value Map, a quadratic surface to be fitted is determined. As a method of finding a quadratic surface, for example, 6 points with high correlation value among 9 points are selected as S (x, y) = a * x * x + b * x * y + c * y * d + e * y + f, Solve the simultaneous equations to find each coefficient. If the value after the decimal point of the coordinate value (both x and y) of the extremum (= maximum value) of this function is within a predetermined range (here, 0.3 to 0.7), the processing execution condition “necessary image It is determined that the number of data: 2, the shift amount: 0.3 to 0.7 is satisfied.

  In addition, what is necessary is just to obtain | require the coordinate of the point which each carries out the partial differentiation of the said quadratic expression, and how to obtain | require the extreme value. At this time, it is more efficient to use the direct correlation values (S1 to S6) because it is not necessary to actually obtain the respective coefficients (a to f). The equation (3) to be obtained is as follows. Here, the origin is the window reference to be focused on.

  Note that although such positional deviation confirmation with sub-pixel accuracy is performed in at least one place, it is desirable to be performed in several places.

  Then, the output image determination unit 114 outputs, to the control unit 109, the determination result as to whether or not the process execution condition is satisfied.

(8-9-2) High Resolution Correction Using Plural Output Target Image Data Next, processing of the high resolution correction unit 225 in the image forming apparatus 200 will be described. The high resolution correction unit 225 performs high resolution correction based on a plurality of output target image data included in the data set received from the mobile terminal device 100.

  Several methods have been introduced in the Journal of the Institute of Image Information and Television Engineers Vol. 62, No. 3, pp. 337 to 342 (2008) as a method of creating a high resolution image from a plurality of image data. In general, high resolution correction is achieved by alignment processing and reconstruction processing of a plurality of images. In this embodiment, the method of normalized correlation pattern matching described in the above (8-9-1) is applied as an example of alignment processing. That is, alignment of a plurality of images can be performed by shifting the above S (x, y) by the shift amount indicating the maximum value.

  Next, the high resolution correction unit 225 performs reconstruction processing. That is, the high resolution correction unit 225 creates reconstructed image data having the number of pixels corresponding to the magnification after resolution conversion. However, the image size is the same as the size of the captured image. Then, the high resolution correction unit 225 determines the pixel value of each pixel of the reconstructed image data as follows. That is, the high resolution correction unit 225 determines, for each pixel (reconstruction pixel) in the reconstruction image data, a pixel (imaging pixel) of a pickup image which is in the vicinity of the reconstruction pixel from among a plurality of pickup images. Interpolation is performed by a general interpolation method (linear interpolation, bicubic interpolation, etc.).

  Specifically, as shown in FIG. 36, the imaging pixel in the vicinity of the reconstruction pixel to be focused on, for example, two points at which the line connecting the lattice of imaging pixels (dotted line in the figure) is closest to the horizontal and vertical directions Select in each of. Here, the two closest points in the horizontal direction are the imaging pixel 1-2 (pixel value: Vi1-2: the same applies hereinafter) and the imaging pixel 1-4 of the first output target image data, It is assumed that the two closest points in the vertical direction are the imaging pixel 2-1 and the imaging pixel 2-2 of the second output target image data. In addition, when selecting an imaging pixel in the vicinity of the reconstruction pixel, it is selected from among a plurality of output target image data on which the correction of the geometric distortion and the correction of the lens distortion are performed. Thereby, high resolution correction can be performed in a state in which geometric distortion and lens distortion are corrected.

  In addition, when calculating the value of the coordinate after correction | amendment, you may obtain | require based on the value of the coordinate which considered correction | amendment of applicable geometric correction and lens distortion of several captured images used as a base. That is, only the correction value of the geometric correction and the lens distortion may be calculated, and after performing the reconstruction processing, coordinate conversion may be performed using the correction value.

  Then, an intersection point of a straight line that is perpendicular to the line connecting the two points selected in each of the horizontal direction and the vertical direction and that includes the point of the reconstructed pixel of interest and the line is obtained. As shown in FIG. 36, when the intersection is internally divided at t: 1-t and u: 1-u for each of two line segments, the high resolution correction unit determines the pixel of the focused reconstruction pixel The value Vs may be obtained based on the following formula 4. In this case, linear interpolation is performed. Then, it is possible to obtain pixel values in the same manner for all reconstructed pixels, and to generate reconstructed image data with high resolution as high-resolution image data.

  Note that another method may be used as the interpolation method. Also, another method introduced in the Journal of the Institute of Image Information and Television Engineers Vol. 62, No. 3, pp. 337 to 342 (2008) may be used. For example, a method such as MAP (Maximum A Posteriori) method may be used which first obtains the evaluation function corresponding to the posterior probability based on the estimation by minimizing.

(8-9-3) Imaging Unit In the above description, it is assumed that a shift occurs in a plurality of captured images due to camera shake when the imaging unit 101 continuously performs imaging a plurality of times. However, the present invention is not limited to this, and the imaging unit 101 may slightly shift the imaging device (CCD / CMOS) or the lens when performing imaging a plurality of times in succession. As a result, a shift is reliably generated between a plurality of captured images.

(8-10) Transmission Timing of Image Data from Mobile Terminal Device to Image Forming Device In the above description, the output target image data stored in the document imaging mode is stored in the mobile terminal device 100, and the transmission instruction is At the input timing, the output target image data accumulated so far is collectively transmitted. However, the timing at which the output target image data is transmitted from the portable terminal device 100 to the image forming device 200 is not limited to this.

  For example, the output target image data may be transmitted to the image forming apparatus 200 each time the output target image data is stored in the storage unit 108 in the document imaging mode. In this case, in most cases, the user is not near the image forming apparatus 200. Therefore, the communication unit 104 of the mobile terminal device 100 may transmit the output target image data to the image forming apparatus 200 via the mobile phone network and the Internet network.

(8-11) Output Processing Information In the above description, it is assumed that the portable terminal device 100 acquires the output processing information and transmits the output processing information to the image forming device 200. However, the present invention is not limited to this. Even when the image forming apparatus 200 acquires user information for user authentication, even if output processing information (information indicating the type of output processing and setting conditions for output processing) is acquired Good.

(8-12) Output Process In the image forming apparatus 200, the control unit 212 converts the high resolution image data generated by the image processing unit 202 into a high compression PDF before performing the filing process or the mail transmission process. It is also good. The high compression PDF data is PDF data in which the background portion and the character portion in the image data are separated, and the compression processing is optimally performed on each portion. As a result, character legibility is good and the image file size can be reduced.

  In addition, the control unit 212 may perform OCR processing on the high resolution image data generated by the image processing unit 202 to generate text data, before performing the filing processing or the mail transmission processing. Then, the control unit 212 may convert the high resolution image data into a PDF, and add the generated text data as a transparent text. Transparent text is data for superimposing (or embedding) recognized characters as text information on image data in a form that can not be seen in appearance. For example, in PDF files, image files in which transparent text is added to image data are generally used. Then, the control unit 212 may output the generated transparent text-added PDF data. This makes it possible to output an electronic document that is easy to use like a text searchable file.

(8-13) Regarding Image Processing Unit Included in Image Forming Apparatus In the above description, it has been described that the image processing unit 202 included in the image forming apparatus 200 performs high resolution correction and the like. However, the image forming apparatus 200 performs high resolution correction and other image processing (higher resolution output data set, etc.) on the high resolution output data set (correction of geometrical distortion, correction of lens distortion, contrast correction, color balance correction, etc.) It may be executed by a server provided with 202. In this case, it can be said that the server is an image forming apparatus that performs high resolution correction on the high resolution output data set received from the mobile terminal device 100 and outputs the corrected high resolution image data.

(9) Program and Recording Medium The present invention transmits an image captured by the portable terminal device 100 described above to the image forming apparatus 200 on a computer readable recording medium storing a program to be executed by a computer. It is also possible to record the method of more output.

As a result, it is possible to provide a recording medium in which the program code (the executable program, the intermediate code program, the source program) for performing the above processing is recorded in a portable manner.
In this embodiment, as the recording medium, a memory (not shown) such as a ROM, which is not shown because the processing is performed by a microcomputer, may be a program medium itself, or a figure. Although not shown, a program reader may be provided as an external storage device, and it may be a program medium readable by inserting a recording medium therein.

In any case, the stored program may be configured to be accessed and executed by the microprocessor, or in any case, the program code may be read and the read program code may be read from the microcomputer. It may be downloaded to a program storage area not shown, and the program may be executed. The download program is stored in advance in the main device.
Here, the program medium is a non-transitory tangible medium configured to be separable from the main body, and is a tape system such as a magnetic tape or a cassette tape, or a magnetic disk such as a flexible disk or a hard disk. Disc system of optical disc such as CD-ROM / MO / MD / DVD, card system such as IC card (including memory card) / optical card, or mask ROM, erasable programmable read only memory (EPROM), EEPROM (registered trademark) (Electrically Erasable Programmable Read Only Memory), a medium that fixedly carries a program code including a semiconductor memory such as a flash ROM.

  Further, in the present embodiment, since it is a system configuration capable of connecting a communication network including the Internet, it may be a medium that carries program code in a fluid manner so as to download program code from the communication network. When the program is downloaded from the communication network as described above, the program for download may be stored in advance in the main device or may be installed from another recording medium. The present invention can also be realized in the form of a computer data signal embedded in a carrier wave, in which the program code is embodied by electronic transmission.

  The above-described image processing method is executed when the recording medium is read by a program reading device provided in the mobile terminal device 100 or the image forming device 200.

  The present invention is not limited to the embodiments described above, and various modifications are possible within the scope of the claims. That is, an embodiment obtained by combining technical means appropriately modified within the scope of the claims is also included in the technical scope of the present invention.

  The present invention can be applied to a captured image processing system that performs data communication between a portable terminal device and an image forming device or an image display device.

10 Shutter Button 100 Mobile Terminal Device (Imaging Device)
101 Imaging unit 105 Display unit 108 Storage unit 109 Control unit 110 Imaging range determination unit 111 Geometric arrangement detection unit (detection unit)
112 display processing unit 113 output target image determination unit 114 switching unit 200 image forming apparatus (image output apparatus)
300 image display device

Claims (12)

In an image processing apparatus including display means for displaying a captured image obtained by capturing an object.
The above object is rectangular,
The captured image includes a first captured image displayed on the display unit before the output target image is determined, and a second captured image determined as the output target image.
Before the output target image is determined, the display means displays the first correction processed image obtained by geometrically correcting the range of the target object of the first captured image into a predetermined rectangular shape, and the output target image And a display processing unit for displaying on the display means an image after second correction processing in which a range of the object of the second captured image is geometrically corrected to a predetermined rectangular shape. Image processing device.   The display processing unit displays the first captured image and the image after the first correction processing in the same screen, and displays the second captured image and the image after the second correction processing in the same screen. The image processing apparatus according to claim 1. The image pickup apparatus further includes a detection unit that detects an edge pixel group connected in a line segment shape in the captured image,
The display processing unit is surrounded by a straight line corresponding to each of four line segments indicated by the four edge pixel groups detected by the detection unit, on the edge pixel group detected by the detection unit. The image processing apparatus according to claim 1, wherein a quadrangular outline indicating the outline of the object is displayed superimposed on the captured image.   The display processing unit receives an instruction to edit the coordinates of the vertex of the outline, and the first corrected image and the first correction process based on an image of a region surrounded by the outline connecting the coordinates of the edited vertex. The image processing apparatus according to claim 3, wherein an image after 2 correction processing is generated.   5. The image processing apparatus according to claim 3, further comprising: a switching unit that switches whether the display processing unit makes the display function of the contour line valid or invalid.   The display processing unit is characterized by causing the display unit to display the image after the first correction processing and the image after the second correction processing when the switching unit sets the display function of the contour line to be effective. The image processing apparatus according to claim 5, wherein And a display unit configured to display a captured image obtained by capturing an object, wherein the target is rectangular, and the captured image is a first captured image displayed on the display unit before the output target image is determined; An image processing method of an image processing apparatus including a second captured image determined as the output target image, the image processing method comprising:
(A) Before the output target image is determined, a first corrected image is displayed on the display means by geometrically correcting the range of the object of the first captured image into a predetermined rectangular shape, An image processing method comprising the step of displaying on the display means an image after second correction processing in which the range of the object of the second captured image is geometrically corrected to a predetermined rectangular shape after the output target image is determined .   In the step (a), the first captured image and the image after the first correction processing are displayed in the same screen, and the second captured image and the image after the second correction processing are displayed in the same screen. The image processing method according to claim 7. The method further includes the step (b) of detecting a group of edge pixels connected in a line segment in the captured image.
In the step (a), on the edge pixel group detected in the step (b), corresponding to each of the four line segments indicated by the four edge pixel groups detected in the step (b) The image processing method according to claim 7, wherein a quadrilateral outline indicating the outline of the object surrounded by a straight line is displayed superimposed on the captured image.   In the step (a), an instruction to edit the coordinates of the vertex of the outline is received, and the image after the first correction processing and the image based on the image of the region surrounded by the outline connecting the coordinates of the edited vertex The image processing method according to claim 9, wherein an image after second correction processing is generated.   The image processing method according to claim 9, further comprising: (c) switching whether to enable or disable the display function of the outline in the step (a).   In the step (a), when the display function of the outline is set to be effective in the step (c), the image after the first correction processing and the image after the second correction processing are displayed on the display means The image processing method according to claim 11.