JP2014239367A - Image processor, image processing method, and program - Google Patents

Abstract

PROBLEM TO BE SOLVED: To solve the problem that it is difficult to detect an original sheet area after the second page and automatically decide the order of pages when stapled double-side original sheets are photographed with a camera, computerized, and printed.SOLUTION: An image processor comprises: staple position decision means for deciding the position of a staple in image data obtained by photographing a plurality of original sheets stapled; original sheet area detection means for detecting an original sheet area for at least one page in the image data; and control means for, when the original sheet areas for a plurality of pages are detected by the original sheet area detection means, determining the order of pages for the original sheet areas for the plurality of pages on the basis of the staple position decided by the staple position determination means.

Description

  The present invention relates to a portable terminal having an imaging function, an image processing apparatus, and a method.

  Due to the advancement of digital camera technology, it has become common to take a document and obtain a document image using a portable terminal such as a mobile phone. In recent years, portable terminals can use the Internet environment. As a result, a photographed original image can be transmitted to a printing apparatus via a network and printed by a simple operation. In other words, if you have a portable terminal and a printing environment, you can use it as a simple digital multifunction device. However, when an original image taken with a mobile terminal is printed as it is, there are various problems such as inclination and distortion of the original image that occurs during shooting. Therefore, it is general to use after performing various image processing on the document image.

  For example, in Patent Document 1, by providing a mark at the edge of an original, distortion and inclination of the original image at the time of shooting are specified and corrected by coordinate conversion. Further, in Patent Document 2, the document area is estimated from the edge information of the rectangle of the document. Then, distortion and inclination of the document area are detected, and distortion of the document image generated at the time of shooting is corrected by coordinate conversion. These are generally called scanner apps. As described above, these techniques are techniques for specifying a document area by detecting a document edge or a document mark in a document image and enabling automatic camera photographing.

JP2012-68746 JP2011-55467A

  However, the techniques using Patent Document 1 and Patent Document 2 still have a problem that an opened page cannot be efficiently read when double-sided originals bound with staples are continuously read. For example, FIG. 8A shows the first page of a document with the upper left bound with staples. FIG. 8B shows a state in which the second and third pages of the original in FIG. 8A are opened. In this case, the first page is one original image, but since the back of the first page is the second page after the second page, two original images can be photographed simultaneously.

  At this time, when document area detection and automatic shooting are performed using the conventional technique, only one area vertex exists in the document image in FIG. 8A, but in FIG. 8B, two area vertices exist. Exists in the original image. Further, in FIG. 8B, since the document area for two pages exists in one document image, image division and sorting are required. However, in the case of a document bound on the right side as shown in FIG. 8D, the page opens on the right side as shown in FIG. 8E. Therefore, the photographer needs to shoot while moving the mobile terminal so that each page is properly shot in consideration of the page order, the position of the target page, and the like.

  In order to solve the above-described problems, an image processing apparatus according to the present invention includes a staple position determining unit that determines the staple position in image data obtained by photographing a plurality of originals, and at least one page of the image data. A document area detecting unit for detecting a plurality of pages, and when the document area detecting unit detects a document area for a plurality of pages, the plurality of pages for the plurality of pages are determined based on the staple position determined by the staple position determining unit. Control means for determining the page order of each page in the document area is provided.

  According to the present invention, in a copying function using a portable terminal, a stapled double-sided document can be automatically photographed and printed in an appropriate page order.

It is a block diagram for demonstrating the whole structure of a system. It is an external view of a portable terminal. It is a module block diagram of a portable terminal. It is a module block diagram of a server. It is a module block diagram of a printer. It is a flowchart for demonstrating the whole operation | movement. 7 is a flowchart for explaining an operation during a photographing operation (S603) for explaining the first embodiment. FIG. 5 is an explanatory diagram of a document stapled at one place. 10 is a flowchart for explaining an operation (S706) of determining a document area. FIG. 5 is an image diagram for explaining an operation of determining a document area stapled at one place. It is explanatory drawing of the document stapled in two places. FIG. 6 is an image diagram for explaining an operation of determining a document area stapled at two places. 10 is a flowchart for explaining an operation during a photographing operation (S603) for explaining the second embodiment. FIG. 6 is an image diagram for explaining an operation of determining a document area when it is not stapled. 10 is a flowchart illustrating an operation (S701) for determining a document area. FIG. 6 is an image diagram for explaining an operation of determining a document area of the first page.

  The best mode for carrying out the present invention will be described below with reference to the drawings.

  FIG. 1 shows the overall configuration of a system used in this embodiment. A printer 103 and a wireless router 102 are connected to the LAN 110. The LAN 110 is connected to the Internet 120 and is also connected to the server 121 that provides the service via the Internet 120. The portable terminal 101 can be connected to the printer 103 and the server 121 via the wireless router 102. Here, the wireless router 102 and the printer 103 are connected to the LAN 110, but this is not a limitation. The wireless router 102 and the printer 103 may be connected to the server 121.

<Configuration of mobile terminal>
FIG. 2 shows the appearance of the mobile terminal 101 used in this embodiment. FIG. 2A shows the surface of the mobile terminal 101. A touch panel 201 and operation buttons 202 are provided on the surface. FIG. 2B is the back surface of the mobile terminal 101. An imaging unit 203 is disposed on the back surface. There are various types of portable terminals, but the present invention can be used as long as the terminal device has a camera function. That is, a smart phone or a mobile phone having a camera function or a digital camera having a communication function may be used. The camera function is equipped with an autofocus device (not shown) that automatically detects the in-focus state based on the contrast ratio and infrared transmission / reception timing. can do. The autofocus device used in this embodiment is an example, and there is no limitation as long as it can measure the focal length and the subject distance.

  FIG. 3 shows an internal configuration of the mobile terminal 101. However, this configuration is limited to the configuration necessary for this embodiment. In FIG. 3, a CPU 301 (processor), a RAM 302, and a ROM 303 send and receive programs and data via a data bus 311. To the data bus 311, a touch panel 201, an imaging unit 203, an image processing device 310, and a motion sensor 309 are connected. In addition, a wireless LAN device 307 is connected to the data bus 311 and data transmission / reception with the wireless router 102 is possible. Further, a storage unit 305 is connected to the data bus 311 and can store image data and various programs. The motion sensor 309 includes a three-axis acceleration sensor, an electronic compass, and a three-axis angular velocity sensor, and can detect the posture and movement of the mobile terminal 101 by using a known technique. Note that the CPU 301 controls the components in the portable terminal 101 based on a program stored in the ROM 303.

  The image processing apparatus 310 includes an edge detector 3101 that detects an edge of a document, a staple detector 3102 that searches for a staple pattern, a document region detector 3103 that detects a rectangular region of the document, and a feature point detection that performs feature point extraction. A container 3104 is provided. These detectors are connected to the data bus 311 of the portable terminal 101 via a bus in the image processing apparatus 310.

<Server configuration>
FIG. 4 is a configuration example of the server 121. The CPU 401 reads a control program stored in the ROM 403 and executes various control processes. The RAM 402 is used as a temporary storage area such as a main memory or work area for the CPU 401. A network I / F 404 connects the server 121 to the Internet 120 and transmits / receives various information to / from other devices. The HDD 405 stores image data and various programs.

  In FIG. 4, image data from the mobile terminal 101 received via the network I / F unit 404 is transmitted and received between the CPU 401, RAM 402, and ROM 403 via the data bus 410. When the CPU 401 executes the image processing program stored in the ROM 403 or the HDD 405, image processing for the image data is realized. Further, the image processed image data is converted into PDL (Page Description Language) for printing and converted into data that can be printed by a normal printer.

<Printer configuration>
FIG. 5 is a diagram illustrating the configuration of the printer 103 used in this embodiment. The printer 103 includes a controller unit 500, an operation unit 509, an external memory 510, and a printer unit 511. In the controller unit 500, the printer CPU 505 outputs an image signal to the printer unit 511 via the printer I / F unit 504 connected to the system bus 508 based on a control program or the like. The control program is stored in the ROM 507 and the external memory 510. The ROM 507 stores a control program for the CPU 505 and the like. The CPU 505 can perform communication processing with the mobile terminal 101 and the server 121 via the network I / F unit 501. Thus, information in the printer, shooting information on the mobile terminal, image processing information on the server 121, and the like can be transmitted to and received from the mobile terminal 101 and the server 121. A RAM 506 functions as a main memory, work area, and the like for the CPU 505.

  The input image data is sent to the controller unit 500 via the network I / F unit 501. Subsequently, the PDL (page description language) data processing unit 502 performs PDL interpretation and rendering, the image processing unit 503 performs image processing for print output, and the printer unit 511 prints the processed image data.

<Detailed description of processing flow>
FIG. 6 is a flowchart showing an operation of printing by scaling the document image area in the captured image by the portable terminal 101, the server 121, and the printer 103 to the size of the output document in the present embodiment. The processing at the portable terminal 101 is controlled by the CPU 301, the processing at the server 121 is controlled by the CPU 401, and the processing flow at the printer 103 is controlled by the CPU 505.

  In step S <b> 601, when an instruction to start shooting is input from the user, the mobile terminal 101 searches the printer in the LAN 110 via the wireless LAN device 307 and the wireless router 102 for a printer to which the captured image data is printed. To do. For example, when a user inputs an IP address of a desired printer, a method of searching for a printer corresponding to the IP address can be considered.

  In step S <b> 602, when the target printer 103 is found, the printer 103 notifies the portable terminal 101 of print setting information that can be set during printing via the LAN 110 and the wireless router 102. The print setting information here is information such as usable paper size, paper type, output resolution, double-sided printing, Nin1 printing, color printing, or stapling. Using the print setting information, the mobile terminal 101 determines which print setting is used when printing a captured image.

  In step S <b> 603, photographed image data is obtained by photographing a document using the photographing unit 203. The document area in the captured image data is acquired by extracting the vertex of the origin. The photographed image data also includes an area where a document other than the document (for example, a desk on which the document is placed) is copied, and the document area is an area where the document is copied in the photographed image data. Further, in the case of a stapled document, the stapling position is determined. At this time, for the stapled document, the vertexes of the document are extracted while continuously photographing the document that has been paginated by the user. S603 will be described in detail later with reference to FIG.

  In step S604, an optimum output paper size or the like that is set in advance by the user with the portable terminal 101 or calculated from information such as a focal length at the time of shooting is calculated as an image processing parameter. In step S605, the captured image data is compressed in the JPEG format or the like. In step S <b> 606, print settings and image processing parameters are transmitted to the server 121 via the wireless router 102, the LAN 110, and the Internet 120 using the wireless LAN device 307 together with the compressed captured image data.

  In step S607, the server 121 decompresses the compressed captured image data. In step S608, image data shaping processing is performed. The shaping process includes image trimming, scaling, rotation, and the like. In the trimming, when there are vertices including document areas for a plurality of pages in image data, the document areas for a plurality of pages included in the image data are divided into areas corresponding to the respective pages of the plurality of pages. . Then, an area corresponding to each page of the plurality of pages is generated as image data of one page of document, and the image data of the document of each divided page is arranged in the order of the pages of the previous page and the subsequent page. The trimming method will be described in detail later. Further, other known scaling techniques such as affine transformation, cubic scaling and linear scaling are used for each image data. Thereby, image data for only the document area is generated. The plurality of image data is sequentially subjected to the processing from S609 onward as the previous page and the subsequent page based on the staple information. Here, while the processing of the previous page is being performed, the subsequent page is stored in the RAM 402 or the HDD 405 and waits for processing.

  In step S609, density unevenness correction of captured image data is performed on each page, and color unevenness due to light sources and shadows during document shooting is corrected. First, filter processing is performed so as to remove noise at the time of shooting, and then tone correction is performed so that white on paper can be reproduced by skipping colors on the background. In step S610, image area determination is performed, and a character or line portion in the captured image data is divided into a portion that is not, and an attribute flag is added to each pixel. In step S611, a filter process is performed on the pixels determined to have the character or line attribute, and the character or line is corrected to be sharp.

  In step S612, a character monochromatic process is performed to reproduce a pixel determined to be a character in a single color, thereby sharpening the character. For example, the RGB values of pixels that reproduce a character are averaged, and the average value is applied as the character color. In step S613, the captured image data is converted from a camera-specific color space to a common RGB color space. Here, the color space conversion is performed from the camera color space of the portable terminal 101 to a colorimetric common RGB color space such as sRGB by a predefined 3 × 3 matrix operation. In S614, the captured image data that has been corrected so far is compressed in the JPEG format or the like.

  In step S615, corrected captured image data and print settings are acquired from the server 121 via the Internet 120 and the LAN 110. Thereafter, the printer 103 performs processing until the photographed image data is printed. The processing of S615 to S619 is performed by the image processing unit 503.

  In step S616, the compressed captured image data is decompressed. In S617, conversion from a common RGB color space to a printer color space composed of CMYK color components suitable for the printer is performed by interpolation using a color conversion LUT (Look Up Table). The color conversion LUT here is a three-dimensional LUT obtained by dividing each of the three RGB components at an appropriate grid point interval, and each LUT entry has an 8-bit bit precision corresponding to the grid point of the LUT. The CMYK value of The three-dimensional LUT is converted into image data composed of CMYK values by performing known interpolation calculation.

  In step S618, tone correction is performed using a one-dimensional LUT to correct the density characteristics so as to match the tone target. When the gradation target is changed between the character part and the other part, the gradation target is changed here to perform correction. In S619, the tone-corrected image data is converted into image data having a pseudo halftone expression of 1 bit for each color of CMYK using a dither matrix. Here, in the pseudo halftone processing, the numerical value on the dither matrix stored in the RAM 506 is compared with the input image data, and if the numerical value of the input image data is large, 1 is set, and if the numerical value of the input image data is small, 0 is set. Perform output processing. In this embodiment, screen processing with 1-bit output is performed for ease of explanation, but the number of output bits is not limited to 1 bit. In addition, a plurality of dither matrices can be switched and used in accordance with an instruction from the CPU 505.

  In step S620, the processed photographed image data is printed on all pages. At this time, if the printer has a stapling function, stapling is performed at the same position as the original. If there are a plurality of documents, the processing from S603 to S606 may be performed in S603, and then S607, which is processing in the server 121, may be performed. At this time, the compressed image, the calculated parameters, and the like are temporarily stored in the RAM 302, and the process proceeds to S607. On the other hand, when processing the captured images of S603 one by one, the processes from S603 to S620 are performed one by one. In step S621, a print end notification is sent to the mobile terminal 101 in order to notify the user that all pages have been printed.

  Finally, in step S622, a print end notification is displayed on the touch panel 201 of the portable terminal 101, and the process ends.

<Detailed Description of Document Shooting (S603)>
FIG. 7 shows a detailed flow of S603 for photographing a document. First, in step S <b> 701, the first page rectangular area of the document is detected using the document area detector 3103. First, the image data being acquired by the photographing unit 203 is stored in the RAM 302. Since the image data is sequentially transferred from the photographing unit 203 and displayed on the touch panel 201, the user can easily adjust the position. At this time, the rectangular area of the first page is detected using this image data. The image data transferred from the photographing unit 203 is about 30 images per second. However, it is not necessary to process all incoming image data. In the rectangular area detection of S701, the rectangular area of the first page can be sufficiently detected if about 2 to 3 sheets can be processed. The process of S701 will be described later in detail.

  In step S702, it is determined whether a rectangular area is detected in step S701. If a rectangular area is detected, shooting is performed using the shooting unit 203 in S703. This captured image is acquired at a higher resolution than the image used in S701. In S704, the four feature point coordinates of the rectangular area acquired in S701 are acquired as vertices. At this time, if the resolution of the image data in S701 is different from the resolution of the image data captured in S703, the vertex coordinates are appropriately converted. Note that the vertex position at this time is used for trimming performed in the image shaping process of S608.

  In step S705, the staple position is determined. As a method for determining the staple position, there are an automatic detection method and a method of receiving an input from the terminal screen 201.

  First, an automatic detection method will be described. In a general printer, there are five stapling positions: one upper left corner, one upper right corner, two left sides, two right sides, and two upper sides. In addition, there are two types: portrait of portrait paper and landscape. FIGS. 8A, 8D, 8G, 11J, and 11A, 11D, 11G, and 11J are examples of combinations of staple positions and numbers. With respect to the image data acquired by the imaging unit 203, binary edge image data is created by the edge detector 3101 using a spatial filter, and the staple detector 3102 is used to create the binary edge image data in the staple region shown in FIGS. The same pattern as the staple is detected. Since various methods for detecting a specific object from image data have been proposed, this embodiment is not limited to this. For example, by detecting a specific object from image data, in the case of FIG. 8A, staples are detected when one rectangular area is determined at the upper left of the original area. Then, from the detected position and number, it is determined that the upper left is stapled.

  As a method of accepting an input from the terminal screen 201, image data acquired by the photographing unit 203 is displayed on the terminal screen 201, and a position clicked by the user is determined as a staple position. At this time, if an area other than the staple position shown in FIG. 8 is clicked, the input is invalid. At this time, it is determined that the document is not stapled.

  In S706, the document area detection for the second and subsequent pages is performed. At this time, the mobile terminal may determine whether or not the document has been paginated by the user before proceeding to S706. Various methods can be considered as the determination method. For example, it is conceivable to determine whether or not the image has changed after a certain time from the image taken in S703. For the second and subsequent pages, two areas are detected from the document area in accordance with the staple position specified in S705. FIGS. 8 (a) and 8 (d) are illustrations of how to open a document that has been printed in portrait on both sides when stapled at one upper left corner or one upper right corner. The process of S706 will be described in detail later using the flowchart of FIG.

  In step S707, it is determined whether an area has been detected in step S706. If the area is detected, the image is captured using the imaging unit 203 in S708. In S709, the coordinates of the 4 or 8 vertexes of the closed region acquired in S706 are acquired. At this time, if the resolution of the image data in S706 is different from that of the image data captured in S708, the coordinates of the vertices are appropriately converted. The acquired vertices are used in the image shaping process in S608.

  In S710, it is determined whether it is the last page. This determination may be manual or automatic. In the case of manual determination, in step S710, a shooting end button is displayed on the terminal screen 201, and determination is made based on input from the user. If the end button is touched, this sequence ends. If not touched for a certain period of time, the process returns to S706.

  On the other hand, in the case of automatic determination, determination is made based on image data captured by the image capturing unit 203. For example, in the case of FIG. 8A in which the upper left is bound in a portrait, the vertices are detected with the spread as shown in FIG. 8B after the second page. Then, the last page is not spread apart, and the side is in the horizontal direction as shown in FIG. When the positions of these four vertices are detected, it can be determined that the page is the last page. The determination of the last page is also determined by the orientation of the original and the stapling position. FIG. 8F shows the final page when the upper right one is bound in the portrait. FIG. 8 (i) shows the final page of a document in which a landscape document is bound at one upper left position. FIG. 8 (l) is the final page of the original that is bound to the top right of the landscape original. FIG. 11C shows the final page of the original that is bound at the top two positions of the landscape original. FIG. 11F shows the final page of the original that is bound at the top two positions of the portrait original. FIG. 11 (i) shows the final page of the original document in which the right two parts of the portrait original are bound. FIG. 11 (l) shows the final page of the original document in which the left two positions of the portrait original are bound. The final page determination can be performed by changing the number of vertices of these documents.

<Detailed Description of First Page Area Detection (S701)>
The page area detection of the first page in S701 will be described with reference to the flowchart of FIG. First, in step S1501, a binary edge image is created from image data using the edge detector 3101 in the image processing apparatus 310. The edge image is obtained by extracting only the high frequency region of the brightness component of the image data using a band pass filter or the like and binarizing it with a predetermined binarization threshold. The shape of the band-pass filter and the binarization threshold value are not specific to the present embodiment, and will not be described in detail.

  In step S1502, feature points are extracted using the feature point detector 3104. Various methods for extracting feature points have been devised, and research is still ongoing. For example, the classical method includes an extraction method by pattern matching. However, in the present embodiment, since this is not specified, detailed description is omitted. FIG. 16 is an example of an image when feature points are extracted from a binary edge image. In addition to the feature points of the document area, those outside the document area (for example, those on the desk) are also photographed, and these corners are also detected as feature points. Furthermore, the corners of characters and figures in the document area are also detected as feature points.

  In S1503, a closed region is extracted. When a line segment of an edge image extending from a feature point that becomes a start point (for example, vertices 001 to 004 in FIG. 16) in a certain area returns to the start point, this area is determined as a closed area.

  In step S1504, the document area detector 3103 is used to select a document closed area from the several closed areas extracted in step S1503. At this time, a closed region that is most likely to be a document is selected. The closed area that seems to be a document is a closed area that is appropriate for the screen size and has little distortion at the relative positions of the vertices. The distortion of the relative positions of the vertices is, for example, that all the vertices are close to a right angle (within 90 degrees ± error), and the error is appropriately set. At this time, if an appropriate closed region is not found, it is determined that no page region is detected.

<Detailed Description of Second Page Area Detection (S706)>
The page area detection for the second and subsequent pages in S706 will be described with reference to the flowchart of FIG. First, in step S <b> 901, a binary edge image is created from image data using the edge detector 3101 in the image processing apparatus 310. The edge image is obtained by extracting only the high frequency region of the brightness component of the image data using a band pass filter or the like and binarizing it with a predetermined binarization threshold. The shape of the band-pass filter and the binarization threshold value are not specific to the present embodiment, and will not be described in detail.

  In step S902, feature points are extracted using the feature point detector 3104. Various methods for extracting feature points have been devised, and research is still ongoing. For example, the classical method includes an extraction method by pattern matching. However, in the present embodiment, since this is not specified, detailed description is omitted. FIG. 10 shows an image when feature points are extracted from a binary edge image. In addition to the feature points of the document area, those outside the document area (for example, those on the desk) are also photographed, and these corners are also detected as feature points. Furthermore, the corners of characters and figures in the document area are also detected as feature points.

  In S903, a closed region is extracted. When a line segment of an edge image extending from a feature point that becomes a start point (for example, vertex 001 in FIG. 10) in a certain region returns to the start point, this region is determined as a closed region.

  In step S904, the document area detector 3103 is used to select a document closed area from the several closed areas extracted in step S903. The closed region at this time can be determined by determining a closed region having eight feature points. These eight feature points are the vertices of the document. In FIG. 10, the closed area surrounded by the vertices 001 to 008 can be specified as the closed area of the document.

  The above description can be used when there is only one staple in S904. For example, FIG. 8 shows the case of one staple in the upper right corner and the upper left corner. When the portrait is stapled in the upper left corner as shown in FIG. 8A, the second and subsequent pages are spread as shown in FIG. 8B. When the portrait is stapled in the upper right corner as shown in FIG. 8D, the second and subsequent pages are spread as shown in FIG. When stapling in the upper left corner of the landscape as shown in FIG. 8 (g), the second and subsequent pages are spread as shown in FIG. 8 (h). Further, when the landscape is stapled in the upper right corner as shown in FIG. 8 (j), the second and subsequent pages are spread as shown in FIG. 8 (k).

  On the other hand, when the number of staples is two, the form of the spread is changed. FIG. 11 shows an example of a spread when two staples are provided. FIG. 11A is a diagram in which the top two places of the landscape are stapled. If this is opened, it will become like FIG.11 (b). FIG. 11D shows a document in which the top two portions of the portrait are stapled, and the second and subsequent pages are as shown in FIG. FIG. 11G shows a document in which the portrait is stapled at two places on the right side, and the second and subsequent pages are as shown in FIG. FIG. 11 (j) shows a document in which the portrait is stapled at two places on the left, and the second and subsequent pages are as shown in FIG. 11 (k). When these two places are stapled, there are four vertices vertices 001 to 004 as shown in FIG. Therefore, since the spread method changes depending on the number of staples, it can be seen that the number and shape of the vertices of the closed region to be detected change.

  Further, when the sheets are not stapled, the spread method is changed. FIG. 14 shows image data when a closed region is detected when the page is spread when not stapled. Since the number of closed areas surrounded by the vertices 001 to 004 and the vertices 005 to 008 is two, it is determined that these two closed areas are selected and detected. At this time, if an appropriate closed region is not found, it is determined that a page region is not detected.

<Trimming Image Data in Image Shaping Process (S608)>
The trimming of image data varies depending on the number of vertices. The number of vertices varies depending on whether or not it is the first page, and is determined in the staple position determination process in S705. The position of the vertex is determined in S704 and S709 and transferred to the server 121.

  When the number of vertices is 8, a line segment connecting the vertices 003 and 007 in FIG. In this case, the image data is trimmed by a line connecting the vertices 003, 004, 005, 006, 007, 003, and this is used as the previous page. Further, the image data is trimmed by a line connecting the vertices 003, 002, 001, 008, 007, and 003, and this is used as a subsequent page. Furthermore, since these images are pentagonal, the intersection of the line extended in the direction from the vertex 004 to 003 and the line extended in the direction from the vertex 006 to 007 is corrected to a rectangle as a new vertex. The rear page can be similarly modified to a rectangle.

  On the other hand, when the number of vertices is four on the second and subsequent pages, the result is as shown in FIG. In this case, the division position is a line segment connecting the intermediate point 1 between the vertices 001 and 004 and the intermediate point 2 between the vertices 002 and 003. A line connecting the vertices 001 and 002 and the intermediate points 1 and 2 is trimmed from the image data to obtain the previous page. In addition, trimming is performed from the image data with a line connecting the vertices 004 and 003 and the intermediate points 2 and 1 to obtain a subsequent page.

  As described above, when the image processing apparatus according to the present embodiment is used, when a stapled double-sided document is scanned with a portable terminal, it is possible to simultaneously capture a spread and further print in page order.

  In the second embodiment, an operation when the staple position of the document cannot be determined in S705 of the first embodiment will be described. In the first embodiment, it has been described that the determination of the staple position in S705 can be automatic or manual. In the case of manual operation, since input from the user is necessary, there is a problem that convenience is impaired. In the case of automatic, it may be impossible to detect depending on the type of staple. Therefore, in the second embodiment, the staple position determination is determined after acquiring image data for the second and subsequent pages. In the following description, only differences from the first embodiment are described.

  FIG. 13 shows a detailed flowchart for S603 in FIG. 6 in the second embodiment. Note that the processing of S1301 to S1304 and S1306 to S1308 in FIG. 13 is the same as S701 to S704 and S707 to S709, and thus the description thereof is omitted.

  In step S1305, the page area is detected. In this embodiment, since the staple position is not fixed, the number of vertices to be detected is unknown. In step S1305, it is necessary to search both the 4-point closed region and the 8-point closed region in the image data. In this case, since the document area is the maximum size of the closed area in the image data, the closed area whose area is equal to or larger than the threshold value is the document area. After steps S1306 to S1308, it is determined in step S1309 whether the staple position has been confirmed. When the shooting pages are 2 pages and 3 pages, the stapling position has not been determined, so the processing of S1310 is performed.

  In step S1310, the staple position is determined from the image data captured in step S1307. For example, in the case of a portrait document in which the upper left side is stapled as shown in FIG. 8A, if the left page is turned sideways as seen in FIG. 8B, it is stapled at one upper left position. I understand. On the other hand, in the case of a portrait document with the upper right stapled as shown in FIG. 8D, if the left page is turned sideways as seen in FIG. 8E, it is stapled at one upper right position. I understand. These differences can be achieved by using template matching based on the positional relationship between the vertices of image data obtained by photographing pages 2 and 3. In template matching, the relative positions of the vertices in FIGS. 8B, 8E, 11H, 11K, 11B, 11E, 11H, and 11K are stored in the ROM 303, and in S1308 This is a technique for scoring and determining the distance to each acquired vertex using statistical means. However, in FIG. 8, the number of vertices is eight, and in the case of FIG. 11, the number of vertices is four. Since the number of vertices has already been determined in S1305, template matching need only be performed in either FIG. 8 with one staple or FIG. 9 with two staples. Various researches have been made on detailed means of template matching, and the present embodiment is not limited to this. Therefore, detailed description is omitted.

  The determination of the staple position by template matching can be processed by the server 121. In this case, even if the stapling position is not fixed, the process of S1310 is not performed in the flowchart of FIG. Further, the staple position information is not transmitted from the portable terminal 101 to the server 121. When trimming is performed in step S608, the processing in step S1306 may be performed simultaneously using the CPU 401 and the ROM 403 included in the server 121.

  As described above, when the image processing apparatus according to the second embodiment is used, when a stapled double-sided document is scanned with a portable terminal, it is possible to photograph a spread simultaneously and further print in page order.

[Other Examples]
The present invention can also be realized by executing the following processing. That is, software (program) that realizes the functions of the above-described embodiments is supplied to a system or apparatus via a network or various storage media, and a computer (or CPU, MPU, etc.) of the system or apparatus reads the program. It is a process to be executed.

101 mobile terminal 310 image processing apparatus 301 CPU of mobile terminal

Claims (10)

Stapling position determining means for determining the position of the staple in image data obtained by photographing a document in which a plurality of sheets are stapled;
A document area detecting means for detecting a document area for at least one page in the image data;
Control for determining the page order of each page in the document area for the plurality of pages based on the staple position determined by the staple position determination means when the document area detection means detects a document area for a plurality of pages. An image processing apparatus comprising: means.   2. The image processing apparatus according to claim 1, wherein when a plurality of document areas is detected by the document area detection unit, the control unit trims image data corresponding to the document area in accordance with the determined page order. Image processing apparatus.   The image processing apparatus according to claim 1, wherein the staple position determining unit determines the staple position based on information input by a user.   The image processing apparatus according to claim 1, wherein the staple position determining unit determines the staple position using a first page of the image data.   The image processing apparatus according to claim 1, wherein the staple position determining unit determines the staple position using the second and subsequent pages of the image data.   6. When one original area is detected by the original area detecting means, the control means determines the original area as the last page of the image data. The image processing apparatus according to any one of the above.   The image processing apparatus according to claim 1, wherein the control unit controls the printing unit to print the image data in accordance with the determined page order.   The image processing apparatus according to claim 1, wherein the document area detecting unit detects a document area based on a rectangular area in the image data. A staple position determining step for determining the position of the staple in image data obtained by photographing a document on which a plurality of sheets are stapled;
A document region detecting step for detecting a document region for at least one page in the image data;
Control for determining the page order of each page in the document area for the plurality of pages based on the staple position determined in the staple position determination step when a document area for a plurality of pages is detected in the document area detection step. An image processing method executed by a processor, comprising a step.   A program for causing a computer to execute the image processing method according to claim 9.

Images