JP4378408B2 - Image processing apparatus, image reading apparatus, and image forming apparatus - Google Patents

Description

The present invention, images processor you combining two images, the image reading apparatus, and an image forming apparatus.

  Conventionally, in an image forming apparatus such as a copying machine, a plurality of input images are partially overlapped and connected to create one image, and image formation may be performed on the created image. Therefore, as an image processing apparatus applied to a copying machine that copies an image of a document onto a recording medium, when copying an image of a document that is larger than the scanner size, which is the maximum size that can be read by the image reading unit, After performing multiple reading processes for each of a plurality of partial images obtained by partially overlapping and dividing the original image, the read partial images are joined together to restore the original document image, and the restored document Are output in accordance with the size of the recording medium (for example, see Patent Document 1).

In the image processing apparatus disclosed in Patent Literature 1, binarization processing and edge portion extraction processing are performed for each of a plurality of input images, and binarization processing and edge extraction processing are performed. Each input image is compared by the pattern matching method, and the two input images are connected so that the portions where the features of the edge portion match are overlapped.
JP-A-4-314263

  However, in the configuration disclosed in Patent Document 1, the image after the binarization process and the edge extraction process is compared using the pattern matching method as the feature data of each input image. In this case, it is implied that the original read is not rotated. Therefore, when pattern matching is performed on two actual images, the matching point cannot be detected accurately. It was.

The present invention has been made in view of such circumstances, and extracts a plurality of connected regions in which pixels are connected from each of two images, and characterizes each image based on feature points included in each extracted connected region. Each feature amount to be expressed is calculated, and the feature points are associated by comparing the calculated feature amounts, and a transformation matrix representing the coordinate transformation of the two images is calculated using the information on the position of the associated feature points In addition, by composing the image by converting one image using the calculated conversion matrix, even when the two input images are inclined to each other, the feature points can be accurately associated with each other. it can be performed, and an object thereof is to provide an image processing MakotoSo location capable of performing high accuracy image synthesis, the image reading apparatus, and an image forming apparatus.

  Another object of the present invention is to provide an image processing apparatus capable of selecting a feature point to be used and improving the accuracy of the transformation matrix (image synthesis accuracy) when calculating the transformation matrix.

  Another object of the present invention is to store the synthesized image data in association with the feature amount, the index representing the document, and the tag information representing the synthesized image, and perform the image synthesizing operation again. An object of the present invention is to provide an image processing apparatus that can extract composite image data using an original document without using the original document.

An image processing apparatus according to the present invention captures two images having areas that should overlap with each other, and combines the two captured images in the areas, in which a pixel is connected from each of the two images. Means for extracting a plurality of regions, means for extracting feature points included in each extracted connected region, means for calculating feature amounts representing the features of each image based on the extracted feature points, and each calculated image Means for associating feature points extracted from each connected region by comparing the feature quantities of the image and the coordinate system of one image using the information of the position of the associated feature point It means for calculating a transformation matrix copy to, and means for combining the two images by transforming the image of one using the calculated transformation matrix, the image matching for matching images captured with the stored image A step of extracting a connected region in which pixels are connected from the captured image, a means for extracting a feature point included in the extracted connected region, and a feature point based on the extracted feature point. Means for calculating a feature amount representing a feature of the image, and means for comparing the calculated feature amount with a feature amount of a stored image stored in advance and voting on a stored image having a matching feature amount, based on the voting results by means and Citea Rukoto to perform analogy determination between an image and storing the image captured.

In the present invention, the feature points of each image are calculated using the feature points extracted from each of the two images, and the feature points are associated by comparing the calculated feature amounts. A transformation matrix representing coordinate transformation is obtained by using the coordinates of the feature points. In conventional image synthesis, pattern matching using various feature points has been used. However, in the present invention, feature amounts that are invariant to geometrical changes including image rotation, translation, and scaling are used. Since feature points are associated with each other, accurate association is possible even when there is an inclination or the like between two images, and the accuracy of image synthesis is improved.
Further, in the present invention, the image collating means is based on the extracted feature points, the means for extracting the connected areas obtained by connecting the pixels from the image, the means for extracting the feature points included in the extracted connected areas, and the extracted feature points. The image processing apparatus includes: means for calculating a feature amount representing the feature of the image; and means for comparing the calculated feature amount with the feature amount of the stored image stored in advance and voting on the stored image with the matching feature amount. By using a part of the function of the matching means, it is possible to associate the feature points extracted from the two images.

  The image processing apparatus according to the present invention is characterized in that the means for extracting feature points removes feature points that become an obstacle when calculating the transformation matrix from the extracted feature points.

  In the present invention, since image synthesis is performed using the selected feature points, the accuracy of image synthesis is improved.

  The image processing apparatus according to the present invention includes a synthesized image data, a feature amount extracted from the synthesized image data, first identification information for identifying each of the synthesized image data, and synthesized image data. A control unit that stores the information in the storage unit in association with the second identification information indicating that

  In the present invention, since the image data synthesized based on the stored information associated with the storage means can be read out, the original document can be used without performing the image synthesizing operation again. Composite image data can be extracted.

  The image processing apparatus according to the present invention includes means for calculating the center of gravity of the extracted connected area, and the calculated center of gravity is used as a feature point of the connected area.

  In the present invention, since the center of gravity of the extracted connected region is calculated and the calculated center of gravity is used as the feature point, the feature point can be extracted from any image, and the feature amount can be calculated quickly and accurately. Become.

  The image processing apparatus according to the present invention is characterized in that the feature amount is a parameter that is invariant to a geometric change including rotation, translation, and enlargement / reduction of the image.

  In the present invention, parameters that are invariant to geometric changes including image rotation, parallel movement, and enlargement / reduction are calculated as feature amounts. Therefore, when the image to be synthesized is scanned upside down. Even if it exists, the accuracy of image composition is maintained.

  The image processing apparatus according to the present invention includes means for calculating a hash value by a hash function formulated using a distance between feature points extracted from one image, and the calculated hash value is used as a feature amount of the image. It is characterized by that.

  In the present invention, the hash value is calculated by the hash function formulated using the distance between the feature points, so the invariant according to the geometrical arrangement of the feature points is calculated.

  The image processing apparatus according to the present invention is characterized in that a region to be synthesized is preset for each image.

  In the present invention, since the areas to be combined are set in advance, the processing time is shortened.

  An image reading apparatus according to the present invention includes a document table on which a document is placed, a reading unit that reads an image from a document placed on the document table, and the image processing device according to any one of the above-described inventions. And two images read by the reading means are combined by the image processing apparatus.

  In the present invention, image synthesis can be performed on an image read by a scanner device, a digital multi-function peripheral, or the like.

  The image reading apparatus according to the present invention is characterized in that an area to be combined is set for the document table.

  In the present invention, since the area to be combined is set for the document table, the document placement position can be easily specified.

  An image forming apparatus according to the present invention includes the image processing apparatus according to any one of the above-described inventions, and a unit that forms an image obtained by combining two images by the image processing apparatus on a sheet. It is characterized by providing.

  The present invention can be applied to a printer device, a digital multi-function peripheral, and the like, and an image obtained by combining can be formed on a sheet.

  An image forming apparatus according to the present invention includes a document table on which a document is placed, and a reading unit that reads an image from the document placed on the document table, and sets an area to be combined to the document table. It is characterized by being.

  In the present invention, since the area to be combined is set for the document table, the document placement position can be easily specified.

  In the case of the present invention, the feature points of each image are calculated using the feature points extracted from each of the two images, and the feature points are associated by comparing the calculated feature amounts. A transformation matrix representing coordinate transformation is obtained by using the coordinates of feature points. In conventional image synthesis, pattern matching using various feature points has been used. However, in the present invention, feature amounts that are invariant to geometrical changes including image rotation, translation, and scaling are used. Since the feature points are associated with each other, the feature points can be accurately associated even if the two images have an inclination or the like, and the two images can be synthesized with high accuracy.

  In the case of the present invention, the image matching means extracts means for extracting a connected area in which pixels are connected from the image, means for extracting feature points included in the extracted connected area, and image features based on the extracted feature points. And a means for comparing the calculated feature value with a feature value of a stored image stored in advance and voting for a stored image with a matching feature value. Therefore, by using a part of the functions of the image collating means, it is possible to associate the feature points extracted from the two images, and perform image synthesis using information on the positions of the associated feature points. be able to. In addition, since some functions of the image composition means are used, the circuit configuration added to perform image composition can be minimized.

  In the case of the present invention, since image synthesis is performed using the selected feature points, the accuracy of image synthesis can be improved.

  According to the present invention, the image data synthesized based on the stored information associated with the storage means can be read out, so that the original document can be synthesized without performing the image synthesis operation again. Image data can be extracted.

  According to the present invention, the center of gravity of the extracted connected area is calculated, and the calculated center of gravity is used as a feature point. Therefore, feature points can be extracted from an arbitrary image, and feature quantities can be calculated with high speed and accuracy.

  According to the present invention, parameters that are invariant to geometrical changes including image rotation, parallel movement, and enlargement / reduction are calculated as feature amounts. Therefore, the image to be synthesized is scanned upside down. However, the accuracy of image composition can be kept above a certain level.

  In the case of the present invention, since the hash value is calculated by the hash function formulated using the distance between the feature points, the invariant according to the geometrical arrangement of the feature points can be calculated. .

  In the case of the present invention, the processing time can be shortened because the areas to be combined are preset.

  According to the present invention, it is possible to perform image composition on an image read by a scanner device, a digital multifunction peripheral, or the like.

  According to the present invention, since the area to be combined is set for the document table, the document placement position can be easily specified.

  According to the present invention, it can be applied to a printer device, a digital multi-function peripheral, and the like, and an image obtained by combining can be formed on a sheet.

Hereinafter, the present invention will be specifically described with reference to the drawings illustrating embodiments thereof.
Embodiment 1 FIG.
FIG. 1 is a block diagram illustrating an internal configuration of an image processing system including an image processing apparatus according to the present invention. The image processing system according to the first embodiment includes an operation panel 1, an image input device 3, an image processing device 5A, and an image output device 7.

  The operation panel 1 includes a liquid crystal display device, various switches, and the like, and accepts display of information to be notified to the user, various selection operations by the user, and the like.

  The image input device 3 is a reading unit that optically reads an image of a document, and includes a light source that irradiates light to a document for reading, an image sensor such as a CCD (Charge Coupled Device), and the like. In the image input device 3, a reflected light image from a document set at a predetermined reading position is formed on the image sensor, and RGB (R: Red, G: Green, B: Blue) analog electrical signals are output. . The analog electric signal output from the image input device 3 is input to the image processing device 5A.

  The image processing device 5 </ b> A converts the analog electrical signal output from the image input device 3 into a digital electrical signal, performs appropriate image processing, and outputs the obtained image data to the image output device 7. The internal configuration and operation of the image processing apparatus 5A will be described in detail later.

  The image output device 7 is means for forming an image on a sheet such as paper or an OHP film based on an image signal output from the image processing device 5A. For this reason, the image output device 7 is a charger for charging the photosensitive drum to a predetermined potential, and laser writing for generating an electrostatic latent image on the photosensitive drum by emitting laser light in accordance with image data received from the outside. An apparatus, a developing device that supplies toner to the electrostatic latent image formed on the surface of the photosensitive drum to make it visible, a transfer device that transfers the toner image formed on the surface of the photosensitive drum onto paper (not shown), etc. And an image desired by the user is formed on a sheet by electrophotography. In addition to image formation by an electrophotographic method using a laser writing apparatus, an image formation may be performed by an inkjet method, a thermal transfer method, a sublimation method, or the like.

  Next, the internal configuration of the image processing apparatus 5A will be described. The AD converter 51 converts RGB analog signals input from the image input device 3 into digital signals. The shading correction unit 52 performs a process of removing various distortions generated in the illumination system, the imaging system, and the imaging system of the image input device 3 on the digital RGB signal output from the AD conversion unit 51. The RGB signal subjected to the shading correction is output to the document matching processing unit 53.

  The document matching processing unit 53 determines whether or not the image captured through the image input device 3 is similar to a stored image (hereinafter referred to as a registered format), and determines that the image is similar. In this case, it is determined whether or not the input image is an image written in the registered format. If it is determined that the image has been written in the registered format, an area corresponding to the writing is extracted, and the image in the extracted area is stored in association with the registered format.

  When the two images captured through the image input device 3 are synthesized, the image synthesis processing unit 54 extracts a feature point common to both images using a part of the function of the document matching processing unit 53, and extracts the extracted features. Points are associated with each other, a transformation matrix that maps the coordinate system of one image to the coordinate system of the other image is obtained, and image synthesis is performed by transforming one image using the obtained transformation matrix. .

  The input tone correction unit 55 performs image quality adjustment processing such as background density removal and contrast. The region separation processing unit 56 performs processing for separating each pixel in the input image into one of a character region, a halftone dot region, and a photographic region from the RGB signals. The region separation processing unit 56 sends a region identification signal indicating which region the pixel belongs to to the subsequent black generation and under color removal unit 58, the spatial filter processing unit 59, and the gradation reproduction processing unit 62 based on the separation result. At the same time, the input RGB signal is output to the subsequent color correction unit 57 as it is.

  The color correction unit 57 performs processing for removing color turbidity based on the spectral characteristics of CMY color materials including unnecessary absorption components in order to faithfully reproduce color reproduction. The color-corrected RGB signal is output to the subsequent black generation and under color removal unit 58. The black generation and under color removal unit 58 generates black (K) signals from the CMY three-color signals after color correction, and generates a new CMY signal obtained by subtracting the K signal obtained by black generation from the original CMY signals. Generate the process. By this processing, the CMY three-color signal is converted into a CMYK four-color signal.

  As an example of the black generation process, there is a method of generating black using skeleton black. In this method, the input / output characteristic of the skeleton curve is y = f (x), the input data is C, M, Y, the output data is C ′, M ′, Y ′, K ′, the UCR rate (UCR) : Under Color Removal) α (0 <α <1), the black generation under color removal processing is expressed by the following equation.

K ′ = f {min (C, M, Y)}
C ′ = C−αK ′
M ′ = M−αK ′
Y ′ = Y−αK ′

  The spatial filter processing unit 59 performs spatial filter processing using a digital filter on the image data of the CMYK signal input from the black generation and under color removal unit 58 based on the region identification signal, thereby correcting the spatial frequency characteristics. Processing is performed to prevent blurring and particulate deterioration of the output image.

  For example, the region separated into characters by the region separation processing unit 56 has a high frequency enhancement amount in the sharp enhancement processing in the spatial filter processing by the spatial filter processing unit 59 in order to improve the reproducibility of black characters or color characters in particular. Increased. At the same time, the gradation reproduction processing unit 62 selects binarization or multi-value processing on a high-resolution screen suitable for high-frequency reproduction. Further, with respect to the region separated into halftone dot regions by the region separation processing unit 56, the spatial filter processing unit 59 performs low pass filter processing for removing the input halftone component. The output tone correction unit 60 performs output tone correction processing for converting a signal such as a density signal into a halftone dot area ratio that is a characteristic value of the color image output device, and then the tone reproduction processing unit 62 performs the final processing. Specifically, gradation reproduction processing is performed in which an image is separated into pixels and processed so that each gradation can be reproduced. In addition, regarding the region separated into photographs by the region separation processing unit 56, binarization or multi-value processing is performed on the screen with an emphasis on gradation reproducibility. Further, the scaling processing unit 61 performs scaling processing as necessary before performing the area reproduction processing.

  The image data subjected to the above-described processes is temporarily stored in storage means (not shown), read at a predetermined timing, and output to the image output device 7.

  The present embodiment is characterized in that the image composition processing unit 54 performs image composition by using a part of the function of the document collation processing unit 53. Hereinafter, details of the document matching processing unit 53 will be described.

  FIG. 2 is a block diagram showing the internal configuration of the document collation processing unit 53. The document collation processing unit 53 includes a control unit 530, a feature point calculation unit 531, a feature amount calculation unit 532, a voting processing unit 533, a similarity determination processing unit 534, and a data storage unit 535.

  The control unit 530 is, for example, a CPU, and controls each of the hardware units described above. The feature point calculation unit 531 extracts a connected part from a character string, ruled line, and the like included in the input image, and calculates the center of gravity of the connected part as a feature point. The feature amount calculation unit 532 uses the feature points calculated by the feature point calculation unit 531 to calculate a feature amount (hash value) that is an invariable amount with respect to rotation, enlargement, and reduction. The voting processing unit 533 uses the feature amount calculated by the feature amount calculation unit 532 to vote for a registration format registered in advance in the data storage unit 535. The similarity determination processing unit 534 determines the analogy between the input image and the registration format using the vote result.

  FIG. 3 is a block diagram showing a configuration of the feature point calculation unit 531. The feature point calculation unit 531 includes an achromatic processing unit 5311, a resolution conversion unit 5312, a filter processing unit 5313, a binarization processing unit 5314, and a centroid calculation unit 5315.

  The achromatic processing unit 5311 is a processing unit for performing achromatic processing when the input image data is a color image and converting it to a lightness signal or a luminance signal. For example, the luminance signal is obtained by the following conversion formula.

  Yj = 0.30Rj + 0.59Gj + 0.11Bj

Here, Yj represents the luminance value of each pixel, and Rj, Gj, Bj represents the color component of each pixel. Instead of this method, RGB signals may be converted into CIE 1976 L ^* a ^* b ^* signals (CIE: Commission International de l'Eclairage, L *: lightness, a ^* , b ^* : chromaticity).

  The resolution conversion unit 5312 is a processing unit that, when the input image data has been optically scaled by the image input device 3, scales again so as to obtain a predetermined resolution. Further, the resolution conversion unit 5312 is also used as resolution conversion for lowering the resolution from the resolution read by the image input apparatus 3 at the same magnification in order to reduce the subsequent processing amount. For example, image data read at 600 dpi (dot per inch) is converted to 300 dpi.

  The filter processing unit 5313 is a processing unit used to absorb that the spatial frequency characteristics of the image input device are different for each model. The image signal output by the CCD is blurred due to optical components such as lenses and mirrors, aperture aperture of the light receiving surface of the CCD, transfer efficiency and afterimage, integration effect due to physical scanning, and scanning unevenness. Degradation such as has occurred. The filter processing unit 5313 performs a process of repairing the blur caused by the degradation of the MTF by performing an appropriate filter process (enhancement process). It is also used to suppress high-frequency components that are not necessary for subsequent processing. That is, the enhancement and smoothing processing is performed using the mixing filter.

  The binarization processing unit 5314 is a processing unit that creates binary image data suitable for centroid calculation from achromatic image data. The center-of-gravity calculation unit 5315 obtains the center of gravity of the connected component from the binarized data, and outputs this to the feature quantity calculation unit 532 as a feature point. As a method for calculating the center of gravity, a conventional method can be used. In other words, each pixel is labeled based on the binarization information of the binary image, the connected area connected by the pixel with the same label is specified, and the center of gravity of the specified connected area is calculated as a feature point. To do.

  FIG. 4 is a schematic diagram illustrating an example of feature point extraction. FIG. 4A is an example in which the character “A” is specified as a connected region by the above-described method, and shows how the points indicated by black circles in the figure are calculated as feature points (centroids). Yes. FIG. 4B shows an example in which a connected area is similarly extracted from the characters “book”, and shows a state in which the connected area is specified by being divided into two areas. In this case, since feature points (center of gravity) are calculated from each connected region, two feature points (feature point A and feature point B) are calculated from one character.

  Next, a feature amount calculation method will be described. FIG. 5 is a block diagram illustrating a configuration of the feature amount calculation unit 532. The feature amount calculation unit 532 includes a neighboring point extraction unit 5321 and a feature amount extraction unit 5322. The neighboring point extraction unit 5321 selects an arbitrary one of the plurality of feature points calculated by the feature point calculation unit 531 as the feature point of interest, and sets the four feature points in the order of decreasing distance from the feature point of interest. Select as a feature point. The feature amount extraction unit 5322 calculates a hash value (feature amount) based on the distances of the four peripheral feature points with respect to the feature point of interest.

  FIG. 6 is an explanatory diagram for explaining the feature point of interest and the peripheral feature points. FIG. 6 shows a state where six feature points P <b> 1 to P <b> 6 are calculated by the feature point calculation unit 531. At this time, when the feature amount calculation unit 532 selects the feature point P3 as the feature point of interest, the feature points P1, P2, P4, and P5 are selected as the peripheral feature points. The feature amount calculation unit 532 uses the selected feature point of interest (P3) and the peripheral feature points (P1, P2, P4, P5) to calculate an invariant that is invariant due to the inclination, movement, rotation, etc. of the input image, The feature amount of the input image is calculated from the calculated invariant.

  FIG. 7 is an explanatory diagram for explaining an example of calculating invariants by the feature point P3 of interest. The invariant H3j is defined by H3j = A3j / B3j using the distance between the target feature point P3 and the peripheral feature points P1, P2, P4, and P5. Here, j = 1, 2, and 3 are taken, and A3j and B3j indicate distances between the feature points, and the distances between the feature points are calculated based on the coordinate values of the respective peripheral feature points. . That is, three invariants are calculated, the value of invariant H31 is A31 / B31 (see FIG. 7A), the value of invariant H32 is A32 / B32 (see FIG. 7B), and invariant H33. Is A33 / B33 (see FIG. 7C). These invariants H3j, for example, do not change even when the document is rotated, moved, or tilted when the document is read, and the image similarity determination can be performed with high accuracy in the subsequent similarity determination.

  FIG. 8 is an explanatory diagram illustrating an example of calculating invariants when the feature point of interest is the feature point P4. The feature amount calculation unit 532 selects feature points P2, P3, P5, and P6 as peripheral feature points. At this time, the invariant H4j (j = 1, 2, 3) can be calculated by H4j = A4j / B4j, as described above. That is, the value of the invariant H41 is A41 / B41 (see FIG. 8A), the value of the invariant H42 is A42 / B42 (see FIG. 8B), and the value of the invariant H43 is A43 / B43 (see FIG. 8B). (Refer FIG.8 (c)).

  The same applies to the case where other feature points P1, P2, P5, and P6 are selected as the feature points of interest, and the feature amount calculation unit 532 sequentially changes the feature points of interest, and sets the feature points P1, P2,. The invariant Hij (i = 1, 2,..., 6: j = 1, 2, 3) when selected is calculated.

Next, the feature amount calculation unit 532 calculates a feature amount (hash value) Hi using the invariant calculated by each feature point of interest. The hash value Hi when the feature point of interest is the feature point Pi is represented by Hi = (Hi1 × 10 ² + Hi2 × 10 ¹ + Hi3 × 10 ⁰ ) / E. Here, i is a natural number and represents the number of feature points, and E is a constant determined by how much remainder is set. For example, when E = 10, the remainder is a value of 0 to 9. This is the range that the hash value calculated can take.

FIG. 9 and FIG. 10 are explanatory diagrams for explaining other examples of invariants when the feature point of interest is P3 and P4, respectively. As a method for calculating the invariant based on the feature point of interest, for example, as shown in FIG. 9, four combinations are selected from the four feature points P1, P2, P4, and P5 around the feature point P3. The variable H3j (j = 1, 2, 3, 4) may be calculated by H3j = A3j / B3j in the same manner as described above. Similarly, when the target feature point is set to P4, four combinations are selected from the four peripheral feature points P2, P3, P5, and P6 of the target feature point P4 (see FIG. 10), and the invariant H4j is selected. (J = 1, 2, 3, 4) may be calculated by H4j = A4j / B4j. In this case, the hash value is calculated as Hi = (Hi1 × 10 ³ + Hi2 × 10 ² + Hi3 × 10 ¹ + Hi4 × 10 ⁰ ) / E.

  The hash value as the feature amount is an example, and the hash value is not limited to this, and other hash functions can be used. In the above description, four feature points are selected as the peripheral feature points, but the number is not limited to four. For example, six may be extracted. In this case, for each of the six methods of extracting five from six feature points and extracting five, three points may be extracted from five points to obtain an invariant and a hash value may be calculated. .

  In this way, the feature amount calculation unit 532 calculates the feature amount (hash value) for each connected region. The voting processing unit 533 searches the hash table based on the hash value calculated by the feature amount calculation unit 532 and votes for the document with the registered index. FIG. 11 is a conceptual diagram showing an example of a hash table. The hash table is composed of each column of a hash value and an index representing a registration format. That is, as shown in FIG. 11A, an index representing a registration format is registered corresponding to the feature amount representing the feature of the connected area. For example, when the calculated hash value is “H1”, the voting processing unit 533 votes for a registration format having an index of “ID1”. When the calculated hash value is “H3”, the voting processing unit 533 votes for two types of registration formats (that is, registration formats having indexes of “ID2” and “ID3”). The same applies when the calculated hash value is another value. When the hash values are equal (H1 = H5), as shown in FIG. 11B, it is possible to combine two entries in the hash table into one.

  FIG. 12 is a graph showing an example of the voting result. The horizontal axis represents the type of registration format, and the vertical axis represents the number of votes. The example illustrated in FIG. 12 illustrates a state where voting is performed for three types of registration formats (“N1” to “N3”). The vote result obtained by cumulatively adding votes is output to the similarity determination processing unit 534.

  The similarity determination processing unit 534 determines the image similarity based on the voting result input from the voting processing unit 533 and notifies the control unit 530 of the determination result. The similarity determination processing unit 534 compares the number of votes (number of votes obtained) input from the voting processing unit 533 with a predetermined threshold, and determines that the input image is similar to the registered format when the number of votes is equal to or greater than the threshold. . When the number of votes input from the voting processing unit 533 is smaller than the threshold value, the similarity determination processing unit 534 determines that there is no similar document and notifies the control unit 530 of the result.

  Note that the above determination method is an example. As another method, for example, after dividing the number of votes by the maximum number of votes for each document (such as the number of feature points required for each document) and normalizing, the similarity determination is performed. You may go.

  In the image composition processing unit 54, when combining two document images using a part of the function of the document collation processing unit 53, first, the coordinates of the feature points of the document image read first time and the second time read. Corresponding to the coordinates of the feature points of the original document image. FIG. 13 is an explanatory diagram for explaining the association between feature points extracted from two images. In the example shown in FIG. 13, four feature points having coordinates (x1, y1), (x2, y2), (x3, y3), and (x4, y4) from the original image read for the first time. Each feature point has been extracted, and (x1 ′, y1 ′), (x2 ′, y2 ′), (x3 ′, y3 ′), and (x4 ′) extracted from the original image read for the second time. , Y4 ′) are associated with four feature points having coordinates.

  When the matrix created using the coordinates of the feature points of the original image read in the first time is Pout, and the matrix created using the coordinates of the feature points of the original image read in the second time is Pin, one original A conversion formula that maps the image coordinate system to the coordinate system of the other document image can be expressed as follows using the conversion matrix A.

Since the matrix Pin is not a square matrix, the transformation matrix A can be obtained by multiplying both sides by the transposed matrix Pin ^T of Pin and further multiplying by the inverse matrix of Pin ^T Pin.

  Therefore, a conversion formula that maps the coordinates (x, y) on one image to the coordinate system on the other image can be expressed as follows using the conversion matrix A.

  The image composition processing unit 54 performs image composition by converting the coordinates on one image using Equation 3.

  Hereinafter, a specific processing procedure for image composition will be described. FIG. 14 is a flowchart for explaining an image composition processing procedure by the image processing system according to the first embodiment. First, the image processing apparatus 5A sets the number k of image data captures to 1 (step S11), and captures the k-th image data (step S12).

  FIG. 15 is an explanatory diagram for explaining a document reading procedure. In the present embodiment, a document (for example, an A2 size document) having a size exceeding the size (for example, A3 size) of the platen 30 on which the document is placed is scanned in two times including some overlapping areas. FIG. 15A shows a state in which the upper area of the document is scanned, and FIG. 15B shows a state in which the lower area of the document is scanned by overlapping a part of the upper area. Is shown. That is, images are captured so as to have overlapping areas.

  Next, a feature amount calculation process is performed by using the feature point calculation unit 531 and the feature amount calculation unit 532 of the document matching processing unit 53 (step S13). At this time, the document matching processing unit 53 stores the index of the feature point corresponding to the hash value calculated as the feature quantity, and stores the coordinates of the feature point in association with each index. FIG. 16 is a diagram showing the correspondence between feature quantities (hash values) and feature point indexes, and FIG. 17 is a diagram showing the correspondence between feature point indexes and coordinates. Such a correspondence relationship is stored in the data storage unit 535 of the document collation processing unit 53, for example.

  Next, the image processing apparatus 5A determines whether or not the captured image data is the second time (that is, whether or not k = 2) (step S14), and if it is not the second time (S14: NO). ), The value of k is incremented by one (step S15), and the process proceeds to step S12.

  When the captured image is the second time (S14: YES), feature points are associated (step S16). The correspondence between the hash value calculated from the first and second captured images and the feature point index is as shown in FIGS. 16A and 16B, respectively. 17A and 17B, the feature points f1, f2, f3, and f4 calculated from one image are respectively the other image. It can be seen that they are associated with the feature points p1, p2, p3, p4 extracted from.

  When two images are combined using these feature points, the image composition processing unit 54 calculates the transformation matrix A described above using two or more corresponding feature points (step S17). By using the calculated conversion matrix A, arbitrary coordinates (x, y) on the first image are converted into coordinates (x ′, y ′) on the second image. As a result, even if the first document has an inclination, the conversion matrix A is used to perform coordinate conversion in accordance with the coordinate system of the second scan image, and can be combined as a continuous image as a whole. Yes (step S18).

  Note that the feature points and feature amounts may be calculated for the entire image, or a region to be overlapped may be set in advance, and the feature points and feature amounts may be calculated only within the set region. FIG. 18 is a schematic diagram showing an example of setting the overlapping area. In the example shown in FIG. 18, the indicator plate 31 is provided on the side surface of the platen 30 in order to clearly indicate the overlapping region. The indication plate 31 includes an index 32, and an area 30a on the platen designated by the index 32 is set as an overlapping area. By executing the first and second scans in accordance with this region and creating input image data, a series of processing may be executed only for the region of the set width.

  When the overlapping area is not set, when associating f1, f2, f3, f4,... Calculated from one image with the feature points p1, p2, p3, p4,. The feature points may be selected. Since the original is read redundantly, most feature points are associated with those in the overlapping region, but in some cases, the hash values of the feature points in the overlapping region match the hash values in the non-overlapping region. It is possible that If the coordinate data of feature points other than the overlapping region is included, the accuracy of the transformation matrix may be lowered. Therefore, only the feature points in the overlapping region are used.

  FIG. 19 is a schematic diagram illustrating an example of feature points extracted from each of two images to be associated. For example, feature points having the same feature amount are extracted from both images, and the four sides E1, E2 of the document are extracted for the extracted feature points f1, f2, f3, f4,... And p1, p2, p3, p4,. , E3, E4, and E1 ′, E2 ′, E3 ′, and E4 ′ are added to the histogram, and each feature point is extracted from an area close to which side of the document (which area is Whether it has been duplicated). FIG. 20 is an explanatory diagram for explaining a histogram to be created. Of the above histogram, the edge from which the feature point is extracted has the smallest histogram (cumulative value of the distance from each point to the edge), so from which region the feature point is extracted by using this feature It can be determined (FIG. 20 shows an example in which the coordinates of two feature points are added). Then, by selecting a feature point within a predetermined range (for example, 500 at the coordinate position of the feature point) from the specified side, unnecessary feature points can be deleted to obtain a transformation matrix. Further, since it is automatically determined in which region (side) the feature point exists, the user can read the document without considering the direction of the document when reading the document.

  In the case of performing image composition using a digital copying machine or a multifunction machine, for example, an image composition mode is selected from the operation panel 1. The setting of the mode is recognized by the control unit 530, the image composition processing is performed using the function of the document collation processing unit 53, and the image data obtained by the composition is a value determined in advance by the scaling processing unit 61 (or the user And is sent to the image output device 7 through the gradation reproduction processing unit 62.

Embodiment 2. FIG.
In the first embodiment, image composition is performed using the document collation processing unit 53. However, a configuration in which the function of document collation processing is incorporated in the image composition processing may be employed. In the present embodiment, a configuration of an image processing apparatus in which a document matching process is incorporated in an image composition process will be described.

  FIG. 21 is a block diagram illustrating an internal configuration of an image processing system including an image processing apparatus according to the present invention. The image processing system according to the second embodiment includes an operation panel 1, an image input device 3, an image processing device 5B, and an image output device 7. Among these, the configuration other than the image processing device 5B is exactly the same as that of the first embodiment.

  The image processing device 5B includes an AD conversion unit 51, a shading correction unit 52, an image composition processing unit 64, an input tone correction unit 55, a region separation processing unit 56, a color correction unit 57, a black generation and under color removal unit 58, and a spatial filter. The processing unit 59, the output tone correction unit 60, the scaling processing unit 61, and the tone reproduction processing unit 62 are configured. Among these, the configuration other than the image composition processing unit 64 is exactly the same as in the first embodiment.

  FIG. 22 is a block diagram showing the internal configuration of the image composition processing unit 64. The image composition processing unit 64 includes a control unit 640, a feature point calculation unit 641, a feature amount calculation unit 642, a voting processing unit 643, a composition processing unit 644, and a data storage unit 645.

  The control unit 640 is, for example, a CPU, and controls each hardware unit described above. The feature point calculation unit 641 extracts a connected part from a character string, ruled line, or the like included in the input image, and calculates the center of gravity of the connected part as a feature point. The feature amount calculation unit 642 uses the feature points calculated by the feature point calculation unit 641 to calculate a feature amount (hash value) that is an invariable amount with respect to rotation, enlargement, and reduction. The voting processing unit 643 uses the feature amount calculated by the feature amount calculation unit 642 to vote for a registration format registered in advance in the data storage unit 645. The composition processing unit 644 obtains the correspondence between the feature points extracted from the two images by using the voting result from the voting processing unit 643, and calculates a transformation matrix that is established between the images. Then, image synthesis is performed by transforming one image into the coordinate system of the other image according to the calculated transformation matrix.

  FIG. 23 and FIG. 24 are flowcharts for explaining an image composition processing procedure by the image processing system according to the second embodiment. First, the image processing device 5B sets the number k of image data fetches to 1 (step S21), and fetches the k-th image data (step S22). Then, the feature amount calculation process is performed by using the feature point calculation unit 641 and the feature amount calculation unit 642 of the image composition processing unit 64 (step S23).

  Next, the image processing device 5B determines whether or not the captured image data is the second time (that is, whether or not k = 2) (step S24). If it is not the second time (S24: NO) ), The value of k is incremented by one (step S25), and the process proceeds to step S22.

  FIG. 25 is a schematic diagram illustrating an example of an image captured by the image processing device 5B. The size of the document read by the image input device 3 is slightly larger than the size of the platen 30. In this embodiment, the document is read in two steps, for example, an upper region and a lower region of the document. At this time, reading is performed so that an overlapping region including a common image is formed in the upper region and the lower region (see FIGS. 25A and 25B).

  In this embodiment, n (n is an integer of 2 or more) search image areas are set for the first read image, and n reference image areas are set for the second read image, and feature points in the search image area are set. Are associated with feature points in the reference image area. FIG. 26 is a schematic diagram showing a state in which a search image area and a reference image area are set. That is, n search image regions T1, T2,..., Tn are set in the first read image shown in FIG. 26A, and the second read image shown in FIG. Shows a state in which n reference image areas S1, S2,..., Sn are set.

  Hereinafter, the description will be returned to the flowchart shown in FIG. 23, and the process steps of feature point association and image composition will be described. When the captured image data is the second time (S24: YES), the image composition processing unit 64 sets the index x of the search image area Tx and the reference image area Sx to 1 (step S26). Then, a reference image in the search image area Tx is searched (step S27), and it is determined whether or not the same feature point exists in both areas (step S28). If it is determined that there is no identical feature point (S28: NO), the process returns to step S27.

  If it is determined that there is the same feature point (S28: YES), the image composition processing unit 64 stores the coordinates of the feature points in the search image region Tx and the reference image region Sx (step S29).

  Next, the image composition processing unit 64 determines whether or not two or more sets of the same feature points have been obtained in order to determine whether the conversion matrix can be calculated (step S30). When it is determined that two or more sets of the same feature points are not obtained (S30: NO), it is determined whether the index x has reached the set number n of the search image region Tx and the reference image region Sx (step S31). ). If it is determined that the index x has not reached the set number n (S31: NO), the image composition processing unit 64 increments the value of the index x by one (step S32), and the process returns to step S27. That is, when two or more sets of the same feature points are not obtained, the transformation matrix A cannot be obtained, so that the feature points are also associated with the remaining search image region Tx and reference image region Sx. On the other hand, if it is determined that the index x has reached the set number n in a state where two or more sets of the same feature points have not been obtained (S31: YES), error processing is performed (step S33), and the processing according to this flowchart ends. To do. In the error processing, for example, the user is notified that the image composition cannot be executed.

  When it is determined in step S30 that two or more sets of the same feature points are obtained (S30: YES), the transformation matrix A described in the first embodiment is calculated using two or more sets of corresponding feature points (step S34). ). Then, all the image data of the image provided with the reference image area are subjected to coordinate conversion (step S35), and synthesized as a continuous image as a whole.

  In this embodiment, a search image area is set in the read image read first time, and a reference image area is provided in the read image read second time. However, as shown in the first embodiment, these search image areas are provided. It is possible to obtain a correspondence relationship between the feature points of the redundantly read areas without providing the image area, and perform coordinate conversion to perform image composition.

Embodiment 3 FIG.
In the first and second embodiments, each process is realized by hardware. However, the present invention may be realized by software processing.

  FIG. 27 is a block diagram illustrating the internal configuration of an image processing apparatus in which the computer program of the present invention is installed. In the figure, reference numeral 100 denotes an image processing apparatus according to the present embodiment, specifically a personal computer, a workstation, or the like. The image processing apparatus 100 includes a CPU 101, and hardware such as a ROM 103, a RAM 104, a hard disk 105, an external storage unit 106, an input unit 107, a display unit 108, and a communication port 109 is connected to the CPU 101 via a bus 102. Yes. The CPU 101 controls the above-described hardware units according to a control program stored in advance in the ROM 103.

  The RAM 104 is a volatile memory that temporarily stores various data generated during the execution of the above-described control program or the program code (execution format program, intermediate code program, source program) of the computer program according to the present invention. . The hard disk 105 is a storage means having a magnetic recording medium, and stores the program code of the computer program of the present invention. The external storage unit 106 includes a reading device for reading the program code from the recording medium M on which the program code of the computer program of the present invention is recorded. As the recording medium M, an FD (Flexible Disk), a CD-ROM, or the like can be used. The program code read by the external storage unit 106 is stored in the hard disk 105. The CPU 101 loads the program code according to the present invention stored in the hard disk 105 onto the RAM 104 and executes it, thereby causing the entire apparatus to function as an apparatus that implements the image processing described in the first embodiment. Based on the feature values calculated from one image, the feature points are associated with each other, a conversion matrix that maps the coordinate system of one image to the coordinate system of the other image is calculated, and the calculated conversion matrix is used. The two images are combined.

  The input unit 107 functions as an interface for acquiring image data from the outside. For example, a color scanner device or the like is connected to the input unit 107. The display unit 108 functions as an interface for displaying image data to be processed, image data during image processing, image data after image processing, and the like. The display unit 108 may be configured to display image data by connecting an external display device such as a liquid crystal display device, or the display unit 108 may include a display device and display image data. The communication port 109 is an interface for connecting the printer 150 to the outside. When printing image data that has undergone image processing by the printer 150, the image processing apparatus 100 generates print data that can be decoded by the printer 150 based on the image data, and transmits the generated print data to the printer 150. .

  In the present embodiment, the CPU 101 performs various calculations. However, a dedicated chip for performing calculations related to image processing may be separately provided and calculations may be performed according to instructions from the CPU 101.

  As the recording medium M for recording the computer program code according to the present invention, in addition to the above-mentioned FD and CD-ROM, optical recording medium such as MO, MD, DVD, magnetic recording medium such as hard disk, IC card, memory card Further, it is possible to use a semiconductor memory such as a card-type recording medium such as an optical card, mask ROM, EPROM (Erasable Programmable Read Only Memory), EEPROM (Electrically Erasable Programmable Read Only Memory), flash ROM, or the like. Further, since the system configuration is such that a communication network including the Internet can be connected, the computer program code according to the present invention may be downloaded from the communication network.

  In addition, the computer program of the present invention may be provided as a single application program or utility program, or may be incorporated into another application program or utility program and provided as a partial function of the program. May be. For example, as one form, a form provided by being incorporated in a printer driver is conceivable. The present invention can also be realized in the form of a computer data signal embedded in a carrier wave in which the program code is embodied by electronic transmission.

Embodiment 4 FIG.
In the first and second embodiments, an example of performing image composition using a digital copying machine or a multifunction peripheral has been described. However, processing may be shared and realized using a server connected via a network.

  FIG. 28 is a schematic diagram showing the overall configuration of the network system according to the present embodiment. As shown in FIG. 28, the network system according to the present embodiment includes a server 70, multifunction peripherals (MFPs) 10A, 10B,..., Printers 81A, 81B, ..., facsimile machines 82A, 82B,. ..., digital cameras 84A, 84B, ..., scanners 85A, 85B, ... are connected via a network N. The configuration of the system is not limited to the above, and for example, a plurality of servers 70 may be connected.

  FIG. 29 is a block diagram showing an internal configuration of the MFP 10A (10B) and the server 70. As shown in FIG. The MFP 10A (10B) includes an MFP control unit 11 that controls each part of the hardware in the apparatus, a feature point calculation unit 12 that calculates a feature point from the read document image, and a feature amount of the document image based on the calculated feature point. A feature amount calculation unit 13 to be calculated, an MFP image composition processing unit 14 that performs image composition processing using the calculated feature amount, and a memory 15 are provided. The processing executed by each part of the hardware is the same as the processing executed by the image processing apparatus 5A described in the first embodiment, and thus detailed description thereof is omitted.

  The server 70 includes a server control unit 71 that controls each hardware unit in the apparatus, a server image composition processing unit 72 that performs image composition processing, and a memory 73. The server image composition processing unit 72 performs a process of calculating a transformation matrix mainly based on feature point and feature amount data calculated by the MFP 10A (10B).

  For example, in the MFP 10A, feature points and feature amounts are calculated from the original image read the first time and the original image read the second time and transmitted to the server 70, and the read original image is temporarily stored. Store it on a hard disk. You may make it compress and store as needed. The server 70 calculates a transformation matrix using the feature point data and transmits the calculation result to the multifunction machine 10A. In the MFP 10A, when the conversion matrix is received, the original image stored in the hard disk or the like is read out (if it is compressed, it is read after being decoded), and the image is synthesized using the conversion matrix. Do.

Embodiment 5 FIG.
The synthesized image data (synthesized image data), an index (first identification information) representing the image data of the composition source, a tag (second identification information) indicating that it is the synthesized image data, and the extracted feature amount (In other words, the synthesized image data, the tag indicating the synthesized image data and the hash table) are stored in the server 70 (or a hard disk such as the MFP 10A) and the like, and the image is synthesized from the operation panel of the MFP 10A (10B). It is also possible to select a mode, read the original document of the composite image data, and search for the corresponding composite image data. In this case, once the composite image data is created, the composite image data can be extracted using the original document without performing the image composition operation again.

  FIG. 30 is a schematic diagram illustrating an example of an operation panel, and FIG. 31 is a schematic diagram illustrating an example of a screen displayed when an image composition mode is selected from the operation panel. Since the image composition mode is selected from the operation panel shown in FIG. 30 and the search can be performed by narrowing down the target to the composite image data from the screen shown in FIG. 31, the search can be quickly performed and erroneous determination is suppressed. can do. Since the original document is a part of the composite image, a threshold value for similarity determination is set so that the composite image data can be extracted from the original document. The threshold value may be determined using various originals so that the corresponding composite image data can be extracted. The composite image data may be stored after being compressed by a method according to the type of document such as MMR or JPEG.

It is a block diagram explaining the internal structure of an image processing system provided with the image processing apparatus which concerns on this invention. It is a block diagram which shows the internal structure of a document collation process part. It is a block diagram which shows the structure of a feature point calculation part. It is a schematic diagram which shows the example of extraction of a feature point. It is a block diagram which shows the structure of a feature-value calculation part. It is explanatory drawing explaining an attention feature point and a periphery feature point. It is explanatory drawing explaining the example of calculation of the invariant by the attention feature point P3. It is explanatory drawing explaining the example of calculation of the invariable when a notable feature point is made into the feature point P4. It is explanatory drawing explaining the other example of calculation of the invariable when a notable feature point is set to P3. It is explanatory drawing explaining the other example of calculation of the invariable when a notable feature point is set to P4. It is a conceptual diagram which shows an example of a hash table. It is a graph which shows an example of a voting result. It is explanatory drawing explaining matching of the feature point extracted from two images. 4 is a flowchart for describing a processing procedure of image composition by the image processing system according to the first embodiment. FIG. 10 is an explanatory diagram illustrating a document reading procedure. It is a figure which shows the correspondence of the feature-value (hash value) and the index of a feature point. It is a figure which shows the correspondence of the index of a feature point, and a coordinate. It is a schematic diagram which shows the example of a setting of the area | region to overlap. It is a schematic diagram which shows an example of the feature point extracted from each of two images which perform matching. It is explanatory drawing explaining the histogram to produce. It is a block diagram explaining the internal structure of an image processing system provided with the image processing apparatus which concerns on this invention. It is a block diagram which shows the internal structure of an image composition process part. 10 is a flowchart for describing a processing procedure of image composition by the image processing system according to the second embodiment. 10 is a flowchart for describing a processing procedure of image composition by the image processing system according to the second embodiment. It is a schematic diagram which shows an example of the image taken in into an image processing apparatus. It is a schematic diagram which shows a mode that the search image area | region and the reference image area | region are set. It is a block diagram explaining the internal structure of the image processing apparatus in which the computer program of this invention was installed. It is a schematic diagram which shows the whole structure of the network system which concerns on this Embodiment. 2 is a block diagram illustrating an internal configuration of an MFP and a server. FIG. It is a schematic diagram which shows an example of an operation panel. It is a schematic diagram which shows an example of the screen displayed when the image composition mode is selected from the operation panel.

Explanation of symbols

1 Operation Panel 3 Image Input Device 5A, 5B Image Processing Device
7 Image output device 53 Document collation processing unit 54, 64 Image composition processing unit 530 Control unit 531 Feature point calculation unit 532 Feature amount calculation unit 533 Voting processing unit 534 Similarity determination processing unit 535 Data storage unit

Claims (11)

In an image processing apparatus that captures two images having areas that should overlap each other, and combines the two captured images in the areas,
Means for extracting a plurality of connected regions in which pixels are connected from each of the two images, means for extracting feature points included in each extracted connected region, and features representing the characteristics of each image based on the extracted feature points A means for calculating each of the quantities, a means for associating the feature points extracted from each connected region by comparing the calculated feature quantities of each image, and using the information on the positions of the associated feature points. Means for calculating a transformation matrix that maps the coordinate system of the image to the coordinate system of the other image, means for synthesizing the two images by transforming the one image using the computed transformation matrix , and Image collating means for collating an image with a stored image,
The image matching means extracts means for extracting a connected area in which pixels are connected from the captured image, means for extracting feature points included in the extracted connected area, and features of the image based on the extracted feature points. Means for calculating a feature quantity to be represented, means for comparing the calculated feature quantity with a feature quantity of a stored image stored in advance, and voting on a stored image having a matching feature quantity. based, the image processing apparatus according to claim Citea Rukoto to perform analogy determination between an image and storing the image captured. 2. The image processing apparatus according to claim 1 , wherein the means for extracting feature points removes feature points that become an obstruction factor when calculating a transformation matrix from the extracted feature points. A feature value extracted from the synthesized image data, first identification information for identifying each of the synthesized image data, and a second identification indicating that the synthesized image data is synthesized image data The image processing apparatus according to claim 1 , further comprising a control unit that stores the information in the storage unit in association with the information. And means for calculating the centroid of the extracted connected region, the image processing apparatus according to any one of claims 3 and the calculated centroid claim 1, characterized in that as a characteristic point of the connecting region. The image according to any one of claims 1 to 3 , wherein the feature amount is a parameter that is invariant to a geometric change including rotation, parallel movement, and enlargement / reduction of the image. Processing equipment. And means for calculating a hash value by the hash function is formulated using the distance between the extracted feature point from one image, the calculated hash value from claim 1, characterized in that as a feature value of the image the image processing apparatus according to any one of claims 3. The image processing apparatus according to claim 1 , wherein a region to be synthesized is preset for each image. Comprising a platen for placing an original, reading means for reading an image from a document placed on the document table, and an image processing apparatus according to claim 1, any one of claims 7, wherein the reading An image reading apparatus characterized in that two images read by the means are synthesized by the image processing apparatus. 9. The image reading apparatus according to claim 8 , wherein an area to be combined is set for the document table. An image processing apparatus according to any one of claims 1 to 7 , and means for forming an image obtained by synthesizing two images by the image processing apparatus on a sheet. Image forming apparatus. An original table on which an original is placed, and reading means for reading an image from the original placed on the original table, and an area to be combined is set for the original table. The image forming apparatus according to 10 .

Images