JP2004288148A - Image processing method, system and device, and computer program - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To provide an image processing method for eliminating the distortion of an image, an image processing system, an image processing device and a computer program. <P>SOLUTION: An imaged object is placed on a sheet having a specific pattern, to take its image, and the obtained image is read in the image processing device (S1). Then the features on the image and the features on the pattern are corresponded to each other (S3) after performing the correction of an imaging device (S2), and an initial value of each element of a projective transformation row is calculated (S4). The evaluation function relating to a position coordinate of converting destination is determined (S5), then each element of the projective transformation row is optimized on the basis of the determined evaluation function (S6), and the composite processing of a front image is performed (S7) to produce the front image free from the distortion. <P>COPYRIGHT: (C)2005,JPO&NCIPI

Description

  The present invention relates to an image processing method for removing distortion generated in a captured image, an image processing system, an image processing device, and a computer program for realizing the image processing device.

  In recent years, in the field of apparel, in order to efficiently design clothes and the like, the work of reading digital image data obtained by reading a full-size pattern with an optical reading device such as a scanner device and reading the same on a computer. Has been done.

  However, in order to read the actual size of the paper pattern, a large-sized optical reader is required, and there is a problem that the introduction cost is increased. In addition, there is a problem in that it takes time and effort to perform the input operation, so that introduction of such a device cannot be easily realized.

  Therefore, conventionally, it has been proposed to take an image of a pattern with an imaging device such as a digital camera and use the image data obtained by imaging instead of the image data read by an optical reading device (for example, Patent Document 1).

As described above, when using image data obtained by imaging with an imaging device such as a digital camera, an obtained imaging image includes an imaging device at the time of imaging unless the imaging object is imaged directly from the front. Will result in distortion due to the position and orientation. Therefore, in the system described in Patent Literature 1, an image obtained by capturing an image is taken into an image processing apparatus, and image processing is performed to remove distortion of the captured image.
JP-A-2002-366947

  In the system described in Patent Literature 1, a transparent sheet on which a grid image is drawn is placed on an imaging target, and an image is captured by an imaging device in that state, and distortion of the grid image in the obtained image is obtained. Is used to correct the distortion of the captured image. Therefore, in a captured image obtained by such a method, a lattice image is always applied to an imaging target, and the same image as an image obtained by an optical reading device cannot be obtained. Have a point.

  The present invention has been made in view of such circumstances, and takes in a captured image of an imaging target placed on a mounting surface on which a predetermined pattern is formed, and stores a plurality of images on the pattern stored in advance. Based on the position information of the feature points and the position information of the corresponding points on the corresponding captured image, determine the geometric relationship between the captured image and the mounting surface when capturing the imaging target, and determine Image processing method and image processing capable of generating a distortion-corrected image without affecting the image of the imaging target object by using a configuration that corrects the distortion of the captured image based on the geometric relationship It is an object to provide a system, an image processing device, and a computer program.

  An image processing method according to a first aspect of the present invention is the image processing method according to claim 1, wherein a captured image obtained by capturing an imaging target object is captured by an image processing apparatus, and a distortion-corrected image of the captured captured image is generated. The captured image is an image obtained by capturing an image of an imaging target placed on a mounting surface on which a predetermined pattern is formed, together with the pattern, and preliminarily stores position information of a plurality of feature points on the pattern. Receiving the position information of the corresponding point on the captured image corresponding to the feature point, and based on the position information of the feature point and the position information of the corresponding point, the placement surface and the imaging object A geometric relationship between the captured image and the captured image is determined, and distortion of the captured image is corrected based on the determined geometric relationship.

  An image processing method according to a second aspect of the present invention is an image processing method for capturing a captured image obtained by capturing an image of an object to be captured by an image capturing apparatus into an image processing apparatus, and generating a corrected image in which the captured captured image is corrected for distortion. In the captured image is an image obtained by capturing an image of an imaging target placed on a mounting surface on which a predetermined pattern is formed, and the imaging device of the imaging device when the imaging target is imaged. A reference image obtained by capturing the pattern with the same posture is taken into the image processing apparatus, and position information of a plurality of feature points on the pattern is stored in advance, and a correspondence on the reference image corresponding to the feature point is stored. Accepting the position information of the point, based on the position information of the feature point and the position information of the corresponding point, determine the geometric relationship between the mounting surface described above and the captured image when capturing the pattern, and determine Based on the geometric relationship And correcting distortion of the captured image Te.

  An image processing stem according to a third aspect of the present invention includes: a mounting section having a mounting surface for mounting an imaging target; an imaging device configured to image the imaging target mounted on the mounting surface; An image processing system comprising: a captured image obtained by the device; and an image processing device configured to generate a corrected image in which a distortion of the captured image is corrected. The image processing apparatus is configured to capture a captured image obtained by capturing an image of an imaging target mounted on the mounting surface together with the pattern, and obtain position information of a plurality of feature points on the pattern. A means for storing in advance, a means for receiving position information of a corresponding point on the captured image corresponding to the feature point, and the placement surface described above based on the position information of the feature point and the position information of the corresponding point. Imaging when capturing an image of an object Characterized in that it comprises means for determining the geometric relationship between the image, and means for correcting distortion of the captured image based on the geometrical relationship determined.

  An image processing system according to a fourth aspect of the present invention provides a mounting section having a mounting surface on which an imaging target is mounted, an imaging device configured to image the imaging target mounted on the mounting surface, An image processing system comprising: a captured image obtained by the device; and an image processing device configured to generate a corrected image in which a distortion of the captured image is corrected. The image processing apparatus may be configured such that the captured image obtained by capturing the imaging target placed on the mounting surface by the imaging device and the orientation of the imaging device when capturing the imaging target are the same and the pattern is Means for pre-storing position information of a plurality of feature points on the pattern, and receiving position information of a corresponding point on the reference image corresponding to the feature point. Means and said features Means for determining a geometric relationship between the placement surface and the captured image when the pattern is captured, based on the location information and the location information of the corresponding point, based on the determined geometric relationship. Means for correcting distortion of the captured image.

  An image processing system according to a fifth invention is the image processing system according to the third invention or the fourth invention, wherein the geometric relationship is represented by projective transformation between the feature point and the corresponding point. Features.

  An image processing system according to a sixth aspect of the present invention is the image processing system according to any of the third to fifth aspects, wherein: a means for converting a point on the corrected image to be generated in accordance with the geometric relationship; Means for acquiring color information from the captured image, wherein a corrected image is generated based on the acquired color information.

  An image processing system according to a seventh aspect of the present invention is the image processing system according to any of the third to sixth aspects, wherein the pattern image is a grid-like image.

  An image processing apparatus according to an eighth aspect of the present invention is an image processing apparatus which captures a captured image of an imaging target object and generates a corrected image in which distortion of the captured image is corrected. Means for pre-storing positional information of a plurality of feature points on the pattern, the means for capturing a captured image obtained by capturing an image of the imaging object mounted on the pattern together with the pattern; Means for receiving position information of a corresponding point on the corresponding captured image, and a captured image when the mounting surface and the imaging target are imaged, based on the position information of the feature point and the position information of the corresponding point. And means for correcting distortion of the captured image based on the determined geometric relationship.

  An image processing apparatus according to a ninth aspect of the present invention is an image processing apparatus which captures a captured image of an imaging target obtained by an imaging apparatus and generates a corrected image in which distortion of the captured image is corrected. A captured image obtained by capturing an image of the imaging target placed on the mounting surface, and a reference image obtained by capturing the pattern with the same orientation of the imaging device when the imaging target is imaged. Means for pre-storing position information of a plurality of feature points on the pattern, means for receiving position information of a corresponding point on the reference image corresponding to the feature point, Means for determining a geometric relationship between the placement surface and a captured image when the pattern is captured, based on the positional information of the point and the positional information of the corresponding point, based on the determined geometric relationship Distortion of the captured image Characterized in that it comprises a means for correcting.

  A computer program according to a tenth aspect is a computer program having a step of causing a computer to generate a corrected image in which a distortion of a received image is corrected, wherein the image is mounted on a mounting surface on which a predetermined pattern is formed. A captured image obtained by capturing the placed imaging target together with the pattern, and causing the computer to receive position information of a corresponding point on the captured image corresponding to a feature point on the pattern; and Based on the position information of the corresponding point and the position information of the feature point stored in advance, a step of determining a geometric relationship between the mounting surface and the captured image when the imaging target is imaged. And causing the computer to correct the distortion of the captured image based on the determined geometric relationship.

  A computer program according to an eleventh aspect of the present invention is a computer program having a step of causing a computer to generate a corrected image in which a distortion of a received image is corrected, wherein the image is mounted on a mounting surface on which a predetermined pattern is formed. A captured image obtained by capturing the placed imaging target by an imaging device, and a reference image of the pattern captured with the same orientation of the imaging device when capturing the captured image. Receiving the position information of the corresponding point on the reference image corresponding to the characteristic point on the pattern; and causing a computer to perform the processing based on the position information of the corresponding point and the position information of the characteristic point stored in advance. Causing the computer to determine a geometric relationship between the mounting surface and the captured image of the pattern. And having a step of correcting distortion of the captured image on the basis of histological relationship.

  In the first invention, the third invention, the eighth invention, and the tenth invention, a captured image of an imaging object mounted on a mounting surface on which a predetermined pattern is formed is captured, and the feature on the pattern is obtained. Position information of the corresponding point on the captured image corresponding to the point is received, and based on the position information of the feature point and the position information of the corresponding point, the captured image and the mounting surface when the imaging target is imaged are captured. The geometrical relationship between them is determined, and the distortion of the captured image is corrected based on the determined geometrical relationship. Therefore, when distortion is generated in the captured image, it is easy to correct based on the distortion of the pattern shown in the captured image, and the imaging target is placed on the mounting surface on which the pattern is formed. Since the image is captured in the state, no extra image is applied to the image of the imaging target in the generated corrected image.

  In the second invention, the fourth invention, the ninth invention, and the eleventh invention, the image of the imaging object mounted on the mounting surface on which the predetermined pattern is formed, and the posture of the imaging device are changed. A reference image obtained by capturing the pattern in the same state is taken, and position information of a corresponding point on a captured image corresponding to a feature point on the pattern is received, and the position information of the feature point and the position information of the corresponding point are received. The geometric relationship between the captured image when the imaging object is captured and the mounting surface is determined based on the geometrical relationship, and the distortion of the captured image is corrected based on the determined geometric relationship. Therefore, when distortion is generated in the captured image, it is easy to correct based on the distortion of the pattern shown in the captured image, and the imaging target is placed on the mounting surface on which the pattern is formed. Since imaging is performed in the state, no extra image is applied to the image of the imaging target in the generated corrected image, and the image area can be effectively used.

  In the fifth invention, the geometric relationship between the mounting surface and the captured image is determined by projective transformation between the feature points on the pattern and the corresponding points on the captured image. Therefore, only by accepting the position information of the corresponding points in the two images, a geometrical relationship established between the two images can be determined, and it becomes easy to generate a corrected image in which the distortion of the captured image has been removed.

  In the sixth aspect, the color information of the point on the corrected image to be generated is such that the point on the corrected image is converted according to a geometric relationship, and the color information of the converted point is obtained from the captured image. Therefore, color information of all points constituting the corrected image can be determined, and the corrected image is easily generated.

  In the seventh aspect, since a grid-like image is used as a pattern formed on the mounting surface, feature points on the pattern can be easily defined, and the pattern on the mounting surface and the captured image Is easily associated with the image of the pattern shown in FIG.

  According to the first, third, eighth, and tenth aspects of the present invention, a captured image of an imaging target mounted on a mounting surface on which a predetermined pattern is formed is acquired and stored in advance. Based on the position information of the corresponding points on the captured image corresponding to the position information of the plurality of feature points on the placed pattern, based on the position information of the feature points on the pattern and the position information of the corresponding points on the captured image, A geometric relationship between a pattern when the imaging object is imaged and the captured image is determined, and distortion of the captured image is corrected based on the determined geometric relationship. Therefore, when the captured image is distorted, the correction can be easily performed based on the distortion of the pattern shown in the captured image, and the object to be imaged is placed on the mounting surface on which the pattern is formed. Since the image is taken while the camera is placed, a corrected image can be generated without applying an extra image to the image of the object to be imaged.

  According to the second, fourth, ninth, and eleventh aspects of the present invention, the captured image of the imaging target mounted on the mounting surface on which the predetermined pattern is formed is the same as the attitude of the imaging device. In this state, a reference image obtained by capturing the pattern is taken in, and position information of a corresponding point on a captured image corresponding to a feature point on the pattern is received, and the position information of the feature point and the position information of the corresponding point are received. Based on the geometric relationship between the captured image and the mounting surface when the imaging target is captured, distortion of the captured image is corrected based on the determined geometric relationship. Therefore, when distortion is generated in the captured image, it is easy to correct based on the distortion of the pattern shown in the captured image, and the imaging target is placed on the mounting surface on which the pattern is formed. Since the image is captured in the state, the corrected image can be generated without applying an extra image to the image of the imaging target, and the image area can be effectively used.

  In the case of the fifth aspect, the geometrical relationship between the pattern and the captured image is determined by projective transformation between the feature points on the pattern and the corresponding points on the captured image. Therefore, only by accepting the position information of the corresponding points in the two images, the geometric relationship that holds between the two images can be determined, and a corrected image in which the distortion of the captured image has been removed can be easily generated.

  In the case of the sixth aspect, the color information of the point on the corrected image to be generated is such that the point on the corrected image is converted according to the geometrical relationship, and the color information of the converted point is obtained from the captured image. The color information of all points constituting the corrected image can be determined, and the corrected image can be easily generated.

  In the case of the seventh aspect, since a grid-like image is used as a pattern, feature points on the pattern can be easily defined, and association with a captured image can be easily performed.

  Hereinafter, the present invention will be specifically described with reference to the drawings showing the embodiments.

Embodiment 1 FIG.
FIG. 1 is a schematic diagram illustrating a configuration of an image processing system according to the present embodiment. In the drawing, T is an imaging target, and the imaging target T is imaged by the imaging device 20 in a state where the imaging target T is placed on the sheet S, and a captured image obtained by the imaging is read into the image processing device 10. The imaging target T is a sheet-like object such as a pattern, a manuscript, a map, or the like when fabricating clothes or the like. The sheet S is formed in a rectangular or square shape from a material such as synthetic resin, metal, wood, paper, or the like, and has a grid pattern formed by grid lines arranged at equal intervals on the surface. The imaging device 20 is a digital camera that can generate digital image data. The image processing device 10 is configured by a personal computer or the like, acquires image data generated by the imaging device 20, and performs image processing. Here, the image data refers to data composed of a set of pixels each having color information.

  When the captured image obtained by the imaging device 20 is used as an image for clothing design or an image for reading character information in the image processing device 10, the captured image captured directly in front of the imaging target T is It is desirable to be able to obtain. For this purpose, it is conceivable to fix the imaging device 20 directly in front of the imaging target T for imaging, but there is a problem that it lacks convenience. Therefore, in the present embodiment, the distortion of the captured image due to the displacement of the position and orientation of the imaging device 20 when the image is captured is removed by image processing as described later, and a front image as if captured from the front is synthesized.

  FIG. 2 is a schematic diagram illustrating a pattern formed on the sheet S. On the surface of the sheet S, a grid-like pattern composed of grid lines arranged at equal intervals as described above is drawn. In the example shown in FIG. 2, a pattern formed by a square grid of 6 rows and 8 columns (six in the vertical direction and eight in the horizontal direction) is drawn, and the image processing apparatus 10 has a geometric shape of this pattern. Is stored in advance. The information on the geometric shape includes position coordinates of each grid point, measured values of the distance between the grid points, and the like. Here, the position coordinates of each grid point are (0, 0) the position coordinates of the lower left grid point, (1, 0) the position coordinates of the lower right grid point, and (1, 0) the position coordinates of the upper right grid point. 1,1), and a relative coordinate system in which the position coordinates of the upper left grid point are (0,1). The distance between the grids is, for example, the distance between the position coordinates (0,0) and the position coordinates (1,0), and the distance between the position coordinates (0,0) and the position coordinates (0,1). The distance between them is measured in advance and stored in the image processing device 10.

  In the present embodiment, a configuration is used in which a square grid of 6 rows and 8 columns is used as a pattern, but the array of the grid is not necessarily limited to 6 rows and 8 columns, and at least four or more grid points together with the imaging object T What is necessary is just a structure which can be imaged. Further, the shape of each grid is not limited to a square, and it is also possible to use a grid image having a shape such as a rectangle or a triangle. Further, a mode may be used in which only grid points are drawn and used without drawing grid lines.

FIG. 3 is a schematic diagram illustrating a relationship between a captured image read by the image processing apparatus 10 and a pattern. When the above-described sheet S and the imaging target T are imaged from an appropriate position and posture using the imaging device 20, the distortion as illustrated in FIG. A captured image 20A in which (trapezoidal distortion) has occurred is obtained. For the present embodiment corrects such distortion, Figure 3, wherein points P1 to P4 are brought on pattern 20B shown (b), the on the captured image 20A corresponding to these feature points P ₁ to P ₄ The corresponding points Q _{1 to} Q ₄ are determined by associating with the feature points of the above, and a geometrical relationship established between them is obtained. Specifically, a projective transformation matrix that moves the position coordinates of the feature points P _{1 to} P _{4 to} the position coordinates of the corresponding points Q _{1 to} Q ₄ is obtained. Then, in order to obtain a corrected image (front image) in which the distortion has been corrected, the position coordinates of each pixel on the pattern 20B are converted by the projection conversion matrix, and color information of the conversion destination is obtained, thereby obtaining each pixel of the front image. Determine the color information of That is, an image from which distortion has been removed is generated by defining color information for all pixels on the pattern 20B.

  FIG. 4 is a block diagram illustrating an internal configuration of the image processing apparatus 10. The image processing apparatus 10 includes a CPU 11. Various hardware such as a ROM 13, a RAM 14, an operation unit 15, a display unit 16, an image input unit 17, an auxiliary storage unit 18, and a storage unit 19 are connected to the CPU 11 via a bus 12. Connected. The CPU 11 controls the above-described hardware according to various control programs stored in the ROM 13, and performs image processing on a captured image received from outside to correct distortion. Note that the image processing need not necessarily be performed by the CPU 11, but an ASIC (Application Specified Integrated Circiut) dedicated to image processing is provided, and the ASIC dedicated to image processing performs image processing by giving an instruction from the CPU 11. There may be. The RAM 14 is configured by an SRAM, a flash memory, or the like, and stores data and the like generated when the control program stored in the ROM 13 is executed.

  The operation unit 15 includes input devices such as a keyboard, a mouse, and a tablet, and receives an instruction, a selection operation, and the like when performing image processing. The display unit 16 includes a display device such as a CRT display or a liquid crystal display. The display unit 16 displays various information input through the operation unit 15 and displays a captured image input from outside, an image being processed, and the like. Is done.

  The image input unit 17 includes an input device for inputting image data generated by the imaging device 20. As the input device, a connection port conforming to the USB (Universal Serial Bus) standard, a reading device for reading image data stored in a portable semiconductor memory, and the like can be used. When the imaging device 20 has a USB interface and can send image data to the outside via a communication cable, the communication cable is connected to the above-described connection port to receive the image data by communication. When the imaging device 20 does not have a USB interface and the image data is transferred by a portable semiconductor memory, the above-described reading device is used. Note that this reading device may be built in the image processing device 10 or may be externally attached. Image data input from the image input unit 17 is stored in a storage unit 19 including an HDD device or the like.

  The auxiliary storage unit 18 includes an FD drive, a CD-ROM drive, and the like that read a computer program from a recording medium M such as an FD or a CD-ROM that stores the computer program of the present invention. It is memorized. When the CPU 11 executes the computer program, the computer program stored in the storage unit 19 is temporarily expanded on the RAM 14, and the CPU 11 decodes an instruction code included in the computer program and interprets the computer program as a series of programs. Run a computer program.

  FIG. 5 is a flowchart illustrating a processing procedure of image processing by the image processing system according to the present embodiment. First, the image processing apparatus 10 reads a captured image obtained by capturing an image using the imaging apparatus 20 (Step S1). The reading of the captured image may be a mode of reading a captured image input through the image input unit 17 or a mode of reading a captured image stored in the storage unit 19 in advance. The read image is temporarily stored in the RAM 14.

  Next, calibration of the imaging device 20 is performed (Step S2), and distortion caused by internal parameters of the imaging device 20 is removed. Generally, various distortions such as circular distortion and barrel distortion are generated in an image obtained by an imaging device under the influence of a lens system of the imaging device. Therefore, in the present embodiment, after removing these distortions, the distortion of the captured image due to the position and orientation at the time of imaging is removed. A known method can be used to calibrate the imaging device 20. For example, a known pattern is imaged, and based on the image and the pattern obtained by the imaging, the internal parameters (focal length, position coordinates of the principal point, and the like) of the imaging device 20 and parameters defining the distortion amount are calculated in advance. I do. Then, at the time of calibration, calibration is performed using these parameters obtained in advance.

  Note that when imaging is performed using the imaging device 20 that has been calibrated in advance, this step can be omitted.

  Next, the image processing apparatus 10 receives the association between the feature points of the captured image and the feature points of the pattern on the sheet S (Step S3). Specifically, the captured image (20A) and the pattern (20B) as shown in FIG. 3 are displayed on the display unit 16 of the image processing apparatus 10, and the feature points corresponding to each other in each image (20A, 20B) are determined. The user is caused to select by a mouse or the like provided in the operation unit 15.

  Next, the CPU 11 of the image processing apparatus 10 obtains an initial value of each element of the projective transformation matrix (Step S4). The initial value of each element of the projective transformation matrix can be obtained as follows.

Position coordinates (u _i, v _i) of feature points of the captured image, the position coordinates (x _i, y _i) of the feature points on pattern, and using the projection transformation matrix H is given by the following formula it can.

Here, H ₁₁ , H ₁₂ ,..., H ₃₃ on the right side represent each element of the projective transformation matrix H. Further, in the equation (1), the scale factor is not considered. When expanded with respect to u _i and v _i , they can be expressed as Expression (2) and Expression (3).

  Furthermore, when each element of the projective transformation matrix H is summarized, it can be represented by the following determinant.

Here, when a matrix of 2n rows × 9 columns on the left side is represented by A, and a vector represented by each element H ₁₁ , H ₁₂ ,..., H ₃₃ of the projective transformation matrix H is represented by h, Equation (4) Is

It becomes. At this time, the vector h can be obtained as an eigenvector corresponding to the minimum eigenvalue of A ^T A.

Next, the CPU 11 of the image processing apparatus 10 sets an evaluation function for optimizing the elements H ₁₁ , H ₁₂ ,..., H ₃₃ of the projection transformation matrix H (Step S5). Here, setting the square sum of the distance between the destination coordinates according to the projective transformation matrix obtained positional coordinates (u _i, v _i) of the points on the captured image and in step S4 as the evaluation function (Equation ( 6)), the elements H ₁₁ , H ₁₂ ,..., H ₃₃ of the projective transformation matrix are optimized so that the evaluation function is minimized (step S6).

  Next, the CPU 11 of the image processing device 10 corrects the distortion using the projection transformation matrix H obtained by the optimization, and performs a front image combining process (Step S7).

  FIG. 6 is a schematic diagram illustrating the process of synthesizing a front image. Since the projection transformation matrix H optimized in step S6 of the above-described flowchart is obtained, the position coordinates of each point on the captured image are transformed by the inverse matrix of the projection transformation matrix H to obtain a front image (corrected image) having no distortion. ) Is obtained. However, since the captured image is originally distorted, when compared with the pattern, there are a region with insufficient pixels and a region with excessive pixels. Accordingly, when the captured image is transformed using the inverse matrix of the projection transformation matrix H, a complete front image without any gap may not be obtained.

Thus, in the present embodiment, the original pattern (20B) is used as a starting point, and the position coordinates (a _i , b _j ) of an arbitrary pixel on the pattern are used instead of using the captured image (20A) as a starting point. The transformation is performed by the projection transformation matrix H, the position coordinates (α _i , β _j ) of the corresponding point on the captured image are obtained, the color information of the converted point is obtained, and the obtained color information is generated on the front image to be generated. Is stored as the color information of the point.

FIG. 7 is a flowchart for explaining the processing procedure of the front image combining processing routine. First, the CPU 11 of the image processing apparatus 10 sets the indices i and j to 0 (step S11), and converts the position coordinates (a _i , b _j ) of the point on the restoration destination image (front image) into the projection transformation matrix H. (Step S12). Since the converted point moves to a point on the captured image, color information corresponding to the position coordinates (α _i , β _j ) of the converted point is obtained (step S13). Then, the acquired color information is stored as the color information of the position coordinates (a _i , b _j ) of the point of the image of the restoration destination (step S14).

  Next, the CPU 11 determines whether or not the index i is equal to or greater than X (step S15). Here, X represents the total number of pixels in the horizontal direction of the front image to be generated. If it is determined that i is not greater than or equal to X (S15: NO), the index i is increased by one (step S16), and the process returns to step S12.

  When it is determined that the index i is greater than or equal to X (S15: YES), the value of the index i is set to 0 (step S17), and it is determined whether or not the index j is greater than or equal to Y (step S18). Here, Y represents the total number of pixels in the vertical direction of the front image to be generated. When it is determined that j is not greater than or equal to Y (S18: NO), the index j is increased by one (step S19), the process returns to step S12, and when it is determined that j is greater than or equal to Y (S18: YES). Then, this routine ends.

  As described above, in the present embodiment, since the restoration destination (front image) is set as the starting point, the color information of each pixel is sequentially determined, so that a complete front image without gaps can be obtained.

  In the present embodiment, the imaging target T placed on the sheet S is configured to be imaged. However, the placement of the imaging target T is not necessarily limited to a sheet, and may be a table-like table or the like. Alternatively, a configuration in which a pattern is formed on a floor surface, a wall surface, or the like and used may be used.

  In the present embodiment, a digital camera is used as the imaging device 20, but a silver halide camera can be used instead of the digital camera. Since a silver halide camera forms a captured image on a film or a photographic paper, it is necessary to convert the image into digital image data for use inside the image processing apparatus 10. For example, a film scanner, an image scanner, or the like is used. be able to.

Embodiment 2 FIG.
In the first embodiment, an image is captured by the imaging device 20 in a state where the imaging target T is placed on the sheet S, the captured image acquired by the imaging device 20 is taken into the image processing device 10, and the image is drawn on the sheet S. The distortion of the captured image is removed based on the positional information of the lattice points appearing around the imaging target T among the lattice points. That is, in the first embodiment, since distortion can be removed by using one captured image obtained by capturing an image from an arbitrary position and posture, the advantage that the degree of freedom in capturing the image of the imaging target T is large. Have. However, when the occupied area of the imaging target T placed on the sheet S is large and at least four or more lattice points do not appear around the imaging target T, the distortion can be removed by the above-described method. It will not be possible.

  In the present embodiment, as a method for coping with a case where a sheet S sufficiently larger than the imaging target T cannot be prepared, a captured image (hereinafter, referred to as a reference image) of the sheet S alone and a A description will be given of a method of removing distortion of a captured image using a captured image obtained by mounting the imaging target T on the sheet S while maintaining the same position and orientation. Note that the configurations of the image processing system and the image processing apparatus 10 according to the present embodiment are completely the same as those in the first embodiment, and thus description thereof will be omitted.

  FIG. 8 is an explanatory diagram illustrating an image to be captured by the image processing apparatus 10. In the present embodiment, first, the imaging device 20 captures an image of the sheet S on which a lattice-like pattern composed of lattice lines arranged at equal intervals is drawn. At this time, the image is taken after the position and orientation of the imaging device 20 are fixed using a fixture such as a tripod. The captured image is taken into the image processing apparatus 10 as the reference image 21A and stored in the storage unit 19 in the image processing apparatus 10 (see FIG. 8A). Next, the image capturing target T is placed on the sheet S, and the image capturing apparatus 20 captures an image while maintaining the same position and orientation of the image capturing apparatus 20. The captured image 21B obtained by capturing the image is taken into the image processing apparatus 10 and stored in the storage unit 19 in the image processing apparatus 10 (see FIG. 8B).

FIG. 9 is an explanatory diagram illustrating a procedure of image processing performed by the image processing apparatus 10. In the present embodiment, first, a geometric relationship that holds between the pattern on the sheet S and the reference image 21A is obtained from the reference image 21A obtained by imaging only the sheet S, and then the imaging target T is placed on the sheet S. The front image is synthesized by applying the geometric relationship to the captured image 21 </ b> B that is mounted and captured. The geometric relationship between the pattern on the sheet S and the reference image 21A can be obtained in the same manner as in the first embodiment. That is, established between the position coordinates (x _i, y _i) of the lattice point that indicates the feature points on pattern position coordinates of the corresponding point corresponding to the grid points on the reference image 21A (u _i, v _i) When the relational expression is expressed as in equation (1) using the projective transformation matrix H, the position coordinates (x _i , y _i ) of the grid point and the position coordinates (u _i , v _i ) can be expressed by Expression (6), and each element of the projective transformation matrix H can be obtained by optimizing the evaluation function. Further, since the reference image 21A and the captured image 21B are images captured with the same position and orientation of the imaging device 20, the relationship that is established between the reference image 21A and the pattern on the sheet S is as follows. The present invention can be applied between the image 21B and the image obtained by projecting the captured image 21B on the sheet S, and the front image can be synthesized by converting the captured image 21B based on the relationship.

  FIG. 10 is a flowchart illustrating a processing procedure of image processing by the image processing system according to the present embodiment. First, the image processing apparatus 10 reads a reference image and a captured image obtained by imaging with the imaging device 20 (Step S21). The reading of the captured image may be a mode of reading a captured image input through the image input unit 17 or a mode of reading a captured image stored in the storage unit 19 in advance. The read image is temporarily stored in the RAM 14.

  Next, calibration of the imaging device 20 is performed (Step S22), and distortion caused by internal parameters of the imaging device 20 is removed. Note that when imaging is performed using the imaging device 20 that has been calibrated in advance, this step can be omitted.

  Next, the image processing apparatus 10 receives the association between the feature points of the reference image and the feature points of the pattern (Step S23). Specifically, as shown in FIG. 9A, the reference image (21A) and the image of the pattern are displayed on the display unit 16 of the image processing apparatus 10, and the corresponding feature points of each image are displayed on the operation unit 15. Is made to be selected by the user using a mouse or the like provided in the device.

Next, the CPU 11 of the image processing device 10 obtains the initial value of each element of the projective transformation matrix (Step S24). Position coordinates of the feature points on the reference image (u _i, v _i) is expressed as Equation (1) using the position coordinates (x _i, y _i), and the projective transformation matrix H of feature points on pattern When each element of the projective transformation matrix H is summarized for n sets of feature points, it can be expressed as shown in Expression (4). That is, the initial value of each element of the projective transformation matrix H can be obtained as an eigenvector corresponding to the minimum eigenvalue of A ^T A when a matrix of 2n rows × 9 columns on the left side of Expression (4) is represented by A. .

Next, the CPU 11 of the image processing apparatus 10 sets an evaluation function to optimize each element of the projection transformation matrix H (Step S25). Here, the set position coordinates (u _i, v _i) of the points on the reference image square sum of the distance between the destination coordinates according to the projective transformation matrix determined in Step S24 as the evaluation function (Equation ( 6)), each element of the projective transformation matrix is optimized such that the evaluation function is minimized (step S26). Next, the CPU 11 of the image processing apparatus 10 corrects the distortion using the projective transformation matrix H obtained by the optimization, and performs a front image combining process (Step S27). The same method as in the first embodiment can be used for the synthesis processing of the front image. That is, by converting each pixel of the front image to be generated according to the calculated projective transformation matrix, and acquiring the color information of the pixel on the captured image 21B corresponding to the conversion destination, the color information of each pixel is determined and the front image is determined. Can be generated.

  In the present embodiment, after the sheet S is imaged alone, the imaging target T is placed on the sheet S for imaging. First, the imaging target T is placed on the sheet S. It is needless to say that an image may be taken in a state where it is placed, and thereafter, the image taking object T may be removed to take an image. Further, in the present embodiment, if the position and orientation of the imaging device 20 are maintained when the imaging target T is imaged, the captured image 21B is captured using the projection transformation matrix H calculated based on the reference image 21A. Since the distortion can be removed, it is not always necessary to take an image of the imaging target T with the imaging target T placed on the sheet S when imaging the imaging target T. That is, since it is possible to take an image with a plain background, there is also an advantage that an extra image such as a grid line is not applied to the front image from which distortion has been removed.

FIG. 1 is a schematic diagram illustrating a configuration of an image processing system according to an embodiment. FIG. 3 is a schematic diagram illustrating a pattern formed on a sheet. FIG. 2 is a schematic diagram illustrating a relationship between a captured image read by an image processing apparatus and a pattern. FIG. 2 is a block diagram illustrating an internal configuration of the image processing apparatus. 5 is a flowchart illustrating a processing procedure of image processing by the image processing system according to the present embodiment. FIG. 9 is a schematic diagram illustrating a process of synthesizing a front image. It is a flowchart explaining the processing procedure of the synthesis processing routine of a front image. FIG. 4 is an explanatory diagram illustrating an image to be captured by the image processing apparatus. FIG. 9 is an explanatory diagram illustrating a procedure of image processing performed by the image processing apparatus. 5 is a flowchart illustrating a processing procedure of image processing by the image processing system according to the present embodiment.

Explanation of reference numerals

10 Image processing device 11 CPU
12 bus 13 ROM
14 RAM
Reference Signs List 15 operation unit 16 display unit 17 image input unit 18 auxiliary storage unit 19 storage unit 20 imaging device S sheet T imaging target

Claims (11)

An image processing method for capturing a captured image obtained by capturing an imaging target object into an image processing apparatus and generating a corrected image in which distortion of the captured image captured is corrected,
The captured image is an image obtained by capturing an image of an imaging target placed on a mounting surface on which a predetermined pattern is formed, together with the pattern, and stores position information of a plurality of feature points on the pattern. It is stored in advance and receives position information of the corresponding point on the captured image corresponding to the feature point, and based on the position information of the feature point and the position information of the corresponding point, the placement surface and the imaging target An image processing method comprising: determining a geometric relationship between a captured image of an object and a captured image; and correcting distortion of the captured image based on the determined geometric relationship. An image processing method of capturing a captured image obtained by capturing an imaging target object by an imaging device into an image processing device and generating a corrected image in which the captured captured image is corrected for distortion.
The captured image is an image obtained by capturing an image of an imaging target placed on a mounting surface on which a predetermined pattern is formed, and indicates a posture of the imaging device when capturing the imaging target. A reference image obtained by capturing the pattern in the same manner is taken into the image processing apparatus, and position information of a plurality of feature points on the pattern is stored in advance, and a corresponding point on the reference image corresponding to the feature point is stored. Position information is received, and based on the position information of the feature points and the position information of the corresponding points, a geometric relationship between the placement surface and the captured image when the pattern is captured is determined, and the determined geometry is determined. An image processing method for correcting distortion of the captured image based on a logical relationship. A mounting portion having a mounting surface for mounting an imaging target, and an imaging device that images the imaging target mounted on the mounting surface described above, and captures an image captured by the imaging device. An image processing system comprising: an image processing device that generates a corrected image in which the captured image is corrected for distortion.
The mounting surface has a predetermined pattern formed thereon, and the image processing apparatus is configured to capture a captured image obtained by imaging the imaging target mounted on the mounting surface together with the pattern. Means for storing in advance position information of a plurality of feature points on the pattern; means for receiving position information of a corresponding point on the captured image corresponding to the feature point; Means for determining a geometric relationship between the placement surface and the captured image when the imaging target is imaged, based on the position information of the points, and Means for correcting distortion. A mounting portion having a mounting surface for mounting an imaging target, and an imaging device that images the imaging target mounted on the mounting surface described above, and captures an image captured by the imaging device. An image processing system comprising: an image processing device that generates a corrected image in which the captured image is corrected for distortion.
The mounting surface has a predetermined pattern formed thereon, and the image processing apparatus captures an image of the imaging target placed on the mounting surface by the imaging device, and captures the imaging target. Means for pre-storing positional information of a plurality of feature points on the pattern, wherein the orientation of the imaging device at the time of the capturing is the same as the reference image obtained by capturing the pattern, Means for receiving the position information of the corresponding point on the corresponding reference image, based on the position information of the feature point and the position information of the corresponding point, the image of the mounting surface and the image captured when the pattern is imaged. An image processing system comprising: means for determining a geometric relationship between the images; and means for correcting distortion of the captured image based on the determined geometric relationship.   The image processing system according to claim 3, wherein the geometric relationship is represented by projective transformation between the feature point and the corresponding point.   The image processing apparatus further includes: a unit configured to convert a point on the corrected image to be generated according to the geometric relationship; and a unit configured to obtain color information of the converted point from the captured image. The image processing system according to claim 3, wherein a corrected image is generated based on the corrected image.   7. The image processing system according to claim 3, wherein the pattern image is a lattice image. In an image processing apparatus that captures a captured image of an imaging target object and generates a corrected image in which distortion of the captured image is corrected,
The apparatus is configured to capture a captured image obtained by capturing an image of an imaging target mounted on a mounting surface on which a predetermined pattern is formed, together with the pattern, and obtain position information of a plurality of feature points on the pattern. Means for storing in advance, means for receiving position information of the corresponding point on the captured image corresponding to the feature point, based on the position information of the feature point and the position information of the corresponding point, Means for determining a geometric relationship between the image and the captured image when the image of the object is captured, and means for correcting distortion of the captured image based on the determined geometric relationship. Image processing device. In an image processing device that captures a captured image of an imaging target obtained by an imaging device and generates a corrected image in which distortion of the captured image is corrected,
An image obtained by imaging an imaging target placed on a mounting surface on which a predetermined pattern is formed is the same as the captured image, and the pattern of the imaging device is the same when the imaging device is imaged. Means for pre-storing position information of a plurality of feature points on the pattern, and receiving position information of a corresponding point on the reference image corresponding to the feature point. Means, based on the position information of the feature points and the position information of the corresponding points, means for determining a geometric relationship between the placement surface and the captured image when the pattern is captured, Means for correcting distortion of the captured image based on a logical relationship. A computer program having a step of causing a computer to generate a corrected image in which a distortion of a received image has been corrected,
The image is a captured image obtained by capturing an image of an imaging target mounted on a mounting surface on which a predetermined pattern has been formed, together with the pattern, and a computer corresponding to feature points on the pattern. Receiving the position information of the corresponding point on the captured image, and causing the computer to image the placement surface and the imaging target based on the position information of the corresponding point and the position information of the feature point stored in advance. A computer program, comprising the steps of: determining a geometric relationship between the captured image and the captured image; and causing a computer to correct distortion of the captured image based on the determined geometric relationship. . A computer program having a step of causing a computer to generate a corrected image in which a distortion of a received image has been corrected,
The image is a captured image obtained by capturing an imaging target placed on a mounting surface on which a predetermined pattern is formed by an imaging device, and a posture of the imaging device when the captured image is captured. Is a reference image obtained by imaging the pattern with the same, and causing a computer to receive position information of a corresponding point on the reference image corresponding to a feature point on the pattern; and Causing the computer to determine a geometric relationship between the placement surface and the captured image when the pattern is captured based on the information and the position information of the feature point stored in advance. Correcting the distortion of the captured image based on a logical relationship.

Images