JP4542821B2 - Image processing method, image processing apparatus, and image processing program - Google Patents

Description

  The present invention relates to an image processing method, an image processing apparatus, and an image processing program for correcting distortion caused by the imaging optical system and the image sensor of image data captured using the imaging optical system and the image sensor.

  2. Description of the Related Art Conventionally, a technique for measuring the position and coordinates of a subject from images photographed through a plurality of photographing optical systems having different optical axes has been known (for example, Patent Document 1 below). This type of shooting method is called stereo shooting or stereoscopic shooting, and the camera used is sometimes called a stereo camera or a stereoscopic camera.

  In addition, a technique (for example, Patent Document 2 below) is known in which images with different shooting distances are combined to obtain an image in which all subjects from a distant view to a close view are in focus. In this type of technology, for example, each of the photographing optical systems is located at different positions so as to have a depth of field at a short distance (N: Near), a medium distance (M: Middle), and a far distance (F: Far). Image sensors for short distance, medium distance, and long distance are arranged, and each image of N (short distance), M (medium distance), and F (far distance) obtained from each image sensor is analyzed. An image with higher sharpness can be obtained by extracting and synthesizing an appropriate part from each image based on conditions such as the sharpness of the image. In the following, for convenience, a shooting technique / image processing technique for increasing the sharpness of an image by synthesizing images at a short distance (N: Near), a medium distance (M: Middle), and a long distance (F: Far) will be referred to as NMF / This is called NMF image processing.

Of course, the NMF shooting / image processing can be applied to the above-described stereo (stereoscopic) shooting technique. In this case, the position of the subject is more accurately based on the image whose sharpness is improved by the NMF image processing. And coordinates can be measured.
JP 2003-281521 A JP 11-37720 A

  In the above known technology, an element such as a CCD is used as an image pickup element. However, there is a considerable amount of manufacturing distortion in this type of element, and the position of a pixel actually picked up is a theoretical position. It is off. Furthermore, various optical aberrations also exist in the imaging lens, and this aberration affects the photographed image. For example, in the coordinate measurement by the above-described stereo (stereoscopic) photographing, the measurement accuracy decreases, and In the NMF image composition processing, and in the above technique, a plurality of different image sensors for at least short distance, medium distance, and long distance are used, but each image sensor has different distortion and optical system aberration. As a result, the image is reflected in the photographed image, resulting in problems such as failure to obtain good image continuity or obtain the desired sharpness.

  The object of the present invention is to solve the above-mentioned problems, correct errors caused by distortion of the image sensor and the photographing optical system, and more accurately when applying stereo (stereoscopic) photographing and NMF photographing / image processing. An object image is obtained, and based on this, the object position and coordinates can be accurately measured.

The present invention relates to an orthogonal vertical line in an image processing method, an image processing apparatus, and an image processing program for correcting distortion caused by the imaging optical system and the imaging device of image data captured using the imaging optical system and the imaging device. A grid image composed of a horizontal line and a horizontal line is photographed by the photographing optical system and the image sensor, an intersection point of the block composed of a vertical line and a horizontal line of the grid image is detected , and the coordinates of the intersection point of the block are determined as the grid of the block. The line is corrected so that it becomes a vertical and horizontal straight line, that is, the block is rectangular, and the coordinates of each intersection defining the block obtained by the grid intersection correction and the block in the grid image are defined. Tomo performing calibration process of calculating a correction function for correcting the actual photographed image based on the coordinates of the intersection point In the detection of the block intersection point, each coordinate of the grid image is sequentially scanned to detect pixels having low luminance constituting the grid line, and an average predetermined threshold percentage of a plurality of pixels having the same coordinate in a plurality of past scanning lines. The evaluation function P (x) = p (x−1) p indicating the continuity of the number of the candidate pixels at the same coordinates in the past plural scanning lines at the coordinates of each pixel, with the pixels lower than The value of (x) + p (x) p (x + 1) (p (i): the number of candidate pixels of the grid line at the coordinate i) is obtained, and when the value P (x) is equal to or greater than the threshold, the coordinates of the pixel are determined. Obtained as the range where the grid line exists, and further detects the intersection of the grid lines obtained by detecting the center of the range where the grid line exists as the intersection of the blocks defined by the vertical and horizontal lines of the grid image It was adopted configuration.

According to the above configuration, calibration of the grid image is performed, and the coordinates of the intersection defining the block are expressed in units of a block composed of vertical and horizontal lines of the grid image. The coordinates of each intersection defining the block obtained by the grid intersection correction, and the coordinates of each intersection defining the block in the grid image are corrected so that it becomes a vertical and horizontal straight line, that is, the block is rectangular. Since a correction function for correcting an actual captured image is calculated based on the above, in actual shooting, errors caused by distortion of the image sensor and the shooting optical system are corrected, and particularly stereo (stereoscopic) shooting and When applying NMF shooting / image processing, a more accurate image of the subject can be obtained, and the position and coordinates of the subject can be accurately measured based on this. It is possible, there is an excellent effect that.

  Hereinafter, for convenience, an imaging system that performs stereo (stereoscopic) imaging and NMF imaging / image processing is illustrated. However, the basic part of the image processing technique of the present invention is not necessarily applied when performing stereo (stereoscopic) shooting and NMF shooting / image processing. For example, the basic portion of the image processing technique of the present invention can be applied to a photographing system that uses a single lens and does not perform NMF image synthesis.

<Hardware configuration>
FIG. 1 shows the overall configuration of an image processing system employing the present invention. The system in FIG. 1 includes a stereo camera 20 and a control system 40.

  The stereo camera 20 is composed of lenses 21 and 22 having optical axes arranged in parallel at a predetermined distance and CCDs disposed behind the lenses 21 and 22 as a photographing optical system for performing stereoscopic photography. Imaging devices 201n, 201m, 201f, and 202n, 202m, 202f.

  The imaging elements 201n (202n), 201m (202m), and 201f (202f) each have a lens 21 so that images of subjects of N (short distance), M (medium distance), and F (far distance) can be captured satisfactorily. They are arranged at different positions on the optical axis (22). FIG. 1 shows a configuration in which the half mirror 204 is arranged to guide the imaging light in the direction of the image sensors 201n (202n) and 201m (202m) for simplification. The configuration of such an optical path dividing unit is as follows. For example, a prism having a reflection / transmission surface that guides imaging light in the direction of the imaging elements 201n (202n), 201m (202m), and 201f (202f) can be used.

  In addition, the stereo camera 20 includes a CPU 203 that controls the focusing operation and exposure control of the lenses 21 and 22, the shooting timing of each image sensor, image output processing, and the like. As the lenses 21 and 22, lenses having the same focal length and aperture ratio are usually used (when the left and right of a stereo image are photographed under the same conditions). The captured image data of the CPU 203 and the image sensors 201n, 201m, 201f, and 202n, 202m, 202f are transmitted to the control system 40 via a predetermined interface (USB or the like).

  The control system 40 can be composed of almost common hardware of an existing PC. That is, the control system 40 includes a CPU 51 for controlling the entire apparatus and the image processing of the present invention, a ROM 52 and a RAM 53 for storing programs and data, and an HDD 54 for storing control programs and photographing data described later. , A display (CRT, LCD, etc.) 41 for a user interface, a keyboard 42, and an interface 55 such as a USB for connecting to the stereo camera 20.

  Even in the above-described configuration as described above, each of the image pickup devices 201n, 201m, 201f, and 202n, 202m, 202f has different distortions depending on the individual depending on the manufacturing conditions and the like. Even if a precision element is used, some aberrations remain in the lenses 21 and 22, which have a considerable influence on stereo (stereoscopic) imaging and NMF image synthesis. In the present embodiment, distortion caused by the image sensor and the lens is corrected.

<Overall image processing flow>
Assume that the image processing of this embodiment is executed by the CPU 51 of the control system 40. Further, the following correction process may be performed once before actual shooting (for example, when the stereo camera 20 is shipped).

  The overall flow of the correction processing of this embodiment is as shown in FIG. 2, and first, a calibration grid is photographed by the stereo camera 20 of FIG. 1 in step S21. As shown in FIG. 3, the grid 60 is a grid in which black and white vertical lines and horizontal lines are drawn on a white background, and is formed by printing or the like on paper or a plastic sheet. The smallest square rectangular block that is formed by adjacent straight lines of the grid 60 and surrounded by the intersections of the straight lines is hereinafter referred to as a “block” for convenience.

  The density and shooting distance of the grid 60 are arbitrary, but the grid 60 is selected so that a sufficient number of grids can be shot on the image sensor. For example, in the example described later, the image sensor has a pixel count of about 1024 × 768, but the density of the grid 60 and the shooting distance so that the grid appears at a density of about one for every 50 to 60 pixels of the image sensor. Is selected.

  In step S22, a range in which vertical and horizontal grid lines exist on the captured image data captured by the image sensor (for example, 201n) is calculated. Here, an equation of a straight line passing through the center of the grid line range (vertical / horizontal) around the intersection is obtained, and further, the intersection is obtained in units of subpixels from the equation of two vertical and horizontal straight lines. The processing for detecting the grid intersection on the captured image data will be described in detail below.

  In the captured image data, the grid is not shown as a straight line due to the influence of the distortion caused by the imaging element and the lens, so that in the subsequent step S23, the grid is a vertical and horizontal straight line, that is, each block is rectangular (square). Then, the corrected grid intersection is obtained. The grid corrected so as to be vertical and horizontal straight lines is a grid that was originally photographed when there was no distortion caused by the image sensor and the lens. This grid intersection correction will also be described in detail below.

  Needless to say, the grid before correction reflects the distortion caused by the image sensor and the lens. Therefore, if the difference between the shape of the grid before correction and the grid after correction is evaluated and the evaluation result is used, It is possible to obtain an image obtained by correcting an actual captured image and removing the influence of distortion caused by the image sensor and the lens. In this embodiment, as shown in step S24, the difference between the shape of the grid before correction and the shape of the grid after correction is evaluated for each block included in the grid based on the shape of the block before correction and the block after correction. This is done by obtaining a conversion function.

  As described above, if an affine transformation function for correcting an actual photographing pixel is obtained for each block, the photographed image is corrected by using the affine transformation function in actual photographing, resulting from the imaging element and the lens. Image data from which the influence of distortion is eliminated can be obtained. (Step S25).

  When performing the above correction processing, it is first important to obtain the grid intersection coordinates on the photographed image in step S22 of FIG. 2, and to correct the grid intersection points in step S23.

  Hereinafter, these correction processes will be described. First, image processing in a combination of a pair of lenses and an image sensor (for example, the lens 21 and the image sensor 201n) not related to stereo (stereo) shooting and NMF image composition will be described. Subsequently, image processing that is important in stereo (stereoscopic) shooting and NMF image composition, particularly NMF image composition, will be described.

<Grid intersection detection processing>
First, the grid intersection detection in step S22 will be described.

  Considering grid line detection, there are the following problems. As a general theory, it is conceivable to set a predetermined threshold value, and to detect a point where the luminance is below the threshold value in the captured image data as a position where the grid line exists, and to perform scanning by this threshold value in the vertical or horizontal direction. The position of the grid line can be obtained. However, in actuality, the grid line position cannot be obtained accurately by the above method due to the following two problems.

(1) Since the periphery of the lens is darker than the central part, the white part may be below the threshold. (2) Since the whole is distorted in a barrel shape, grid lines appear and disappear on the vertical and horizontal scanning lines. .

  Regarding the problem (1), FIG. 4 shows a luminance graph obtained by scanning the grid photographed image in the horizontal direction with one scanning line. In FIG. 4 (and FIGS. 5 and 6 described later), the vertical axis represents luminance, and the horizontal axis represents pixel address. As shown in the figure, sharp brightness drops corresponding to the vertical lines of the grid appear periodically, but it can be seen that the brightness drops at the periphery of the lens.

  Further, the barrel distortion of the problem (2) is as shown in FIG. 8 and is a phenomenon in which grid lines appear or disappear on the scanning lines L. In simple horizontal scanning, it may be erroneously detected that the grid line is not within the edge of the image but within the broken line in FIG. The scanning line data in FIG. 5 has a reduced luminance on the right side of the figure, and the influence of a horizontal grid distorted in a barrel shape appears in this portion.

  And the change of the brightness | luminance in the part which the problem (1) and the problem (2) compound is as shown in FIG. Further, FIG. 7 shows the change in luminance on a plurality of scanning lines in one graph. However, if FIGS. 4 to 7 are considered, the position of the grid line can be accurately calculated by cutting with a simple threshold. Proves extremely difficult.

  Therefore, in this embodiment, the following method is used to detect grid lines.

(A1) Detect low-luminance pixels that make up the grid lines. Here, the calibration image is scanned in the horizontal direction (in the case of vertical grid line detection) or the calibration image is scanned in the vertical direction (in the case of horizontal grid line detection). Candidate pixels are detected. Here, the luminance is lower than the average 60% (or other appropriate threshold percentage) of a plurality of pixels (for example, about 4 to 5 pixels) having the same pixel address in a plurality of past scanning lines (for example, about 4 to 5 lines). A pixel is a candidate pixel. (A3) A range where a grid line exists is detected. Here, the number of grid line candidate pixels at each pixel address is determined, and the continuity is evaluated. The following evaluation function is used for this evaluation.

  In this equation, p (i) is the number of grid line candidate pixels at the pixel address (coordinate) i. When the value of the evaluation function P (x) is equal to or greater than the threshold value, it is detected as a range where the grid line exists.

  FIG. 9 shows how the range is detected by the above evaluation function P (x). In FIG. 9, four scanning lines in the horizontal direction are shown, and the state of evaluation performed using these four scanning lines at the bottom is shown in the lower table. The upper part of this table shows the value of p (x) on the right side of the above evaluation function P (x) at each (horizontal) pixel address, and the number of black pixels at each (horizontal) pixel address in the upper four scanning lines. Matches. The lower part of the table is the value of the calculated evaluation function P (x). For example, the central “20” is the upper p (x) value “3”, “4”, “2” to P (x ) 20 = (3 × 4) + (4 × 2). In FIG. 9, the range of the value of P (x) with an appropriate threshold (for example, 10 or more) is detected as a grid line region.

  10 and 11 show examples of actual grid line ranges in the vertical and horizontal directions obtained from the grid image as described above. 10 and 11, the grid line area detected as shown in FIG. 9 is indicated by a black line. However, as shown in the figure, a wide range of grid lines is detected by the influence of distortion / aberration at the periphery of the screen. You can see that

  As described above, vertical and horizontal grid line ranges can be detected. Further, as shown in FIGS. 10 and 11, the grid intersection must be detected in the intersection region of the wide grid line range.

   When the vicinity of the intersection region of the wide grid line range as shown in FIGS. 10 and 11 is enlarged, the grid line range intersects as shown in FIG. 12, and the inside of the grid line range is filled with pixels having different luminances. The intersection coordinates to be detected here are

It is necessary to find the center of the vertical and horizontal lines. For this purpose, using the luminance value of one pixel before and after the pixel having the lowest luminance when viewed in the thickness direction of the line (a total of three pixels), the center of the line is obtained as follows.

(B1) When obtaining the center of the horizontal grid line in FIG. 12, as shown on the right side of FIG. 12, the luminance of one pixel before and after the pixel having the lowest luminance when viewed in the line thickness direction (total of three pixels) (2) Find a quadratic function that passes through these three luminance values using the luminance value of the pixel data in the grid line area using an appropriate arithmetic method such as the Gaussian method. The coordinates at which the brightness is reduced are detected as the center coordinates of the line.

  As described above, the coordinates of the grid intersection on the grid image can be obtained in units of subpixels.

  FIG. 13 is a graph showing the intersection (part) obtained from the grid image as described above, and FIG. 14 is an enlarged view of one grid line in the upper part of FIG. 13 and 14, the intersection coordinates detected as described above are indicated by black dots. These intersection coordinates (here, y coordinates) are calculated in units of subpixels. In FIG. 14, since the vertical direction is enlarged as compared with other figures, the curve of the grid line is shown with more emphasis.

  In addition, although the calculation regarding the scanning line regarding x (horizontal) direction was shown above, it cannot be overemphasized that the same calculation can be performed also about y (vertical) direction.

  As described above, grid intersection detection in step S22 of FIG. 2 can be performed.

<Grid intersection correction processing>
Next, the grid intersection correction in step S23 of FIG. 2 will be described.

  As is clear from the above description, the grid image captured by the calibration imaging is affected by the distortion caused by the imaging element and the lens, and is distorted as shown on the right side of FIG. In this embodiment, an affine transformation function reflecting this distortion state is obtained, and an actual captured image is corrected.

  In order to generate this affine transformation function, in addition to the actually captured grid image, the original undistorted grid image data is required. As this undistorted grid image data, it is also possible to use fixed grid image data that is presumed to be photographed on the image sensor in advance, while strictly controlling the photographing conditions such as the photographing distance, magnification, and center of angle of view. However, it is not so easy to accurately control shooting conditions during calibration.

  Therefore, in this embodiment, grid correction for obtaining a grid that is not distorted is performed based on the grid image that has been calibrated.

  In the grid correction of this embodiment, an original undistorted grid is generated over the entire screen using the four intersections P00, P01, P10, and P11 of the center portion of the grid image that has been calibrated as shown in FIG. . These intersections P00, P01, P10, and P11 can be easily identified by searching the grid image data outward from the coordinates of the screen center. Note that the intersections P00, P01, P10, and P11 at the center of the screen are used when this part is a part that has been imaged under the condition that the optical characteristics of the lens and the mechanical distortion of the image sensor are least affected. It is possible.

  If the intersection points P00, P01, P10, and P11 have the coordinate values of (x00, y00), (x01, y01), (x10, y10), and (x11, y11), respectively, these intersection points P00, The width Dx and height Dy of the block defined by P01, P10, and P11 can be obtained by determining appropriate two points out of these and subtracting the x coordinate value and the y coordinate value. If Dx is calculated as an average value such that Dx = ((x01−x00) + (x11−x10)) / 2 and the height Dy is calculated as Dy = ((y10−y00) + (y11−y01)) / 2, the calculation is performed. Accuracy can be improved.

  After that, if a value obtained by multiplying the width Dx or the height Dy thus obtained by an integer is added to the xy coordinates of the intersections P00, P01, P10, and P11, an intersection group without distortion can be obtained. For example, the coordinate values of the intersections where the x and y coordinates are large or small at the i and j-th (i and j are integers) with respect to the intersections P00, P01, P10 and P11 can be obtained as follows.

Intersection Pij whose x coordinate is smaller than P00 and y coordinate is also smaller:
Pij (x00−Dx × i, y00−Dy × j)
Intersection Pij having an x coordinate larger than P01 and a smaller y coordinate:
Pij (x01 + Dx × i, y01−Dy × j)
Intersection Pij having an x coordinate smaller than P10 and a larger y coordinate:
Pij (x10−Dx × i, y10 + Dy × j)
Intersection Pij whose x coordinate is larger than P11 and whose y coordinate is also larger:
Pij (x11 + Dx × i, y11 + Dy × j)
If the above calculation is performed while incrementing the i and j counters within a range where the value of Dx × i or Dy × j does not exceed the size of the screen, a grid intersection group without distortion can be obtained. If the value of Dx × i or Dy × j exceeds (or is equal to) the size of the screen during this calculation, the corrected intersection Pij at the end of the screen is the previous one as shown in FIG. By replacing one value of the x or y coordinates of the corrected intersection Pij with the coordinate value corresponding to the screen edge, the corrected intersection Pij of the screen edge can be obtained. The example of FIG. 18 is an example of the corrected intersection Pij at the upper end of the screen, and one value of the x or y coordinate of the previous corrected intersection Pij is replaced with 0 which is a coordinate value corresponding to the screen edge. . The corrected intersections on the other four sides can be obtained by the same calculation.

  As described above, it is possible to obtain grid intersection groups that respectively define corrected blocks without distortion based on grid images captured by calibration imaging.

  Note that the grid correction performed based on the grid image photographed by calibration photographing as described above does not require strict photographing condition control at the time of calibration, and can be easily performed at low cost.

<Correction function calculation processing>
From the pre-correction block and the post-correction block obtained as described above, an affine transformation function to be used in the actual captured image correction process (step S25 in FIG. 2) is obtained. For example, in FIG. 16, the block defined by the four intersections 1-1, 1-2, 2-1, 2-2 before correction at the upper left of the image is the corrected intersection 1-1, 1-2, 2- What is necessary is just to obtain | require the affine transformation function which can be matched with the block defined by the four intersection of 1 and 2-2, and can change the shape of this block before correction | amendment to the shape of the block after correction | amendment. The affine transformation function for this grid correction is generated for each block before and after each correction.

  In the program process (step S25 in FIG. 2) for handling an actual captured image, a process for correcting the luminance value of a pixel at a specific address (xy coordinate) on the screen is performed. Therefore, considering the processing efficiency, a specific address on the screen is considered as a specific point in the corrected block, and the address from which the luminance value that should exist at the specific address in this screen can be obtained is calculated. For this reason, it is convenient to obtain an affine transformation function for the block.

  In general, the affine transformation from coordinates (x, y) to coordinates (u, v) is

It is represented by In these two equations, p1, p2, p3, q1, q2, and q3 are affine transformation coefficients that determine the affine transformation function.

Here, in order to obtain the affine transformation coefficients p1 to q3 for a certain block, for example, p1 to q3 satisfying the following condition may be calculated:
(C1) Convert corrected intersection 1-1 (x11, y11) to uncorrected intersection 1-1 (u11, v11) (c2) Convert corrected intersection 1-2 (x12, y12) before correction Convert to 1-2 (u12, v12) (c3) Convert corrected intersection 2-1 (x21, y21) to intersection 2-1 (u21, v21) before correction, that is, affine transformation that satisfies the above conditions Coefficients are corrected by intersection point 1-1 (x11, y11), intersection point 1-2 (x12, y12), intersection point 2-1 (x21, y21), intersection point 2-2 (x22, y22). Transform the block into the intersection 1-1 (u11, v11), the intersection 1-2 (u12, v12), the intersection 2-1 (u21, v21), the intersection 2-2 (u22, v22), and the block before correction It is something that can be done.

  here,

And put

Holds. Expressions (10) and (11) are obtained by applying the above conditions (c1) to (c3) to the affine transformation function. Then, the affine transformation coefficients can be obtained by solving these equations (10) and (11). There are several ways to solve this equation. In this embodiment, since the matrix is as small as 3 × 3, the Kramel formula is used.

  For example, in the case of equation (10),

Can be calculated. Where ││ is a determinant whose value is

The affine transformation coefficients p1 to p3 can be obtained. The affine transformation coefficients q1 to q3 can be obtained by performing the same calculation for Expression (11).

<Photograph image correction processing>
The affine transformation in which the coefficient is obtained as described above is a transformation from a pixel address to a pixel address, so that a corresponding pixel address in the block before correction can be obtained from a predetermined pixel address in the corrected block. A coefficient is required. Therefore, in the program processing (step S25 in FIG. 2) for handling an actual captured image, if a certain target pixel address is determined, a pixel address corresponding to the target pixel address in the block before correction is obtained and obtained at that pixel address. Can be output as the luminance value of the target pixel address.

  For example, when the address (xy coordinates) of the target pixel to be processed is (150, 150) using the affine transformation coefficient obtained for a certain block, the affine transformation coefficient is used (152.5, 151. A pixel address such as 3) is obtained. That is, the luminance value of the target pixel at the pixel address (150, 150) is distorted and output to the address (152.5, 151.3) in the actual shooting data. If the luminance in 151.3) is output as the luminance of the pixel address (150, 150), distortion caused by the image sensor and the lens can be corrected.

  Here, since the coordinates corrected by the affine transformation are calculated in units of sub-pixels as (152.5, 151.3), in the actual calculation, the luminances of the four pixels at the positions surrounding the real number coordinates ( The luminance value may be obtained from the color data by linear interpolation or the like.

  For example, to linearly interpolate the luminance value c at the upper pixel address (152.5, 151.3), c00 (152, 151) c10 (153, 151) c01 (152, 152) c11 ( 153, 152) are used to perform the following linear interpolation:

  Here, a and b are decimal parts of the input coordinate values (152.5, 151.3). In this example, a = 152.5-152 = 0.5, and b = 151.3-151 = 0. .3.

  In the program process for handling an actual captured image (step S25 in FIG. 2), the captured image area is scanned one pixel at a time, and the luminance value (eg, RGB data) to be output at the target pixel address as described above. Just calculate.

  As described above, it is possible to correct errors caused by distortion of the image sensor and the photographing optical system, and obtain a more accurate image of the subject.

  Note that the coordinate values corrected by affine transformation may exceed the address range of the actual captured image data at the periphery of the screen, so for such a part, the image data is discarded as invalid or an appropriate calculation is performed. You may make it output the value approximated with the system.

<Correction function calculation processing in NMF shooting>
The correction processing that does not depend on stereo shooting and NMF shooting has been described above. The following correction can be performed in an imaging system that performs stereo imaging and NMF imaging as shown in FIG.

  As is apparent from FIG. 1, in NMF imaging, three pieces of image data of N (near), M (middle), and F (far) are obtained. For example, the image sensors 201n (202n), 201m (202m), and 201f (202f) in FIG. 1 may have different manufacturing distortions. The above correction processing may be performed independently for each image data obtained by each of these image pickup devices, but there is a problem that the calculation cost becomes extremely high.

  Therefore, after photographing in step S21 in FIG. 2, the image data of N (near), M (medium), and F (far) are respectively detected until the intersection coordinates are detected in step S22, and then in step S23. The grid intersection coordinate correction process is performed on one of N (near), M (middle), and F (far) image data, for example, an M (middle) image. Here, an uncorrected corrected intersection group is obtained over the entire screen from the four grid intersections in the center of the screen imaged as described above.

  For the N (near) and F (far) image data, the corrected intersection coordinates of the M (medium) image are used to match the corrected intersection coordinates of the three pixels. , M (medium), and F (far) image data for all affine transformation functions are obtained (step S24).

  For example, in the case of F (far) image data, the corrected grid intersection coordinates obtained from the M (middle) image data and the grid intersection coordinates before correction of the F (far) image data as shown in FIG. Is used to obtain the affine transformation function (step S24).

  In this way, when performing NMF image processing, the amount of calculation can be greatly reduced, and the image processing time can be shortened. In the case of a stereo photographing system, the above image processing is performed for each lens system (in the case of FIG. 1, the system of the image sensors 201n, 201m, 201f and the system of the image sensors 202n, 202m, 202f are respectively different). Not too long.

  The present invention can be implemented in any stereoscopic image processing system that performs stereoscopic image capturing and image processing. In the above, the configuration in which the image processing is performed by the PC hardware is exemplified, but the device for performing the image processing is not limited to the PC, and has an image processing device configured as a dedicated machine and a function of inputting image data. It may be an image forming apparatus (such as a printer) or a stereo camera itself. The program for controlling the image processing of the present invention can be introduced into the target image processing apparatus (the above-mentioned PC, stereo camera, etc.) via an appropriate storage medium or via a network.

It is explanatory drawing which showed the structure of the image processing system which employ | adopted this invention. It is the flowchart figure which showed the flow of the whole image processing by this invention. It is explanatory drawing which showed the structure of the grid used in this invention. It is the diagram which showed the luminance change of the scanning line image | photographed in the image processing by this invention. It is the diagram which showed the luminance change of the scanning line image | photographed in the image processing by this invention. It is the diagram which showed the luminance change of the scanning line image | photographed in the image processing by this invention. It is the diagram which showed the luminance change of the scanning line image | photographed in the image processing by this invention. It is explanatory drawing which showed the state of barrel distortion used as a problem in the image processing by this invention. It is explanatory drawing which showed application of the evaluation function in the image processing by this invention. It is explanatory drawing which showed the grid line range (vertical) detected in the image processing by this invention. It is explanatory drawing which showed the grid line range (horizontal) detected in the image processing by this invention. It is explanatory drawing which showed the detection method of the grid intersection coordinate in the image processing by this invention. It is the diagram which showed the grid intersection coordinate detected in the image processing by this invention. It is the diagram which expanded and showed the grid intersection coordinate detected in the image processing by this invention. It is explanatory drawing which showed the grid image | photographed in the image processing by this invention, and the image | photographed grid image. It is explanatory drawing which showed the correction method of the grid intersection in the image processing by this invention. It is explanatory drawing which showed the case where the correction method of the grid intersection in the image processing by this invention is applied to an NMF image. It is explanatory drawing which showed the correction method of the grid intersection of the screen edge part in the image processing by this invention.

Explanation of symbols

20 Stereo camera 21, 22 Lens 40 Control system 41 Display 42 Keyboard 51 CPU
54 HDD
55 Interface 201n, 201m, 201f Image sensor 202n, 202m, 202f Image sensor 203 CPU

Claims (5)

In an image processing method for correcting distortion caused by the photographing optical system and the image sensor of image data photographed using the photographing optical system and the image sensor,
An imaging process in which a grid image composed of vertical lines and horizontal lines orthogonal to each other is captured by the imaging optical system and the imaging element
Each coordinate of the grid image is sequentially scanned to detect pixels with low luminance constituting the grid line, and pixels that are below the predetermined threshold percentage of the average of a plurality of pixels at the same coordinate in a plurality of past scan lines are selected as grid line candidates. An evaluation function P (x) = p (x−1) p (x) + p (x) p () indicating the continuity of the number of candidate pixels at the same coordinates in the past plural scanning lines at the coordinates of each pixel. x + 1) (p (i): the number of candidate pixels of the grid line at the coordinate i) If the value P (x) is equal to or greater than the threshold value, the coordinates of the pixel are determined as the range where the grid line exists, Further, an intersection detection process for detecting the intersection of the grid lines obtained by detecting the center of the range in which the grid lines exist as the intersection of the blocks defined by the vertical and horizontal lines of the grid image ,
Grid intersection correction process for correcting the intersection coordinates of the block detected in the intersection detection process so that the grid lines of the block are vertical and horizontal straight lines, i.e., the block is rectangular.
A correction function for correcting an actual captured image is calculated based on the coordinates of each intersection defining the block obtained by the grid intersection correction process and the coordinates of each intersection defining the block in the grid image. An image processing method comprising: performing a calibration process including a correction function calculation process.   2. The image processing method according to claim 1, wherein in the correction function calculation process, the actual value in the block is determined based on the coordinate value of each block defining the block in the grid image and the block obtained by the grid intersection correction process. An image processing method comprising calculating an affine transformation function for correcting the captured image.   The image processing method according to claim 1, wherein a plurality of imaging elements arranged at different positions on the optical axis are arranged behind the imaging optical system so that subject images with different imaging distances can each be satisfactorily imaged. In the imaging process, each grid image is captured by each imaging device, and in the correction function calculation process performed for each of the obtained grid images, the grid performed for a predetermined grid image among the plurality of grid images. Calculate a correction function for correcting the actual captured image based on the coordinates of each intersection defining the block obtained by the intersection correction process and the coordinates of each intersection defining the block in the grid image. An image processing method characterized by the above. The image processing apparatus for implementing the image processing method according to any one of claims 1-3. An image processing program for performing a predetermined image processing apparatus an image processing method according to any one of claims 1-4.

Images