JP2013254300A - Image processing method - Google Patents

Abstract

PROBLEM TO BE SOLVED: To perform distortion correction processing quickly in a relatively small circuit.SOLUTION: Provided is an image processing method for processing an image by stereo matching by which identical points on a pair of photographed images photographed from different positions are searched to measure a three-dimensional position. The image processing method includes a step of creating, in performing segmentation processing of segmenting a part of an area in the photographed images and searching processing of performing said searching with reference to the image segmented in the segmentation processing, a lookup table for causing coordinates of the area before and after the segmentation in the segmentation processing to correspond to each other (S101a).

Description

  The present invention relates to an image processing method for processing an image by stereo matching that searches for the same point on a pair of captured images having different shooting positions and measures a three-dimensional position.

  In the development and production process of industrial products, an information system using a computer is utilized to centralize and manage all information related to design, development and production, and to improve process efficiency and shorten the period.

  In this information system, management of design-related data consisting of drawing data such as CAD data and document data such as specifications, management of component data constituting products and linkage with purchasing and material systems, design and production It consists mainly of functions such as workflow management to grasp the schedule and improve efficiency.

  In the production process, it is possible to perform operations such as picking and assembling components placed on the table by controlling the robot by 3D distance measurement by the stereo matching method and 2D template matching.

Here, 3D ranging by the stereo matching method is an image taken by the left camera using a pair of images taken by two cameras (stereo cameras) arranged on the left and right. This is a method of estimating the three-dimensional position of each point by triangulation using the area correlation calculation, which part of the image taken by the right camera corresponds to the area correlation calculation.
The 2D template matching is a technique for comparing a template image of a target object held in advance with a captured image and searching for a specific subject (target object) from the captured image.

  If a plurality of objects on the table can be photographed from directly above without tilting or rotating, 3D distance measurement is unnecessary and only 2D template matching is required. However, in actuality, since the tilt or rotation occurs in the direction in which the component is placed or the shooting direction of the camera, 3D distance measurement using the stereo matching method described above is necessary as preprocessing for 2D template matching. Become.

  Actually, distortion correction that removes barrel-type and pincushion-type distortion caused by the photographic lens using the result of 3D distance measurement, or when the subject surface and the optical axis are not perpendicular By performing the tilt correction for removing the tilt, the captured image can be compared with the template image.

  Techniques related to these are described in the following patent documents.

JP 2010-71743 A JP 2010-133718 A PCT / JP / 060184

  Patent Document 1 describes a robot system that performs 2D conversion on a captured image based on measurement data obtained by 3D distance measurement and performs object detection and robot control.

  Patent Document 2 describes position detection for a work object based on a photographed image, and describes that affine transformation is used for image rotation.

  Patent Literature 3 describes various distortion correction algorithms for captured images.

  However, neither patent document describes an efficient process of 3D distance measurement and 2D template matching, and each process needs to be processed in order. In addition, a POC process that can perform highly accurate 3D ranging processing by decomposing a captured image into amplitude information and phase information and searching for the same point using the phase information has been proposed. However, this POC process requires a Fourier transform, requires a high-performance computing device, and has a problem that it takes a long processing time. As a result, since the processing takes time as a whole, high-speed processing has been desired as a whole.

  In view of such a problem, the present invention can accurately recognize a subject by 3D distance measurement and 2D template matching, and can shorten the processing time with a relatively small circuit. An object of the present invention is to provide an image processing method.

  The above object is achieved by the invention described below.

  (1) The present invention provides an image processing method for processing an image by stereo matching that searches for the same point on a pair of photographed images with different photographing positions and measures a three-dimensional position, and is one of the photographed images. When performing a cutout process for cutting out a region of a part and a search process for performing the search with reference to an image cut out by the cutout process, a look for associating the coordinates of the areas before and after the cutout in the cutout process An up table is generated.

  (2) The present invention provides an image processing method for processing an image by stereo matching that searches for the same point on a pair of photographed images with different photographing positions and measures a three-dimensional position. When performing a cutout process for cutting out a region of a part and a search process for performing the search with reference to an image cut out by the cutout process, a look for associating the coordinates of the areas before and after the cutout in the cutout process An up table is generated, and the look-up table is a coordinate in a state in which distortion correction processing for correcting distortion in the image is performed as coordinates of a region after being cut out in the cut-out processing. And

  (3) The present invention is characterized in that, in the above (1)-(2), the pair of photographed images are decomposed into amplitude information and phase information, and the same point is searched using the phase information. To do.

  (4) The present invention, in (3), when performing the search for the same point using the phase information, repeatedly executing the cutout process while changing the resolution from low resolution to high resolution. Features.

  (5) In the present invention, the same point on a pair of photographed images with different photographing positions is searched, the three-dimensional position is measured, and the subject image is extracted, and the stereo matching preprocessing and the preprocessing are extracted. A post-processing by template matching for detecting a match between the subject image and a template image held in advance, wherein in the pre-processing, the posture of the subject is detected, and the tilt and the A lookup table for associating coordinates is generated so that a distortion correction process for correcting the rotation and distortion at the time of photographing is simultaneously performed.

  (6) This invention is characterized in that, in the above (5), the posture correction is determined based on a measurement result of the subject in the captured image.

  (7) The present invention is characterized in that, in the above (5) to (6), the posture correction is processed by rotating the subject around an optical axis at the time of photographing.

  (8) The present invention is characterized in that, in the above (5) to (6), the posture correction is performed by rotating the subject around an axis in a horizontal direction from an optical axis.

  (9) In the present invention, in the above (1) to (8), the distortion correction processing includes correcting an incident angle when the subject is imaged by the imaging unit and correcting distortion aberration generated when the imaging unit is captured. The relationship between the coordinates on the surface of the image sensor of the image pickup unit is taken into account as data, and the image after being cut out with the coordinates on the image sensor when the shot image obtained by the image pickup unit is cut out A look-up table is generated that correlates these coordinates with distortion correction.

  In the present invention, when an image is processed by stereo matching that searches for the same point on a pair of photographed images with different photographing positions and measures a three-dimensional position, a part of the photographed image is cut out. When executing the process and the search process for performing a search with reference to the image cut out by the cutout process, a lookup table is generated to correspond the coordinates of the area before and after the cutout in the cutout process. In the look-up table, the coordinates of the region after being cut out in the cut-out process are the coordinates in a state where the distortion correction process for correcting the distortion in the image is performed. As a result, the cut-out process for creating a reduced image and the distortion correction process for correcting various distortions are executed simultaneously, and high-speed processing is possible as a whole.

  Further, in the present invention, a pre-processing by stereo matching that searches for the same point on a pair of photographed images with different photographing positions, measures a three-dimensional position, and extracts a subject image, and a subject extracted by the pre-processing When performing post-processing by template matching that detects a match between an image and a template image held in advance, the pre-processing detects the tilt and rotation of the subject image, and the post-processing performs posture (a A lookup table for associating coordinates is generated so that (rotation) correction and distortion correction at the time of photographing are performed simultaneously as one distortion correction process. As a result, posture (tilting and rotation) correction and distortion correction are executed simultaneously, and high-speed processing is possible as a whole.

It is a block diagram which shows the structure of the apparatus which performs the process which concerns on 1st Embodiment. It is a block diagram which shows the principal part structure of the apparatus which performs the process which concerns on 1st Embodiment. It is a flowchart which shows the control flow of a comparative example. It is explanatory drawing which shows typically the mode of the process of a comparative example. It is a flowchart which shows the control flow of 1st Embodiment. It is explanatory drawing which shows typically the mode of the process of 1st Embodiment. It is a block diagram which shows the principal part structure of the apparatus which performs the process which concerns on 2nd Embodiment. It is a flowchart which shows the control flow of a comparative example. It is explanatory drawing which shows typically the mode of the process of a comparative example. It is a flowchart which shows the control flow of 2nd Embodiment. It is explanatory drawing which shows typically the mode of the process of 2nd Embodiment.

  Hereinafter, the present invention will be described based on specific examples, but the present invention is not limited to the specific examples shown in the embodiments.

[A] Configuration of the first embodiment:
FIG. 1 is a configuration diagram showing a configuration of an image processing system or an information system that realizes the image processing method according to the first embodiment. Hereinafter, the image processing system and the information processing system are collectively referred to as an image processing system.

  The image processing system includes a processing device 100 that performs the image processing method of the present embodiment, imaging units 110a and 110b that generate a pair of captured images with different imaging positions for stereo matching, and processing performed by the processing device 100. The display unit 120 displays results and various situations, and the operation unit 130 inputs various settings and the like to the processing apparatus 100.

  The processing apparatus 100 is used in an image processing system using a computer in industrial product development and production processes, and receives a pair of captured images from the imaging units 110a and 110b and performs preprocessing by stereo matching. And post-processing by template matching.

  Here, the processing apparatus 100 holds in advance a control unit 101 that performs various controls when performing the image processing method of the present embodiment, pre-processing by stereo matching, and subject images extracted by the pre-processing. An image processing unit 103 that performs post-processing by template matching for detecting matching of template images, and a storage unit 105 that stores in advance a template image used for comparison with a subject image at the time of template matching. Configured.

  The processing result of the processing apparatus 100 is displayed on the display unit 120 and also sent to a production apparatus such as a robot (not shown).

  The detailed configuration of the image processing unit 103 is shown in FIG. The image processing unit 103 includes a 3D distance measurement processing unit 1031, a 2D attitude measurement unit 1032, a corrected image creation processing unit 1033, and a 2D template matching processing unit 1039.

  Here, the 3D distance measurement processing unit 1031 is a set of two images taken by two cameras (stereo cameras) arranged on the left and right as a pre-process before the 2D template matching process (post-process) described later. Using a pair of captured images, a 3D distance measurement process using the stereo matching method is used to calculate which part of the image captured by the left camera corresponds to the portion of the image captured by the right camera by calculating the area correlation. The three-dimensional position of each point is estimated by triangulation using the correspondence.

  The 2D attitude measurement unit 1032 performs a two-dimensional attitude measurement process (2D attitude measurement process) such as rotation and offset of the subject included in the subject image according to a known method.

  The corrected image creation processing unit 1033 receives the measurement result of the 3D distance measurement processing unit 1031 and the measurement result of the 2D attitude measurement unit 1032 and applies a tilt to the subject image included in the subject image (inclination with respect to the optical axis of the imaging target surface). ), A photographed image in a state in which distortion generated during photographing such as rotation and offset is corrected is created.

  As a post-processing, the 2D template matching processing unit 1039 compares the template image of the target object from the storage unit 105 with the captured image created by the corrected image creation processing unit 1033, and performs a specific subject (target) in the captured image. Search for things.

  In addition, the 3D distance measurement processing unit 1031 includes a POC process for performing highly accurate 3D distance measurement processing by decomposing a captured image into amplitude information and phase information and searching for the same point using the phase information. Part 1031a.

  In addition, the POC processing unit 1031a includes a pyramid processing unit 1031b that repeatedly executes search for the same point using phase information while changing the resolution from low resolution to high resolution in the cutout process.

  In addition, the pyramid processing unit 1031b generates therein a lookup table for associating the coordinates of the area before and after being cut out in the cutout process, and the lookup table is used as the coordinates of the area after being cut out in the cutout process. , A distortion correction processing unit 1031c is provided which has coordinates in a state where distortion correction processing for correcting distortion in the captured image is performed.

[B] Operation of the first embodiment:
The operation of the first embodiment will be described below with reference to the flowcharts of FIGS. 3 and 5 and the explanatory diagrams of FIGS. 4 and 6 while comparing with the conventional operation.

  First, the operation of the conventional example corresponding to the first embodiment will be described with reference to FIG.

  With respect to the captured image (for example, G1 in FIG. 4) obtained by the imaging unit, downsize images G2, G3, and G4 having a small size and low resolution are generated by a downsize process such as a known thinning process (FIG. 3). Middle step S11a). Note that the generation of the above-described downsized image is performed on a pair of two captured images captured by two cameras (stereo cameras) arranged on the left and right.

  Then, with each layer image generated from the above pair of captured images, the image captured by the left camera corresponds to which part of the image captured by the right camera by the 3D ranging process by the stereo matching method. Whether to do so is determined by calculating area correlation, and the three-dimensional position of each point is estimated by triangulation using the corresponding relationship (step S11b in FIG. 3).

  In this case, for each layer, for G4, G3, G2, and G1 from the low-resolution image to the high-resolution image, a predetermined region including the subject is cut out, such as ga, gb, gc, and gd, and the stereo matching method is used. 3D distance measurement processing is performed. Note that the 3D distance measurement process by the stereo matching method is a preprocess, and the process is performed in a state where distortion such as distortion aberration included in the captured image remains.

  Thereafter, various distortion correction processes such as distortion correction are performed on the subject image extracted by the above pre-processing by stereo matching (step S12 in FIG. 3). The correction of the distortion aberration may be executed corresponding to the characteristics of the optical system of the imaging unit. Then, a template matching process for detecting a match with a template image held in advance for the subject image subjected to the distortion correction process is performed (step S13 in FIG. 3).

  Next, the operation of the first embodiment will be described with reference to FIG.

  For the captured images (for example, G1 in FIG. 4) obtained by the imaging units 110a and 110b, the POC processing unit 1031a in the 3D distance measurement processing unit 1031 decomposes the captured image into amplitude information and phase information, High-precision 3D distance measurement processing is performed by searching for the same point using the phase information.

  Here, the pyramid processing unit 1031b inside the POC processing unit 1031a searches for the same point using the phase information while changing the resolution from the low resolution to the high resolution in the clipping process such as ga to gd in FIG. Run repeatedly.

  In addition, the distortion correction processing unit 1031c in the pyramid processing unit 1031b generates a lookup table for associating the coordinates of the region before and after the extraction in the above-described extraction processing in the pyramid processing, and performs the extraction processing by the lookup table. Do. In this lookup table, the coordinates after the distortion correction processing for correcting the distortion in the photographed image is performed as the coordinates of the region after the clipping processing.

  That is, in the first embodiment, the pre-processing is executed in parallel with the distortion correction processing while performing the clipping processing for the pyramid processing in the 3D distance measurement processing (step S101a in FIG. 5). Here, as distortion correction processing, distortion correction that removes barrel-shaped and pincushion-shaped distortion aberration caused by the photographing lens, and tilt correction that occurs when the photographing target surface and the optical axis are not perpendicular to each other are removed. , Etc. are included. Therefore, since the 3D distance measurement processing in the 3D distance measurement processing unit 1031 is executed in a state where various distortions are removed from the captured image, the processing accuracy is improved as compared with the conventional method.

  Then, with a pair of images of each layer generated with the above distortion correction, any part of the image captured by the left camera is captured by the left camera by the 3D ranging process using the stereo matching method. Is obtained by area correlation calculation, and the three-dimensional position of each point is estimated by triangulation using the correspondence (step S101b in FIG. 5). As described above, the 3D distance measurement processing unit 1031 outputs the tilt of the subject image (inclination with respect to the optical axis of the imaging target surface) and the three-dimensional position information as the measurement result by the 3D distance measurement process using the pyramid process with distortion correction. The

  And the correction which received 3D position information which is a measurement result of 3D ranging processing in 3D ranging processing part 1031 and 2D attitude information which is a measurement result of 2D attitude measurement processing in 2D attitude measurement part 1032 The image creation processing unit 1033 creates a captured image in a state where various distortions are corrected, and sends the captured image to the 2D template matching processing unit 1039. The 2D template matching processing unit 1039 compares the template image of the target object from the storage unit 105 with the captured image created by the corrected image creation processing unit 1033 by template matching processing as post-processing, and A specific subject (object) is searched for (step S102 in FIG. 5).

  In the first embodiment described above, when an image is processed by stereo matching that searches for the same point on a pair of photographed images with different photographing positions and measures a three-dimensional position, some of the photographed images are processed. Generates a lookup table for associating the coordinates of the area before and after clipping in the clipping process when executing the clipping process for clipping the area and the search process for searching with reference to the image clipped by the clipping process In this lookup table, the coordinates of the region after being cut out in the cut-out process are coordinates in a state in which the distortion correction process for correcting the distortion in the image has been performed. As a result, the cut-out process and the distortion correction process for correcting various distortions are executed simultaneously, and high-speed processing as a whole becomes possible.

  Hereinafter, how the 3D distance measurement processing unit 1031 performs distortion correction processing in parallel while performing cutout processing for pyramid processing in 3D distance measurement processing will be described with reference to FIG.

  For the coordinate conversion that performs the clipping process with the distortion correction process, for example, a coordinate conversion method described in WO 2011/158344 proposed by the applicant of the present application can be used. This coordinate transformation method is a method for converting image data of a virtual projection plane composed of a curved surface set in the world coordinate system or a plane connecting a plurality of planes to a camera coordinate system using a distortion correction coefficient. This is image processing that is calculated based on the coordinates converted into the camera coordinate system and the pixel data of the image sensor. Note that this coordinate conversion method is merely an example, and other methods may be used.

  FIG. 6A is an example of a captured image obtained by the imaging units 110a and 110b. FIG. 6A corresponds to the high resolution image G1 in FIG.

  When the region Obj (hatched portion in FIG. 6B) is cut out from the captured image in the pyramid process, the virtual range included in the imaging range Val by the imaging units 110a and 110b (Ca in FIG. 6C). This can be done by coordinate transformation designating the projection plane Vobj.

  In this case, the change in the size of the clipping process by the pyramid process can be dealt with by the zoom process in FIG. Further, the change of the position of the clipping process can be dealt with by changing the position of the virtual projection plane Vobj or changing the position of the viewpoint Ca.

  In addition, the coordinate conversion look table including distortion correction parameters so that the above-described various distortion (distortion aberration, tilt, etc.) correction is executed when executing coordinate conversion for cutting out an image of a desired region. Create In this case, a look-up table is created so as to perform coordinate conversion using a distortion correction coefficient based on a correction parameter for tilt and rotation of the subject image and a distortion correction parameter for the lens.

  That is, the distortion correction processing unit 1031c creates a lookup table for converting each coordinate on the virtual projection plane to the camera coordinate system based on the set position and size of the virtual projection plane Vobj, The pixel data photographed by the imaging units 110a and 110b is processed. As a result, the distortion correction processing unit 1031c performs coordinate transformation that applies to which coordinate position of the camera coordinates the world coordinate position corresponds to based on the lens data. At this time, the correction of distortion and the calculation of the position of the virtual projection plane Vobj are performed together with rotation, tilt (tilt), translation, enlargement, and reduction.

  By executing such coordinate conversion, a subject image cut out in a state where distortion correction processing is performed as shown in FIG. 6D is obtained.

[C] Configuration of the second embodiment:
FIG. 7 shows a detailed configuration of the image processing unit 103 ′ in the image processing system that implements the image processing method according to the second embodiment. As the image processing system to which the second embodiment is applied, the one shown in FIG. 1 can be applied.

  The image processing unit 103 ′ includes a 3D distance measurement processing unit 1035, a 2D attitude measurement unit 1036, a corrected image creation processing unit 1037, and a 2D template matching processing unit 1039.

  Here, the 3D ranging processing unit 1035 is a set of two images taken by two cameras (stereo cameras) arranged on the left and right as a pre-process before the 2D template matching process (post-process) described later. Using a pair of captured images, a 3D distance measurement process using the stereo matching method is used to calculate which part of the image captured by the left camera corresponds to the portion of the image captured by the right camera by calculating the area correlation. The three-dimensional position of each point is estimated by triangulation using the correspondence.

  The 2D attitude measurement unit 1036 performs a two-dimensional attitude measurement process (2D attitude measurement process) such as rotation and offset of a subject included in the subject image according to a known method.

  In response to the measurement result of the 3D distance measurement processing unit 1035 and the measurement result of the 2D attitude measurement unit 1036, the corrected image creation processing unit 1037 performs pre-processing on the subject image included in the subject image as A photographed image is created in a state in which distortions occurring during photographing such as (tilt with respect to the optical axis), rotation, and offset are corrected.

  The 2D template matching processing unit 1039 compares the template image of the target object from the storage unit 105 with the captured image created by the corrected image creation processing unit 1037 as post-processing subsequent to the above pre-processing. Find a specific subject (object).

[D] Operation of the second embodiment:
The operation of the second embodiment will be described below with reference to the flowcharts of FIGS. 8 and 10 and the explanatory diagrams of FIGS. 9 and 11 while comparing with the conventional operation.

  First, the operation of the conventional example corresponding to the second embodiment will be described with reference to FIG. Note that a template image (FIG. 9A) of an object is held in the storage unit 105 in advance.

  With respect to the captured image (FIG. 9B) obtained by the imaging unit, a two-dimensional posture, for example, the rotation θ of the subject image is measured by a known method such as Hough transform (step S21 in FIG. 8). Various known methods can be used to measure the rotation θ. As a reference for rotation, a template image may be used, or a preset algorithm (long side is horizontal, etc.) may be used as a reference.

  Then, with respect to a pair of captured images (FIG. 9B) obtained by the imaging unit, which of the images captured by the left camera is the image captured by the left camera by the 3D distance measurement process using the stereo matching method. It is calculated by area correlation calculation whether it corresponds to a part, and the three-dimensional position of each point is estimated by triangulation using the correspondence, and the three-dimensional posture (tilt etc.) of the subject image is measured (Fig. 8 step S22).

  Thereafter, with respect to the subject image extracted by the preprocessing by stereo matching described above, first, the above-described rotation and tilt are corrected by a known technique such as affine transformation (step S23 in FIG. 8). Here, FIG. 9C shows a photographed image before correction and FIG. 9D shows a photographed image after correction.

  Next, various distortion correction processes such as distortion correction due to the optical system of the imaging unit are performed (step S24 in FIG. 8). The correction of the distortion aberration may be executed corresponding to the characteristics of the optical system of the imaging unit. Here, FIG. 9D shows a captured image before distortion correction, and FIG. 9E shows a captured image after distortion correction.

  Then, a template matching process for detecting a match with a template image (FIG. 9A) held in advance is performed on the subject image (FIG. 9E) subjected to the distortion correction process (step S25 in FIG. 8). ).

  In the above series of processing, rotation and tilt correction processing, distortion correction processing, and two-step correction processing are performed.

  Next, the operation of the second embodiment will be described with reference to FIG.

  With respect to one of the captured images obtained by the imaging units 110a and 110b, the 2D orientation measurement unit 1036 uses a known method such as Hough transform, so that the 2D orientation of the subject included in the subject image, for example, rotation θ of the subject image Is measured according to a known method (step S201 in FIG. 10). Various known methods can be used to measure the rotation θ. As a reference for rotation, a template image may be used, or a preset algorithm (long side is horizontal, etc.) may be used as a reference.

  Then, with respect to the pair of photographed images obtained by the imaging units 110a and 110b, the image photographed by the left camera is photographed by the right camera by the 3D ranging processing unit 1035 by the 3D ranging process by the stereo matching method. Determine which part of the image corresponds to the area by calculating the area correlation, and estimate the three-dimensional position of each point by triangulation using the correspondence, and measure the three-dimensional posture (tilting, etc.) of the subject image (Step S202 in FIG. 10). In addition, as this 3D ranging process, a highly accurate 3D ranging process is performed by decomposing a captured image into amplitude information and phase information by POC processing and searching for the same point using the phase information as necessary. May be performed.

  As described above, as preprocessing, two-dimensional posture information such as rotation of the subject image in the photographed image is output from the 2D posture measurement unit 1036 as a measurement result, and the three-dimensional distance measurement processing unit 1035 is in the photographed image. And the three-dimensional position information of the subject image are output as measurement results.

  Then, two-dimensional posture information such as tilt and three-dimensional position information that is a measurement result of the 3D distance measurement processing in the 3D distance measurement processing unit 1035 and rotation that is a measurement result of the 2D posture measurement processing in the 2D posture measurement unit 1036 The corrected image creation processing unit 1037 that receives the above, unlike the processing of the conventional example, executes distortion correction processing as one process including posture correction (rotation and tilt correction) and distortion correction. Then, a captured image with various distortions corrected is created and sent to the 2D template matching processing unit 1039.

  The 2D template matching processing unit 1039 compares the template image of the object from the storage unit 105 with the captured image created by the corrected image creation processing unit 1037 by template matching processing as post-processing. A specific subject (object) is searched for (step S204 in FIG. 10).

  In the second embodiment described above, the same point on a pair of photographed images with different photographing positions is searched, the three-dimensional position is measured, and the subject image is extracted, and the stereo matching preprocessing and the preprocessing are extracted. When performing post-processing by template matching that detects a match between a pre-held template image and a template image held in advance, the pre-processing detects the posture (tilt and rotation) of the subject image, and A lookup table for associating coordinates is generated so that (rotation) correction and distortion correction at the time of photographing are performed simultaneously as one distortion correction process. As a result, posture (tilting and rotation) correction and distortion correction are simultaneously performed as one distortion correction process, and high-speed processing as a whole becomes possible.

  Hereinafter, how the corrected image creation processing unit 1037 executes posture (tilting and rotation) correction and distortion correction simultaneously as one distortion correction process will be described with reference to FIG.

  In FIG. 11, the viewpoint Ca represents the imaging positions of the imaging units 110a and 110b, and Vobj is a virtual projection plane corresponding to the imaging plane of the subject image. Then, when tilt and rotation correction is performed on the subject images captured by the imaging units 110a and 110b, the viewpoint conversion function for the virtual projection plane Vobj in the distortion correction processing can be used.

  For distortion correction processing that simultaneously performs tilt correction, rotation correction, and distortion correction, for example, a coordinate conversion or viewpoint conversion method described in WO 2011/158344 proposed by the applicant of the present application may be used. it can. As described above, the coordinate conversion method can use conversion of coordinates in the world coordinate system to a camera coordinate system using a distortion correction coefficient. In addition, as the viewpoint conversion method, a process of changing the position of the virtual projection plane with the image center of the camera coordinate system as the rotation center (or movement center) can be employed. Note that the coordinate conversion and viewpoint conversion methods are merely examples, and other methods may be used.

  FIG. 11 is an example in which the position (including rotation and tilt) of the virtual projection plane Vobj is changed with the viewpoint Ca of each of the imaging units 110a and 110b as the rotation center (or movement center).

  As shown in FIG. 11, rotation about the X axis with the viewpoint Ca as the center of rotation is pitch (also referred to as pitch or tilt). The rotation around the Y axis is yaw (also called yaw or pan). Further, the rotation around the Z axis is a roll.

  In FIG. 11, viewpoint conversion by rotation (pitch) around the X axis is equivalent to rotating the subject in the virtual projection plane Vobj from the optical axis in the r direction around the horizontal axis, and in the X axis direction. Can be corrected. The viewpoint conversion by rotation around the Y axis (yaw) is equivalent to rotating the subject in the virtual projection plane Vobj in the y direction around the vertical axis from the optical axis, and correcting the tilt in the Y axis direction. it can. In addition, the viewpoint is converted by rotating (rolling) around the X axis, so that the subject is rotated in the r direction, and the rotation of the subject can be corrected. That is, various viewpoint corrections of the subject image in the photographed image can be performed by the above viewpoint conversion function.

  In FIG. 11, although the viewpoint conversion is based on rotation, the virtual projection plane Vobj is moved by translation of the viewpoint Ca (not shown), and the optical axis is moved in the X or Y direction with the viewpoint Ca as the rotation center. (Not shown), the virtual projection plane Vobj can be moved, and the posture of the subject image can be corrected by these movements.

  Then, various corrections (blue / rotation correction, blue / rotation correction, so as to execute distortion correction due to the optical system of the imaging units 110a and 110b at the same time as correction of tilt and rotation by viewpoint conversion in the above distortion correction processing are performed. Create a coordinate conversion look-up table that includes parameters for distortion correction. In this case, a look-up table is created so as to perform coordinate conversion using a distortion correction coefficient based on a viewpoint conversion parameter for correcting tilt and rotation of the subject image and a lens distortion correction parameter.

  That is, the corrected image creation processing unit 1037 creates a lookup table for converting each coordinate on the virtual projection plane into the camera coordinate system based on the set position and size of the virtual projection plane Vobj, and the lookup table Is used to process pixel data photographed by the imaging units 110a and 110b. As a result, the corrected image creation processing unit 1037 performs coordinate conversion that applies to which coordinate position of the camera coordinates the world coordinate position corresponds to based on the lens data. At this time, the correction of distortion and the calculation of the position of the virtual projection plane Vobj are performed together with rotation, tilt (tilt), translation, enlargement, and reduction.

DESCRIPTION OF SYMBOLS 100 Processing apparatus 101 Control part 103 Image processing part 105 Storage part 110a, 110b Imaging unit 120 Display part 130 Operation part 1031 3D distance measurement process part 1032 2D attitude | position measurement part 1033 Correction image creation process part 1035 3D distance measurement process part 1036 2D attitude | position Measurement unit 1037 Correction image creation processing unit 1039 2D template matching processing unit

Claims (9)

In an image processing method for processing an image by stereo matching that searches for the same point on a pair of captured images having different shooting positions and measures a three-dimensional position,
When performing a cutout process for cutting out a partial area of the captured image and a search process for performing the search with reference to the image cut out by the cutout process,
Generating a lookup table for associating the coordinates of the area before and after the cutout in the cutout process;
An image processing method. In an image processing method for processing an image by stereo matching that searches for the same point on a pair of captured images having different shooting positions and measures a three-dimensional position,
When performing a cutout process for cutting out a partial area of the captured image and a search process for performing the search with reference to the image cut out by the cutout process,
Generate a lookup table for associating the coordinates of the area before and after cutting in the cutting process,
The look-up table is a coordinate in a state where a distortion correction process for correcting distortion in the image is performed as the coordinates of the area after being cut out in the cutout process.
An image processing method. Decomposing the pair of captured images into amplitude information and phase information, and searching for the same point using the phase information;
The image processing method according to claim 1-2. When performing a search for the same point using the phase information, repeatedly executing while changing the resolution from low resolution to high resolution in the cutout process,
The image processing method according to claim 3. Pre-processing by stereo matching that searches for the same point on a pair of captured images with different shooting positions, measures a three-dimensional position, and extracts a subject image, and the subject image extracted by the pre-processing are stored in advance. A post-processing by template matching for detecting a match with a template image, and an image processing method comprising:
In the preprocessing, a look-up table that associates coordinates before and after correction so as to simultaneously perform posture correction of the captured image and distortion correction of the captured image,
An image processing method. The posture correction is determined based on a measurement result of the subject in the captured image.
The image processing method according to claim 5. The posture correction is processed by rotating the subject around the optical axis at the time of shooting.
An image processing method according to claim 5-6. The posture correction is performed by rotating the subject around the horizontal axis from the optical axis.
An image processing method according to claim 5-6. The distortion correction process
The relationship between coordinates on the imaging device surface of the imaging unit is considered as data, taking into consideration the incident angle to the imaging unit when shooting the subject and the correction of distortion occurring when shooting with the imaging unit. And
Generating a look-up table that correlates the coordinates on the image pickup element and the coordinates of the image after being cut out with distortion correction when cutting a captured image obtained by the image pickup unit;
The image processing method according to claim 1, wherein: