JP5742179B2 - Imaging apparatus, image processing apparatus, image processing method, and program - Google Patents

Description

  The present invention relates to an imaging device, an image processing device, an image processing method, and a program. More specifically, the present invention relates to an imaging apparatus, an image processing apparatus, an image processing method, and a program for correcting an image captured by the imaging apparatus.

  In recent years, imaging apparatuses such as a still camera, a video camera, a single-lens reflex camera, a camera with a built-in mobile device, a surveillance camera, and a camera with a personal computer have been reduced in size, weight, and price. However, these compact and lightweight cameras have many restrictions on the design of an optical system such as a lens, and this causes a problem that image deterioration is likely to occur due to, for example, lens aberration.

Lens distortion (distortion) is a phenomenon that tends to appear as the zoom magnification of the camera increases. Specific examples of lens distortion include the following.
(A) Barrel distortion in which a photographed image is rounded like a barrel and bends outward.
(B) A pin-cushion distortion in which the four corners of the image are stretched and bent into a pincushion mold.
(C) Distortion in which both the above (a) and (b) are mixed. For example, there is lens distortion aberration.

  This distortion can be avoided to some extent by improving the accuracy of the lens design, but it is difficult to remove it completely. In particular, as described above, there is a problem that it is practically difficult to mount a lens subjected to high-precision processing in a small-sized, light-weight, and low-cost camera. In order to solve this problem, a camera equipped with a function of correcting a distortion of a captured image by performing geometric transformation by image processing has recently appeared.

  Another problem associated with the reduction in size and weight of the camera is that the camera shake of the photographer and the vibration of the shooting situation (such as a vehicle) impair the sense of stability of the shot image. This method of suppressing the blur component is a method of suppressing the blur by optically guiding the blur in the lens part or the image sensor part (optical correction), and the amount of blur from the image data after imaging. Two methods of canceling and reading (electronic correction) are the mainstream. In both cases, as a correction method, for example, the blur component is divided into parallel movement vectors on two axes in the left-right direction (X direction) and the up-down direction (Y direction), and the respective correction amounts are calculated. Actually, however, there are many hand shake phenomena in which a rotation vector component is added to the translation vector, and this rotation hand shake cannot be removed by correcting the translation in the X and Y directions.

  By the way, humans can see an object in three dimensions because of the principle that both eyes observe the object from a different direction at a certain interval, and a slightly different image appears in each retina. . That is, the stereoscopic effect is recognized by the parallax between both eyes.

  A so-called stereo camera for capturing a stereoscopic image with an imaging device, that is, a camera having two optical imaging systems, applies this principle, and is configured to capture two slightly different images that are the basis of a stereoscopic image. Have.

  As described above, the stereo camera generates two systems of images, that is, a left-eye image and a right-eye image, and records them in the memory. For example, a left-eye image and a right-eye image are alternately displayed on the 3D display using these images. The observer can observe each image as a stereoscopic image by wearing shutter glasses for observing each image with only the left eye or only the right eye. There are various 3D display methods other than a method using shutter glasses.

  A 3D display image can recognize a sense of depth according to the parallax of a captured image, but a stereo camera that captures an image for realizing comfortable viewing while suppressing fatigue of an observer requires several conditions. . Specifically, it is required to adjust the parallax setting and strictly optimize the mounting position and direction of the two optical imaging systems of the left-eye image and the right-eye image.

The two optical imaging systems of the left-eye image and the right-eye image of the stereo camera use independent lenses, and individual lens distortion occurs independently. Therefore, in order to remove distortion, individual correction for each lens is required.
In addition, as a prior art disclosed about the correction | amendment process of two images image | photographed with such a stereo camera, there exists patent document 1 (Japanese translations of PCT publication No. 2008-524673), for example.

  As described above, there are lens distortion aberration correction, translational camera shake correction, and rotational camera shake correction as image correction processing for an image photographed by the camera, and in the case of a stereo camera, parallax correction for two optical imaging systems. Etc., corrections according to many different purposes are required.

JP 2008-524673 A

  The present invention has been made in view of, for example, the above-described situation, such as lens distortion correction for an image photographed by a camera, parallel camera shake correction, rotational camera shake correction, parallax correction for two optical imaging systems, and the like. An object is to provide an imaging apparatus, an image processing apparatus, an image processing method, and a program that reliably execute many different corrections.

The first aspect of the present invention is:
A plurality of imaging units;
A correction unit that performs correction processing on the captured images of the plurality of imaging units;
A control unit that calculates a correction parameter to be applied to the correction process in the correction unit;
The correction unit is
While performing distortion correction and camera shake correction for each of the captured images,
The imaging apparatus performs an image characteristic matching correction process for matching characteristics of a plurality of images photographed by the plurality of imaging units.

  Furthermore, in an embodiment of the imaging apparatus of the present invention, the correction unit performs a zoom magnification correction process for matching zoom magnifications of a plurality of images photographed by the plurality of imaging units as the image characteristic adjustment correction process. Run.

  Furthermore, in one embodiment of the imaging apparatus of the present invention, the correction unit performs optical axis center correction for matching the optical axis centers of a plurality of images taken by the plurality of imaging units as the image characteristic adjustment correction process. Execute the process.

  Furthermore, in one embodiment of the imaging apparatus of the present invention, the correction unit further executes a parallax correction process for adjusting parallax of a plurality of images photographed by the plurality of imaging units.

  Furthermore, in an embodiment of the imaging apparatus of the present invention, the control unit calculates a correction parameter to be applied to the correction process in the correction unit and provides the correction unit with the calculation.

  Furthermore, in an embodiment of the imaging apparatus of the present invention, the correction parameter is a correction vector applied to image coordinate conversion in the correction unit.

  Furthermore, in an embodiment of the imaging apparatus of the present invention, the correction unit includes a distortion aberration correction unit that performs distortion correction on each of the captured images, and a rotation camera shake correction unit that performs rotational camera shake correction on each of the captured images. A translational camera shake correction unit that performs translational camera shake correction on each of the captured images, a zoom magnification correction unit that matches zoom magnifications that are characteristics of a plurality of images captured by the plurality of imaging units, and the plurality An optical axis center correction unit that matches the optical axis centers, which are characteristics of a plurality of images captured by the plurality of imaging units, and a parallax correction process that adjusts the parallax of the plurality of images captured by the plurality of imaging units. It is the structure which has a parallax correction part.

  Furthermore, in an embodiment of the imaging apparatus of the present invention, the control unit calculates an integrated correction vector obtained by integrating correction vectors applied to a plurality of different correction processes executed in the correction unit, and provides the integrated correction vector to the correction unit. The correction unit is configured to collectively execute a plurality of different correction processes by an image conversion process to which the integrated correction vector is applied.

  Further, in one embodiment of the imaging apparatus of the present invention, the control unit includes a distortion aberration correction process, a rotation camera shake correction process, a parallel movement camera shake correction process, a zoom magnification correction process, an optical axis center correction process, A parallax correction process, an integrated correction vector obtained by integrating correction vectors applied to each correction process is calculated and provided to the correction unit, and the correction unit corrects distortion aberration by image conversion processing using the integrated correction vector. In this configuration, the process, the rotational camera shake correction process, the parallel movement camera shake correction process, the zoom magnification correction process, the optical axis center correction process, and the parallax correction process are executed in a batch.

  Furthermore, in an embodiment of the imaging apparatus of the present invention, the imaging apparatus has a distortion aberration data storage unit that stores distortion aberration data corresponding to the imaging unit to be applied to distortion aberration correction, and the control unit includes the distortion unit. A distortion aberration correction parameter is generated based on the data acquired from the aberration data storage unit.

  Furthermore, in an embodiment of the imaging apparatus according to the present invention, the imaging apparatus applies optical axis center correction applied to optical axis center correction that matches the optical axis centers, which are characteristics of a plurality of images captured by a plurality of imaging units. An optical axis center correction value storage unit storing data is provided, and the control unit generates an optical axis center correction parameter based on data acquired from the optical axis center correction value storage unit.

  Furthermore, in an embodiment of the imaging apparatus of the present invention, the imaging apparatus stores zoom magnification correction data to be applied to zoom magnification correction for matching zoom magnifications that are characteristics of a plurality of images taken by a plurality of imaging units. The control unit generates a zoom magnification correction parameter based on data acquired from the zoom magnification correction value storage unit.

  Furthermore, in an embodiment of the imaging apparatus of the present invention, the imaging apparatus has a parallax data storage unit that stores parallax data to be applied to parallax correction for adjusting parallax of a plurality of images captured by a plurality of imaging units. Then, the control unit generates a parallax correction parameter based on the data acquired from the parallax data storage unit.

Furthermore, the second aspect of the present invention provides
A correction unit that executes correction processing on captured images of a plurality of imaging units;
A control unit that calculates a correction parameter to be applied to the correction process in the correction unit;
The correction unit is
While performing distortion correction and camera shake correction for each of the captured images,
The image processing apparatus executes an image characteristic matching correction process for matching characteristics of a plurality of images captured by the plurality of imaging units.

Furthermore, the third aspect of the present invention provides
An image processing method executed in an image processing apparatus,
A parameter calculating step in which the control unit calculates a correction parameter to be applied to the correction processing in the correction unit;
The correction unit executes a correction step of executing correction processing to which the correction parameter is applied to the captured images of the plurality of imaging units,
The correction step includes
While performing distortion correction and camera shake correction for each of the captured images,
The image processing apparatus method is a step of executing an image characteristic matching correction process for matching characteristics of a plurality of images photographed by the plurality of imaging units.

Furthermore, the fourth aspect of the present invention provides
A program for executing image processing in an image processing apparatus;
A parameter calculation step for causing the control unit to calculate a correction parameter to be applied to the correction process in the correction unit;
Causing the correction unit to execute a correction step of executing correction processing to which the correction parameter is applied to the captured images of the plurality of imaging units;
The correction step includes
While executing distortion correction and camera shake correction for each of the captured images,
The program is a step of executing image characteristic matching correction processing for matching characteristics of a plurality of images photographed by the plurality of imaging units.

  Note that the program of the present invention is a program provided by, for example, a storage medium to an information processing apparatus or a computer system that can execute various program codes. By executing such a program by the program execution unit on the information processing apparatus or the computer system, processing according to the program is realized.

  Other objects, features, and advantages of the present invention will become apparent from a more detailed description based on embodiments of the present invention described later and the accompanying drawings. In this specification, the system is a logical set configuration of a plurality of devices, and is not limited to one in which the devices of each configuration are in the same casing.

  According to an embodiment of the present invention, it is possible to provide a configuration that reliably and efficiently implements a plurality of different image corrections on a captured image. Specifically, a zoom magnification correction, a distortion aberration correction, a rotation camera shake correction, a parallel camera shake correction, an optical axis center correction, a parallax correction, and a configuration that reliably executes all of these corrections are provided. In addition, a configuration is provided in which these correction processes are collectively executed to realize an efficient process. Especially for zoom magnification correction, optical axis center correction, and parallax correction required for a stereo camera that captures two images from two different viewpoints, the quality can be improved by correcting at least one of the images captured by the two imaging units. High 3D images can be provided. Furthermore, according to the configuration in which an integrated correction vector obtained by integrating correction vectors applied to each correction is calculated and corrected, efficient and reliable correction is realized.

It is a figure explaining the structural example (1st Example) of the imaging device which concerns on this invention. It is a figure explaining the structural example (2nd Example) of the imaging device which concerns on this invention. It is a figure explaining the structural example (3rd Example) of the imaging device which concerns on this invention. It is a figure explaining a distortion aberration. It is a figure explaining the structure and attachment example of a camera shake sensor. It is a figure explaining a response | compatibility with a zoom position and the gain value of a camera shake sensor. It is a figure explaining a distortion aberration correction vector and a parallel movement camera-shake correction vector. It is a figure explaining a rotation camera shake correction vector. It is a figure explaining a parallax correction vector. It is a figure explaining the example of addition vector setting of distortion aberration correction and rotation hand-shake correction. It is a figure explaining the correction example of distortion aberration correction and rotation hand-shake correction. It is a figure which shows the flowchart explaining the image correction process sequence which the image processing apparatus which concerns on this invention performs. It is a figure which shows the flowchart explaining the image correction process sequence which the image processing apparatus which concerns on this invention performs. It is a figure explaining the setting of the data for correction | amendment according to the position of a zoom lens or a focus lens, and an example of an acquisition process. It is a figure which shows the example of the correction vector as a zoom magnification correction parameter applied to a zoom magnification correction process. It is a figure explaining the example of a representative point setting, and the production | generation process example of an interpolation vector. It is a figure explaining the correction vector as an optical axis center correction parameter applied to an optical axis center correction process. It is a figure which shows the process example in case the correction process which applied the correction vector by each correction | amendment part is performed. It is a figure explaining the addition process example of a correction vector. It is a figure explaining the addition process example of a correction vector.

Hereinafter, an imaging device, an image processing device, an image processing method, and a program according to the present invention will be described in detail with reference to the drawings. The description will be made according to the following items.
1. 1. Configuration of imaging apparatus and processing example 1-1. Configuration of first embodiment 1-2. Configuration of second embodiment 1-3. Configuration of the third embodiment Details of processing in the apparatus of the present invention.
3. 3. Image correction processing sequence in imaging apparatus 3-1. Correction sequence for sequentially executing individual correction processes 3-2. 3. Correction sequence for executing a plurality of correction processes as a batch correction process Correction vector as a correction parameter according to the correction purpose 4-1. (A) Correction vector as a zoom magnification correction parameter applied to zoom magnification correction processing 4-2. (B) Correction vector as distortion aberration correction parameter applied to distortion aberration correction processing 4-3. (C) Correction vector as a rotation camera shake correction parameter applied to the rotation camera shake correction processing 4-4. (D) Correction vector as parallel camera shake correction parameter applied to parallel camera shake correction processing 4-5. (E) Correction vector as optical axis center correction parameter applied to optical axis center correction processing 4-6. (F) Correction vector as parallax correction parameter applied to parallax correction processing Specific examples of correction vector integration processing

[1. Regarding configuration and processing example of imaging apparatus]
With reference to FIG. 1 and subsequent figures, a configuration and processing example of an imaging apparatus which is an example of an image processing apparatus of the present invention will be described.
The image pickup apparatus of the present invention reduces image quality caused by various factors such as lens distortion generated in an optical image pickup system, camera shake caused by a photographer or a shooting situation, for example, parallel camera shake and rotary camera shake. Perform image correction for removal.

  In particular, in an apparatus having a plurality of imaging systems such as a stereo camera, distortion aberration correction, translational camera shake correction, rotation camera shake correction, and parallax correction for adjusting the positional relationship between output images are performed. . For example, by outputting an image generated by these corrections to a 3D image display device, a high-quality image can be comfortably viewed without feeling tired or uncomfortable.

  Each of the image correction processes such as distortion correction, translational camera shake correction, rotational camera shake correction, and parallax correction performed as a correction process on the image can be realized by image geometric conversion and coordinate conversion. Therefore, for example, by performing image correction by adding correction components to be executed according to each purpose, simultaneous processing of a plurality of types of correction is possible, and efficient correction can be realized.

  Hereinafter, a plurality of embodiments of the imaging apparatus of the present invention will be described. These embodiments include an embodiment in which each correction such as distortion aberration correction, parallel movement camera shake correction, rotational camera shake correction, and parallax correction is executed individually, and an embodiment in which these corrections are executed collectively. .

(1-1. Configuration of the first embodiment)
FIG. 1 is a block diagram showing an embodiment of an imaging apparatus of the present invention.
The imaging apparatus shown in FIG. 1 is a stereo camera, and has two systems of imaging units that capture a left-eye image and a right-eye image, that is, a first imaging unit 111 and a second imaging unit 121 as shown in FIG.

  The first imaging unit 111 collects incident light to form an image, and in addition to a lens group 112 (unit of one or more lenses), an image sensor 113 such as a CCD or CMOS on which the output light of the lens group 112 forms an image, a lens A lens driving unit 114 that drives a predetermined lens in the group 112 and reads the position thereof, a distortion data storage unit 115 that holds distortion data of the lens group 112, and an optical imaging unit are attached in the vicinity of the lens. The camera shake sensor 116 to detect is provided.

  The second imaging unit 121 has the same configuration as the first imaging unit 111, and includes a lens group 122, an image sensor 123, a lens driving unit 124, a distortion aberration data storage unit 125, and a camera shake sensor 126.

  The distortion aberration data stored in the distortion aberration data storage units 115 and 125 is data indicating distortion aberration modes corresponding to the lens groups 112 and 122 of the respective imaging units. The distortion data is a value unique to the lens group, or changes depending on the zoom position or the like. Therefore, each of the distortion aberration data storage unit 115 and the distortion aberration data storage unit 125 has several representatives from the wide-angle end to the telephoto end that are allowed as the zoom range of the lens group 112 and the lens group 122 of the respective imaging units. Discrete distortion data corresponding to a typical zoom position is stored. In addition to a plurality of zoom positions, distortion data corresponding to the focus position may be stored.

  In each of the imaging units 111 and 121, the position information of the zoom lens and the focus lens is given from the lens driving units 114 and 124 to the control unit 172. In addition, distortion data corresponding to the lens position is read from the distortion data storage units 115 and 125 and provided to the control unit 172. Based on these input values, the control unit 172 calculates a distortion aberration correction value for correcting the distortion aberration of the captured image, and outputs it as a distortion aberration correction parameter executed by the distortion aberration correction unit 153.

  The camera shake sensors 116 and 126 of the imaging units 111 and 121 are attached in the vicinity of the imaging unit, and output an electrical signal corresponding to a camera shake mode including a camera shake amount and a direction as camera shake data, and output it to the control unit 172. The control unit 172 adjusts the camera shake data input from the camera shake sensors 116 and 126 according to the positions of the zoom lens and the focus lens obtained from the lens driving units 114 and 124, and calculates the rotation camera shake correction value and the parallel camera shake correction value. These are output as correction parameters to the rotation camera shake correction unit 154 that executes the rotation camera shake correction and to the translation camera shake correction unit 155 that executes the translation camera shake correction.

  In the configuration of the imaging apparatus shown in FIG. 1, two imaging units, that is, a first imaging unit 111 and a second imaging unit 121, respectively, a right camera that captures a right-eye image and a left-eye image that are used for 3D image display. The left camera to shoot. In such a setting, the output signals of the image sensors 113 and 123 are read out by the respective imaging units 111 and 121, and are output to a set of camera signal processing units 151 while being switched by the switching multiplexing unit 141 at regular intervals. To convert the image signal into a predetermined format.

Correction processing is sequentially performed on the image signal input to the camera signal processing unit 151 in the zoom magnification correction unit 152 to the parallax correction unit 157 shown in the drawing.
Some of the correction parameters executed by the zoom magnification correction unit 152 to the parallax correction unit 157 are calculated by the control unit 172. The control unit 172 inputs the following data from the imaging units 111 and 121.
Zoom / focus position information from the lens driving units 114 and 124,
Distortion data from the distortion data storage units 115 and 125,
Camera shake data from the camera shake sensors 116 and 126,
In addition,
Parallax data from the optical system parallax data storage 131,
Zoom magnification correction data from the zoom magnification correction value storage unit 132,
Optical axis center correction data from the optical axis center correction value storage unit 133,
Each of these data is input, a correction parameter to be executed in the zoom magnification correction unit 152 to the parallax correction unit 157 is calculated, and a correction parameter to be executed in the zoom magnification correction unit 152 to the parallax correction unit 157 is provided.

  The zoom magnification correction unit 152 to the parallax correction unit 157 apply the correction parameters input from the control unit 172, and execute image correction processing. The corrected image executed in each of the zoom magnification correction unit 152 to the parallax correction unit 157 is temporarily stored in the image data storage unit 171, and each correction unit receives the correction result of the preceding correction unit from the image data storage unit 171. Remove and execute correction.

  Note that the zoom magnification correction unit 152 to the parallax correction unit 157 perform correction processing by alternately switching the captured image of the first imaging unit 111 and the captured image of the second imaging unit 121 at predetermined intervals. That is, the switching multiplexing unit 141 alternately switches and outputs the captured image of the first imaging unit 111 and the captured image of the second imaging unit 121 at predetermined intervals. The control unit 172 calculates a parameter to be applied to correction of each image, for example, a correction vector according to the switching timing, and provides the calculated parameter to the zoom magnification correction unit 152 to the parallax correction unit 157.

  The zoom magnification correction unit 152 executes correction such as image enlargement / reduction according to the zoom magnification at the time of image shooting. Specifically, the process of matching the zoom magnifications of a plurality of images made up of a pair of images taken by the first imaging unit 111 and the images taken by the second imaging unit 121, that is, matching the characteristics of the plurality of images. This is image characteristic matching correction processing.

  The control unit 172 acquires zoom magnification correction data from the zoom magnification correction value storage unit 132 based on the zoom magnification information input from the lens driving units 114 and 124, and provides this to the zoom magnification correction unit 152 as a correction parameter. The zoom magnification correction unit 152 performs correction by applying the input parameter.

  When there are a plurality of imaging units as shown in FIG. 1, even if the user performs shooting at a certain zoom position, a slight difference occurs in the position of the zoom lens of each imaging unit or the optical zoom magnification, resulting in a result. A plurality of images may be slightly different from each other. When generating a 3D image (stereoscopic image) using the captured image of the first imaging unit 111 and the captured image of the second imaging unit 121 as illustrated in FIG. 1, it is important to reduce this difference in zoom magnification as much as possible. It becomes.

The zoom magnification correction value storage unit 132 holds a correction value for reducing the difference in zoom magnification as much as possible. The control unit 172 reads a correction value corresponding to the state of the zoom lens or the focus lens from the zoom magnification correction value storage unit 132, calculates a zoom magnification correction parameter based on the read value, and provides the calculated value to the zoom magnification correction unit 152. The zoom magnification correction unit 152 executes image enlargement or reduction processing according to the zoom magnification correction parameter, and generates a zoom magnification correction image corresponding to the zoom magnification set at the time of shooting. The zoom magnification correction unit 152 performs processing on only one or each of the captured images of the plurality of imaging units, and performs adjustment to match (match) the respective magnifications.

  The distortion correction unit 153 performs correction of lens distortion generated in the captured image. The parameter applied to the correction process is calculated by the control unit 172. The control unit 172 reads necessary distortion aberration data from the distortion aberration data storage units 115 and 125 according to the zoom lens position information obtained from the lens driving units 114 and 124, and calculates distortion aberration correction parameters. The calculated parameter is provided to the distortion aberration correcting unit 153, and the distortion aberration correcting unit 153 performs correction by applying the input parameter. By this distortion correction, the coordinates of each pixel are converted to coordinates that should be imaged without distortion, and an image signal that is more faithful to the shape of the subject is obtained with distortion removed or reduced. .

The rotational camera shake correction unit 154 corrects the camera shake in the rotation direction for the image signal in which the distortion is corrected.
Further, the translational camera shake correction unit 155 corrects vertical and horizontal translational camera shake to stabilize the image.
The control unit 172 calculates parameters to be applied to these camera shake corrections and provides them to the rotating camera shake correction unit 154 and the translational camera shake correction unit 155.

  The control unit 172 reads the rotation camera shake data and the parallel movement camera shake data from the camera shake sensors 116 and 126, and inputs the zoom position / focus position obtained from the lens driving units 114 and 124. Based on these input values, the control unit 172 calculates a correction parameter to be applied to the rotation camera shake correction and the parallel movement camera shake correction. Note that the zoom position / focus position obtained from the lens driving units 114, 124 and the rotational shake data / translational shake data obtained from the shake sensors 116, 126 are switched and multiplexed between the first imaging unit 111 and the second imaging unit 121. The data to be read is switched in conjunction with the conversion.

  Due to the camera shake correction performed by the rotation camera shake correction unit 154 and the parallel movement camera shake correction unit 155, various shakes mixed in the camera shake are decomposed into the camera shake in the rotation direction and the camera shakes in the vertical and horizontal translations. Blur is removed or reduced. In the output image, a stationary object is in a state where it can be viewed in a stationary manner, and a high-quality image with a sense of stability for the user is obtained.

The optical axis center correction unit 156 corrects the optical axis center for a set of output images obtained from the two imaging units 111 and 121.
A set of output images obtained from the two imaging units 111 and 121 is viewed by a user as a stereoscopic image (3D image) by executing a 3D image display process such as alternating switching display in the image display unit 159, for example. Can do. However, there may be a slight difference in the image depending on the mounting position of the optical imaging system of the first imaging unit 111 and the second imaging unit 121, and there may be a shift in the optical axis center. The optical axis center correction unit 156 corrects the optical axis center related to a set of output images obtained from the two imaging units 111 and 121 so that this becomes an optimal positional relationship. Specifically, the process of matching the optical axis centers of a plurality of images made up of a pair of images captured by the first imaging unit 111 and images captured by the second imaging unit 121, that is, the characteristics of the plurality of images are matched. The image characteristic matching correction process is executed.

  The control unit 172 acquires the optical axis center correction data from the optical axis center correction storage unit 133 and provides it to the optical axis center correction unit 156, and the optical axis center correction unit 156 performs correction based on this correction value.

  In the configuration having a plurality of imaging units as shown in FIG. 1, the centers of the optical axes passing through the lens unit are slightly shifted from each other on the image sensor due to the mounting (assembly) accuracy of the lens unit and the image sensor. , Does not coincide with the center of the image obtained. In some cases, the optical center coordinates do not coincide with each other in a plurality of obtained images. Since optical zoom and lens distortion aberration are based on the center of the optical axis, this is a process of matching them, that is, as a process of matching the optical center coordinates of the captured image of the imaging unit 111 and the captured image of the imaging unit 121. Optical axis center correction processing is required.

  The optical axis center correction value storage unit 133 holds a correction value that is optical axis center correction data for this correction. The control unit 172 reads the correction value from the optical axis center correction value storage unit 133 according to the state of the zoom lens / focus lens, and provides correction parameters provided to the optical axis center correction unit 156, for example, captured images of the two imaging units. A correction vector for matching the centers of the two is calculated. The optical axis center correction unit 156 performs processing on one or each of the captured images of the plurality of imaging units, and performs correction processing to match the optical axis center coordinates with each other.

  The parallax correction unit 158 performs parallax correction regarding a set of output images obtained from the two imaging units 111 and 121. The parallax correction unit 158 performs parallax correction based on the parallax correction parameter calculated by the control unit 172. By using the focus position and zoom position obtained from the lens driving units 114 and 124, it is possible to know how far the main subject is from the imaging apparatus. The control unit 172 calculates this distance, reads the parallax data from the optical system parallax data storage unit 131 according to the distance between the two imaging units 111 and 121, calculates the parallax correction parameter, and sends it to the parallax correction unit 158. provide. The parallax correction unit 158 performs parallax correction for correcting the parallax distortion and the like using the input parameters.

  With these corrections, the user can comfortably view the stereoscopic image, and the feeling of fatigue and discomfort can be minimized. Note that as the parallax distortion correction processing executed in the parallax correction unit 158, for example, the processing described in published patent publication 2008-524673 can be applied.

  The 3D image (stereoscopic image) thus obtained is displayed to the user by the image display unit 159. Further, the data compression unit 158 performs compression processing as necessary to reduce the capacity, and the data is output to the image storage medium / input / output terminal 160 to be subjected to storage processing or external output processing.

  As shown in FIG. 1, each correction processing unit exchanges control data and setting parameters with the control unit 172. Each correction unit acquires a correction target image from the image data storage unit 171 and stores a correction result. Each correction processing unit may be set to directly transfer image data, or may be set to transfer via the image data storage unit 171.

(1-2. Configuration of the second embodiment)
The imaging apparatus shown in FIG. 2 is a second embodiment of the imaging apparatus according to the present invention. The imaging device shown in FIG. 2 has a configuration in which each of the two imaging units individually includes a correction processing unit after the camera signal processing unit.

  That is, with respect to an output signal from the first imaging unit 111, a camera signal processing unit 151, a zoom magnification correction unit 152, a distortion aberration correction unit 153, a rotation camera shake correction unit 154, a translational camera shake correction unit 155, and an optical axis center correction. Correction processing is executed in the unit 156 and the parallax correction unit 157.

On the other hand, with respect to the output signal from the second imaging unit 121, the camera signal processing unit 201, the zoom magnification correction unit 202, the distortion aberration correction unit 203, the rotation camera shake correction unit 204, the parallel movement camera shake correction unit 205, and the optical axis center correction. Correction processing is executed in the unit 206 and the parallax correction unit 207.
As described above, this embodiment has two processing systems corresponding to the respective imaging units.

  For example, distortion aberration data, rotating camera shake data, parallel movement camera shake data, etc. are acquired as individual data in each imaging unit, and these data are input to the control unit 172 to calculate correction parameters corresponding to each imaging unit, Provided to each correction unit of each processing system. As described above, an independent signal processing system corresponding to each imaging unit is provided, and the processing speed can be increased by operating in parallel. In the configuration shown in FIG. 2, the image data storage unit that stores the image of the correction process also includes an image data storage unit 171 corresponding to the output of the first imaging unit 111 and the second imaging unit 121 as shown in the figure. The image data storage unit 211 corresponding to the output is set individually.

(1-3. Configuration of Third Embodiment)
FIG. 3 shows a third embodiment of the imaging apparatus according to the present invention. The imaging apparatus illustrated in FIG. 3 includes the zoom magnification correction unit 152, the distortion correction unit 153, the rotation camera shake correction unit 154, the translational camera shake correction unit 155, the optical axis center correction unit 156, and the parallax correction unit 157 illustrated in FIG. In this configuration, two image conversion correction units 251 are replaced.

  The image magnification correction unit 152, the distortion correction unit 153, the rotation camera shake correction unit 154, the parallel movement camera shake correction unit 155, the optical axis center correction unit 156, and the parallax correction unit 157 illustrated in FIG. In 251, batch processing is performed. Since the corrections executed in the respective correction units shown in FIG. 1 are all a kind of coordinate conversion processing, any two or more or all of them can be processed simultaneously. Such simultaneous processing realizes simplification, cost reduction, and power reduction of the apparatus. In addition, since the plurality of correction units are used in common, the transfer of image data to and from the image data storage unit is remarkably simplified, leading to higher processing speed and lower power consumption of the apparatus.

  In the embodiment shown in FIG. 3, as in the configuration shown in FIG. 1, camera shake sensors 116 and 126 and distortion data storage units 115 and 125 are individually provided in the first imaging unit 111 and the second imaging unit 121, respectively. Yes. Similar to the configuration shown in FIG. 1, the reference destination is switched in synchronization with the switching multiplexing of the image signals. However, since there are two imaging units in the same imaging device, it is possible to regard the camera shake value and lens distortion characteristics as the same, and the camera shake sensor and the distortion aberration data storage unit must be set separately. It is also possible to simplify the configuration as a common configuration for each imaging unit. That is, only one camera shake sensor and one distortion aberration data storage unit may be set and used as information common to a plurality of imaging units.

[2. Details of processing in the apparatus of the present invention]
Next, details of individual processing executed in the image processing apparatus of the present invention will be described.
First, an example of distortion will be described with reference to FIG.
FIG. 4 shows an image in which lens distortion occurs.
(A) A barrel distortion in which the image is rounded and bent outward (b) shows a pincushion distortion that is bent so that the four corners of the image are stretched outward. Is also known. The distortion aberration correction unit or the image conversion correction unit of the image processing apparatus of the present invention described with reference to FIGS. 1 to 3 includes (a) barrel distortion, (b) pincushion distortion, or distortion in which these are mixed. Any of these images can be easily corrected in accordance with the value of distortion aberration data (parameter) stored in advance in the distortion aberration data storage unit.

FIG. 5 is a diagram illustrating an example of attaching a camera shake sensor.
As a shake detection device, there are a mechanical detection device such as a gyro sensor and a detection device that extracts a motion vector from a plurality of images by image processing. Any of the hand movement sensors in the present invention can be used.

As an example of a mechanical detection device such as a gyro sensor, a gyro sensor that knows the rotation speed with a columnar sensor as shown in FIG. 5 is widely known. In order to detect the values of rotating camera shake and parallel camera shake,
Translational camera shake (movement of the image in the X-axis direction) in which the shooting field angle of the imaging unit fluctuates left and right,
Parallel movement hand movement (movement in the Y-axis direction of the image)
Shake in the rotational direction around the axis parallel to the optical axis of the imaging unit (movement that rotates the image),
It is necessary to provide a shake sensor for each of these three axes, and to attach the three sensors so as to be orthogonal to each other. Also, it is necessary to obtain a more accurate camera shake sensor value by attaching it very close to the imaging unit. When there are two or more imaging units, either a camera shake sensor is attached at an optimum position for each of them, or one camera shake sensor is attached at a position suitable for both of the two imaging units, and the configuration is shared. .

  FIG. 6 shows the adjustment gain of the camera shake sensor value according to the zoom lens position. When the camera shake detection device is a mechanical detection device such as a gyro sensor and the camera shake correction unit is an electronic type (a method that shifts and cuts out image signals), the zoom lens position moves to the telephoto even with the same camera shake sensor value. The closer to the image, the larger the vertical and horizontal correction amounts with respect to the image. Therefore, a lookup table for the adjustment gain is required. This look-up table reference and hand shake value adjustment are performed by the control unit. As a prior art for the camera shake correction processing, for example, there is a registration No. 3279342.

  As another example of the camera shake detection apparatus, there is a motion vector detection apparatus. This is a process to which a method of storing a plurality of images obtained by photographing at regular time intervals and extracting a motion vector from the plurality of images by image processing is applied. Not only the translational motion vector but also the rotational motion vector can be known. This processing configuration is also described in, for example, Registration No. 4212109 (Panasonic) and Registration No. 4487811 (Sony).

FIG. 7A is a diagram illustrating an example of setting correction vectors for an image in which barrel distortion has occurred. A vector (arrow) shown in the image is a correction vector.
When barrel-shaped distortion occurs when shooting a grid-like object perpendicular to the vertical and horizontal directions, the captured image is in a state where the vertical and horizontal straight lines are rounded and bent like a barrel toward the outside of the lens. The vector for correcting this is a set value that enlarges the image to the outside of the lens according to the distance from the lens center (near the image center), as shown in FIG. Generally, the distance from the lens center increases the magnitude of the correction vector toward the outside of the lens.

FIG. 7B is a diagram illustrating an example of setting correction vectors for an image in which translational hand movement occurs. A vector (arrow) shown in the image is a correction vector.
The example shown in FIG. 7 (2) shows an example of setting correction vectors for an image in which a captured image is taken so as to flow from the upper right to the lower left to cause translational hand movement.

  Normally, a necessary image closer to the center is cut out from a slightly larger image on the image sensor, but if the image pickup device shakes up, down, left, or right due to camera shake, the entire image obtained by the image sensor is used. Shake can be canceled by shifting out the coordinates by the amount of shake. In other words, this is equivalent to correcting the entire image uniformly with vectors of the same size and direction.

  For detection of translational hand movement, it is only necessary to obtain values from the X-axis shake sensor and Y-axis shake sensor described above with reference to FIG. 5 to calculate correction values, and to shift the cutout position by that value. This shift of the cut-out position may be executed when the image sensor is read out. After reading the entire image from the image sensor and performing some processing, the write address or read address when storing in the image data holding unit is shifted. Sometimes it is done.

FIG. 8 is a diagram illustrating an example of setting a correction vector for an image in which a rotational camera shake has occurred. A vector (arrow) shown in each image is a correction vector.
The captured images in FIGS. 8A to 8E are all taken in a counterclockwise rotational camera shake on the imaging device side, and as a result, the images are captured in a state of being tilted clockwise. The correction vector setting changes depending on where the center of the image is provided in the image. Incidentally, the center of the rotation correction does not need to be the center of the actual rotation blur, and is provided virtually when setting the correction vector, so that there is basically no problem where it is provided in the image. The center of rotation correction is preferably not far from the image in order to maintain the accuracy of the set value of the correction vector, but may be set for convenience either inside or outside the image.

  In FIG. 8A, the center of the rotation correction is provided near the lower right corner of the image (may be inside or outside the image), and the correction vector is set counterclockwise around that point. . The magnitude of the correction vector increases with the distance from the center of rotation correction.

The center of the rotation correction is located at the lower left of the image in FIG. 8B, at the upper right of the image in (c), at the upper left of the image in (d), and substantially at the center of the image in (e).
As shown in FIGS. 8A to 8E, the direction and size of the correction vector change depending on the setting of the center position of the rotation correction even in an input image in which similar rotational camera shake occurs.

FIG. 9 is a diagram illustrating an example of parallax correction.
In FIG.
(A) Captured image of first imaging unit (b) Captured image of second imaging unit These are shown.

(A) A vector (arrow) shown in the captured image of the first imaging unit is a parallax correction vector.
According to the processing order in the embodiment of FIG. 1 described above, the input image of the parallax correction unit 157 is an image that has been subjected to each correction in the zoom magnification correction unit 152 to the optical axis center correction unit 156. At this time, the captured image of the first imaging unit 111 and the captured image of the second imaging unit 121 are images that need only parallax correction. The captured image of the first imaging unit 111 and the captured image of the second imaging unit 121 are slightly different from each other.

  The parallax correction unit 157 obtains correction data from the optical system parallax data storage unit 131 so that a stereoscopic image composed of both has a preferable parallax, and the captured image of the first imaging unit 111 or the second imaging unit 121 Parallax correction is performed on at least one of the captured images. Generally, parallax correction can be realized by parallel movement of an image, and enlargement / reduction processing is performed as necessary. Either the case where the parallax correction data is adjusted or the case where fixed data is used is applied according to the distance from the imaging device to each subject.

  For example, as can be understood from the description with reference to FIGS. 7 to 9, distortion aberration correction, parallel movement camera shake correction, rotational camera shake correction, and parallax correction are all processes that can be executed as coordinate conversion processing. Among them, the parallel movement camera shake correction can be realized by shifting the image by generating a correction vector uniformly over the entire screen by a combination of vertical and horizontal translation. In addition, since the correction vector is set according to the position in the image in both the distortion aberration correction and the rotational camera shake correction, the image conversion correction shown in FIG. It is possible to perform correction processing simultaneously in one correction unit such as the unit 251.

  FIG. 10 shows a correction vector applied to the barrel distortion correction described with reference to FIG. 7A and a correction applied to the rotation correction process when the rotation center shown in FIG. It is an example of a vector setting that integrates both vectors. A vector (arrow) shown in the image is a correction vector for executing barrel distortion correction and rotation correction together.

  For each coordinate in the screen, correction values can be integrated by adding a plurality of vectors as coordinate conversion. Various correction vectors are set as one integrated correction vector, and the image conversion shown in FIG. A plurality of types of correction processing can be performed simultaneously by executing coordinate transformation processing to which the integrated correction vector is applied in one correction unit such as the correction unit 251.

  The example shown in FIG. 10 shows an example of integration processing only for distortion aberration correction and rotational camera shake correction, but it is possible to integrate vectors including other correction vectors such as parallel camera shake correction and parallax correction. Yes, it is possible to simultaneously execute a plurality of types of correction processing by the coordinate conversion processing to which the integrated correction vector is applied.

FIGS. 11A and 11B are images before and after correction of an image in which barrel distortion and rotational camera shake have occurred.
Each of the two sets of optical imaging units captures an image including barrel distortion and rotational camera shake, and both images are slightly different due to the difference in their attachment positions.
As shown in FIG. 9 (a), the two shot images are rounded into a barrel shape due to lens distortion, and rotational camera shake occurs around an axis parallel to the lens optical axis. On the other hand, by performing barrel distortion correction and rotational camera shake correction, the object returns to a lattice-like object whose vertical and horizontal directions are orthogonal (or distortion is reduced). Furthermore, the binocular parallax is corrected so that the user can comfortably view the image, and the final output image is obtained.

[3. Image correction processing sequence in imaging apparatus]
Next, a sequence of image correction processing in the imaging apparatus of the present invention will be described.

(3-1. Correction sequence for sequentially executing individual correction processes)
12 and 13 are flowcharts for explaining the processing procedure of the image correction processing in the imaging apparatus of the present invention.
FIG. 12 is a processing flow corresponding to the apparatus shown in FIG. That is, it is a diagram illustrating a flow for explaining a sequence of image correction processing in an apparatus including an individual correction unit corresponding to each correction purpose.
FIG. 13 is a processing flow corresponding to the apparatus shown in FIG. That is, it is a diagram illustrating a flow for explaining a sequence of image correction processing in an apparatus including an image conversion correction unit 251 that collectively executes a plurality of corrections according to a plurality of correction purposes.

  First, with reference to the flow shown in FIG. 12, the processing of the apparatus shown in FIG. 1 or 2, that is, the procedure of image correction processing in an apparatus provided with an individual correction unit corresponding to each correction purpose will be described.

  First, in step S101, exposure is performed by the image sensors of the two imaging units, and in step S102, the imaging light is converted into an electrical signal and read from the image sensor. The read image from the image sensor is an image in which distortion occurs due to the characteristics of the lens of the imaging unit, and image blurring occurs due to user shake or various shooting situations. Further, there are cases where the optical zoom magnifications do not exactly match each other due to variations in the lens units of the imaging unit, and cases where the optical axis center coordinates do not coincide with each other due to variations in the attachment between the lens unit and the image sensor. Furthermore, the two images input from the image sensors of the two imaging units are images having parallax depending on the distance between the mounting positions of the imaging units.

  In step S103, in the configuration of FIG. 1, the two image signals captured by the two imaging units are output to the camera signal processing unit 151 while being switched at regular intervals by the switching multiplexing unit 141. In the case of the configuration of FIG. 2, two image signals photographed by the two imaging units are individually output to the respective camera signal processing units 151 and 201. In step S <b> 104, the camera signal processing unit executes processing for conversion into an image signal having a predetermined format for output to the correction unit.

  In step S105, the control unit reads the zoom lens position and the focus lens position from the lens driving unit.

  Next, in step S <b> 106, the control unit performs a data reading process necessary for the correction process according to these acquired values, and calculates correction values to be applied to correction parameter calculation in each correction unit. Furthermore, correction value interpolation processing is performed as necessary, that is, when correction values cannot be obtained directly from the correction value storage unit, interpolation processing for calculating necessary correction values from a plurality of correction values is executed. Interpolation processing is performed, for example, by linear interpolation processing using nearby correction values.

Specifically, for example, the following processing is executed.
Calculation of the zoom magnification correction value for obtaining the zoom / focus position from the lens driving unit and providing it to the zoom magnification correction unit,
Processing for reading out distortion aberration data corresponding to each lens position at the time of shooting from the distortion aberration data storage unit, that is, acquisition of distortion aberration data for calculating parameters to be applied to distortion aberration correction processing in the distortion aberration correction unit,
Acquisition of optical axis center correction data to be applied to correction in the optical axis center correction unit from the optical axis center correction value storage unit,
Acquisition of parallax data for provision to the parallax correction unit from the optical system parallax data storage unit,
Interpolation processing to calculate correction values that cannot be directly acquired,
These processes are executed.

For example, each component shown in FIG. 1 includes data to be applied for correction processing such as optical zoom magnification correction, lens distortion correction, optical axis center coordinate correction, and parallax correction performed in the apparatus of the present invention. The following data obtained from:
Zoom / focus position information acquired from the lens driving units 114 and 124,
Distortion data acquired from the distortion aberration data storage units 115 and 125;
Parallax data acquired from the optical system parallax data storage 131,
Zoom magnification correction data acquired from the zoom magnification correction value storage unit 132;
Optical axis center correction data acquired from the optical axis center correction value storage unit 133,
These data are used.

  The distortion aberration data storage units 115 and 125, the optical system parallax data storage unit 131, the zoom magnification correction value storage unit 132, and the optical axis center correction value storage unit 133 correspond to the positions of various zoom lenses and focus lenses in advance. Data is stored.

  FIG. 14 is a diagram showing how these data are stored discretely. In the configuration in which the zoom lens and the focus lens move from end to end of the movable range, several representative points (pos0 to posN) of the lens position are provided, and each correction value is discretely held for that point. The example shown in the figure is an example in which the representative points (posX) are set at equal intervals. Data as correction values corresponding to these representative points are discretely converted into distortion data storage units 115 and 125, an optical system parallax data storage unit 131, a zoom magnification correction value storage unit 132, and an optical axis center correction value storage unit 133. Store it in.

  It should be noted that the representative points may be equally spaced or densely provided in a range where the data change rate is large. For any lens position other than the representative point, a correction value is calculated by interpolation processing from a neighboring set value. The interpolation value calculation is executed by, for example, the control unit.

  As shown in FIG. 14, for example, the correction value of representative point 1 (pos1) and the correction value of representative point 2 (pos2) are obtained from the data storage unit. The lens position at the time of actual photographing is representative point 1 (pos1). ) And the representative point 2 (pos2), the correction values of the representative point 1 (pos1) and the representative point 2 (pos2) are acquired and the representative point 1 ( The correction value corresponding to the lens position between pos1) and the representative point 2 (pos2) is calculated by primary interpolation. Alternatively, the number of points may be further increased, such as representative point 0 (pos0) and representative point 3 (pos3), and obtained by high-order interpolation.

  Note that the parallax data read from the optical system parallax data storage unit is parallax data corresponding to the subject distance. The control unit acquires each lens position information at the time of shooting from the lens driving unit, calculates the distance to the main subject in the captured image based on the acquired lens position information, and adjusts the binocular parallax according to the distance Parallax data for this is acquired from the parallax data storage unit.

  Returning to the flowchart of FIG. After the processing of step S106 is completed, in steps S107 to S108, the camera shake sensor values for the three channels in the vertical direction, the horizontal direction, and the rotation direction are read from the camera shake sensor, and the zoom lens position described above with reference to FIG. Based on the relationship data with the adjustment gain of the camera shake sensor value, etc., the camera shake sensor value acquired according to each lens position is adjusted, and a correction parameter to be provided to the rotating camera shake correction unit and the parallel camera shake correction unit is calculated. The adjustment value of the camera shake sensor value is calculated.

  As described above with reference to FIG. 6, even when the camera shake sensor value is the same, the camera shake in the vertical and horizontal directions becomes larger in the image as the zoom lens position is on the telephoto side. The control unit reads the adjustment gain value shown in FIG. 6 from, for example, a lookup table previously stored in a memory, and multiplies the camera shake sensor read value to adjust the input value from the camera shake sensor.

  The processing in steps S109 to S114 is correction parameter calculation processing applied to the correction processing in each correction unit shown in FIGS. This parameter calculation process is executed in the control unit. In addition, it is good also as a structure which provides the data required for parameter calculation to each correction | amendment part from a control part, and calculates a parameter in each correction | amendment part.

In step S109, a zoom magnification correction parameter in the zoom magnification correction unit is calculated. The zoom magnification correction parameter is set based on a zoom magnification correction value calculated by the control unit based on zoom / focus position information acquired from the lens driving unit.
In step S110, a distortion correction parameter in the distortion correction unit is calculated. This distortion aberration correction parameter is calculated based on the distortion aberration data corresponding to each lens position at the time of photographing read from the distortion aberration data storage unit.

  In step S111, a rotation camera shake correction parameter in the rotation camera shake correction unit is calculated. This rotation camera shake correction parameter is the relationship between the adjustment value of the camera shake sensor value in the rotation direction read from the camera shake sensor in steps S107 to S108, that is, the relationship between the zoom lens position and the adjustment gain of the camera shake sensor value described with reference to FIG. It is calculated based on an adjustment value adjusted based on data or the like.

  In step S112, a translational camera shake correction parameter in the translational camera shake correction unit is calculated. This parallel movement camera shake correction parameter is the adjustment value of the vertical and horizontal camera shake sensor values read from the camera shake sensor in steps S107 to S108, that is, the zoom lens position and camera shake sensor value adjustment gain described with reference to FIG. Is calculated based on the adjustment value adjusted based on the relationship data.

  In step S113, an optical axis center correction parameter in the optical axis center correction unit is calculated. The optical axis center correction parameter is calculated based on the optical axis center correction data read from the optical axis center correction value storage unit.

  In step S114, a parallax correction parameter in the parallax correction unit is calculated. The parallax correction parameter is calculated based on the parallax data read from the optical system parallax data storage unit. As described above, the parallax data read from the optical system parallax data storage unit is used for binocular parallax adjustment according to the distance to the main subject in the captured image calculated by the control unit based on each lens position information at the time of shooting. It is data of. The parallax correction parameter is calculated based on parallax data corresponding to the subject distance.

The next processing in steps S115 to S120 is an execution step of correction processing in each correction unit shown in FIGS.
In steps S115 to S120, the following correction processes are sequentially executed.
In step S115, the zoom magnification correction process is performed by applying the zoom magnification correction parameter in the zoom magnification correction unit.
In step S116, a distortion aberration correction process using the distortion aberration correction parameter is executed in the distortion aberration correction unit.

In step S117, the rotating camera shake correction unit executes the rotating camera shake correction process to which the rotating camera shake correction parameter is applied.
In step S118, the parallel camera shake correction unit executes the parallel camera shake correction process to which the parallel camera shake correction parameter is applied.

In step S119, an optical axis center correction process in which the optical axis center correction parameter is applied in the optical axis center correction unit is executed.
In step S120, the parallax correction process which applied the parallax correction parameter in a parallax correction part is performed.

  In the case of the configuration shown in FIG. 1, the correction processes in steps S <b> 116 to S <b> 120 are performed while alternately switching the captured image of the first imaging unit 111 and the captured image of the second imaging unit 121. On the other hand, in the case of the configuration of FIG. 2, correction processing is performed in parallel on the captured image of the first imaging unit 111 and the captured image of the second imaging unit 121 in each correction unit.

If each correction process of step S116-S120 is completed, it will progress to step S122.
In step S122, image display processing of the 3D image to which the corrected image based on the captured image of the first imaging unit 111 and the corrected image based on the captured image of the second imaging unit 121 are applied is executed using the image display unit. . Alternatively, the corrected set of image data is compressed and recorded on an image storage medium, or output from the output terminal to the outside of the apparatus.

(3-2. Correction Sequence for Executing Multiple Correction Processes as Batch Correction Process)
Next, in an apparatus including an image conversion correction unit 251 that collectively executes a plurality of corrections corresponding to a plurality of correction purposes, corresponding to the apparatus illustrated in FIG. 3 with reference to the flowchart illustrated in FIG. A sequence of image correction processing will be described.

  The processes in steps S201 to S208 are the same as the processes in steps S101 to S108 described with reference to FIG.

First, in step S201, exposure is performed by the image sensors of the two imaging units, and in step S202, the imaging light is converted into an electrical signal and read from the image sensor.
In step S203, the two image signals captured by the two imaging units are output to the camera signal processing unit 151 while being switched at regular intervals in the switching multiplexing unit 141 having the configuration of FIG.

In step S <b> 204, the camera signal processing unit executes processing for conversion into an image signal having a predetermined format for output to the correction unit.
In step S205, the control unit reads the zoom lens position and the focus lens position from the lens driving unit.
Next, in step S206, the control unit performs reading processing of data necessary for the correction processing according to these acquired values, and calculation of correction values to be applied to calculation of correction parameters in each correction unit. Furthermore, an interpolation process for calculating a correction value that cannot be directly acquired from the correction value data storage unit or the like is executed as necessary.

The process of step S206 is, for example, the following process as described above as the process of step S106 of FIG.
Calculation of the zoom magnification correction value for obtaining the zoom / focus position from the lens driving unit and providing it to the zoom magnification correction unit,
Processing for reading out distortion aberration data corresponding to each lens position at the time of shooting from the distortion aberration data storage unit, that is, acquisition of distortion aberration data for calculating parameters to be applied to distortion aberration correction processing in the distortion aberration correction unit,
Acquisition of optical axis center correction data to be applied to correction in the optical axis center correction unit from the optical axis center correction value storage unit,
Acquisition of parallax data for provision to the parallax correction unit from the optical system parallax data storage unit,
Interpolation processing to calculate correction values that cannot be directly acquired,
These processes are executed.

  Next, in steps S207 to S208, the camera shake sensor values for the three channels in the vertical direction, the horizontal direction, and the rotation direction are read from the camera shake sensor, and the zoom lens position and the camera shake sensor value described above with reference to FIG. 6 are adjusted. Adjusting the camera shake sensor value to calculate the correction parameters provided to the rotating camera shake correction unit and the parallel camera shake correction unit by adjusting the camera shake sensor value acquired according to each lens position based on the relationship data with the gain, etc. Calculate the value.

  The processing of the next steps S209 to S210 is processing specific to the configuration of FIG. When each correction unit includes an image conversion correction unit 251 that is shared as shown in FIG. 3, the control unit integrates a plurality of calculated correction parameters after calculation of correction parameters according to individual correction purposes. A process for calculating one correction parameter is executed.

That is, the control unit calculates an integrated correction parameter obtained by integrating the following correction parameters applied to the correction process in each correction unit shown in FIGS.
(A) Zoom magnification correction parameter applied to zoom magnification correction processing in the zoom magnification correction unit (b) Distortion aberration correction parameter applied to distortion aberration correction processing in the distortion aberration correction unit (c) Rotation camera shake correction processing in the rotation camera shake correction unit (D) Parallel camera shake correction parameter applied to parallel camera shake correction process in parallel camera shake correction unit (e) Optical axis center correction parameter applied to optical axis center correction process in optical axis center correction unit (f) ) Parallax correction parameters applied to the parallax correction process in the parallax correction unit An integrated correction parameter is calculated by integrating the correction parameters applied to each of these correction processes.

In order to integrate the correction parameters, the sum of the respective coordinate conversion vectors may be obtained.
That is,
Integrated correction vector = zoom magnification correction vector + distortion aberration correction vector + rotation camera shake correction vector + translational camera shake correction vector + optical axis center correction vector + parallax correction vector Integration by integrating a plurality of correction vectors by adding such correction vectors A correction vector is calculated.
The image conversion correction unit 251 performs coordinate conversion processing using the integrated correction vector.
Note that the integrated correction vector, which is an integration parameter, is calculated for each captured image such as a captured image of the first imaging unit 111 and a captured image of the second imaging unit 121.

  The control unit 172 in FIG. 3 calculates an integrated correction parameter obtained by integrating the correction parameters (a) to (f) and provides the integrated correction parameter to the image conversion correction unit 251. Note that the control unit 172 only obtains or calculates data necessary for the integrated parameter calculation process, and provides the data to the image conversion correction unit 251 from the control unit to provide to the image conversion correction unit 251. It may be configured to execute the calculation of the integrated correction parameter.

In step S211, the image conversion correction unit 251 executes image correction processing to which the integrated correction parameter is applied.
In step S212, the correction image generated by the image correction process applied with the integrated correction parameter for the captured image of the first imaging unit 111 and the image correction process applied with the integrated correction parameter for the captured image of the second imaging unit 121 are generated. The image display process of the 3D image to which the corrected image is applied is executed by applying the image display unit. Alternatively, the corrected set of image data is compressed and recorded on an image storage medium, or output from the output terminal to the outside of the apparatus.

[4. About correction vectors as correction parameters according to the correction purpose]
Next, a correction vector as a correction parameter corresponding to the correction purpose will be described.

The correction processing for the captured image executed in the apparatus shown in FIGS. 1 to 3 includes the following correction processing: (a) Zoom magnification correction processing (b) Distortion aberration correction processing (c) Rotation camera shake correction processing (d) Parallel camera shake correction process (e) Optical axis center correction process (f) Parallax correction process These correction processes.
As described above with reference to FIG. 3, any of these correction processes can be executed as a coordinate conversion process for a captured image.

  As a correction parameter applied to the corrections (a) to (f), a coordinate conversion vector for executing a coordinate conversion process can be used. That is, it is a vector for converting each pixel position of the image before correction into a pixel position of the image after correction.

In the apparatus configuration of FIG. 1 or FIG. 2, the following individual vectors are calculated as correction parameters in each correction unit, and coordinate conversion using the correction vector is executed in each correction unit.
(A) Correction vector as a zoom magnification correction parameter applied to zoom magnification correction processing (b) Correction vector as a distortion aberration correction parameter applied to distortion aberration correction processing (c) Rotation camera shake correction parameter applied to rotation camera shake correction processing (D) Correction vector as parallel camera shake correction parameter applied to parallel camera shake correction processing (e) Correction vector as optical axis center correction parameter applied to optical axis center correction processing (f) Application to parallax correction processing Correction vectors as parallax correction parameters to be performed Hereinafter, examples of correction vectors applied to the correction processes (a) to (f) will be described.

(4-1. (A) Correction vector as a zoom magnification correction parameter applied to zoom magnification correction processing)
FIG. 15 shows an example of a correction vector as a zoom magnification correction parameter applied to the zoom magnification correction processing. A vector (arrow) shown in the image is a correction vector.

  The zoom magnification correction vector is a vector as shown in FIG. For example, the pixel position for adjusting (matching) the optical zoom magnification slightly different between a plurality of images obtained from a plurality of imaging units such as the imaging unit 111 and the imaging unit 121 shown in FIG. The vector to be moved is used for zoom magnification correction processing. The vector shown in FIG. 15 is an example of a zoom magnification correction vector, and is an example of one vector setting for performing enlargement processing by image processing. By coordinate conversion processing using this zoom magnification correction vector, a plurality of images obtained from a plurality of image capturing units such as the image capturing unit 111 and the image capturing unit 121 shown in FIG. 1 are adjusted to the same zoom magnification at any zoom position. be able to. When the image is reduced, a correction vector is set in a direction opposite to that in FIG. 13A, that is, from the periphery toward the center.

  Note that FIG. 15 shows only 14 vectors in the image, but this shows representative vectors corresponding to representative points set in the image. A vector corresponding to a pixel position of an arbitrary coordinate other than the representative point is calculated by interpolation processing from a vector of neighboring representative points, and coordinate conversion is executed based on the vector. The same applies to correction processing using other correction vectors described below.

A representative point setting example and an interpolation vector generation processing example will be described with reference to FIG.
FIG. 16A is an example in which representative points are provided in a lattice shape, and the representative points are regularly arranged vertically and horizontally. For arbitrary coordinates other than the representative point, a vector is calculated by interpolation processing from a vector of neighboring representative points.

  FIG. 16B shows an example in which representative points are provided concentrically around the optical axis, focusing on the fact that the distortion of the lens changes with the image height (distance from the center of the optical axis). Near the optical axis and near the center of the image, the vector is small, and the vector tends to increase and the rate of change increases toward the outside of the image. Therefore, it is suitable to provide the representative points denser toward the outside of the image. Again, for any coordinate other than the representative point, the vector is calculated by interpolation processing from a nearby vector or by calculating the distance from the center of the optical axis.

(4-2. (B) Correction vector as distortion aberration correction parameter applied to distortion aberration correction processing)
The correction vector as the distortion correction parameter applied to the distortion correction process has been described with reference to FIG.
As described above with reference to FIG. 7A, the vector shown in FIG. 7A is an example of setting a correction vector for an image in which barrel distortion has occurred. A vector (arrow) shown in the image is a correction vector.

  When barrel-shaped distortion occurs when shooting a grid-like object perpendicular to the vertical and horizontal directions, the captured image is in a state where the vertical and horizontal straight lines are rounded and bent like a barrel toward the outside of the lens. The vector for correcting this is a set value that enlarges the image to the outside of the lens according to the distance from the lens center (near the image center), as shown in FIG. Generally, the distance from the lens center increases the magnitude of the correction vector toward the outside of the lens.

  As described above with reference to FIG. 4, the distortion includes a barrel-type distortion, a pincushion-type distortion, and a mixed type (for example, Jinkasa-type distortion). These distortion aberration modes differ depending on the lens configuration of the imaging apparatus and each lens position at the time of shooting. The control unit reads optimal distortion aberration data corresponding to the lens position at the time of photographing from the distortion aberration data storage unit, and sets a distortion aberration correction vector based on the read data.

(4-3. (C) Correction vector as rotation camera shake correction parameter applied to rotation camera shake correction processing)
Next, a correction vector as a rotation camera shake correction parameter applied to the rotation camera shake correction process will be described.

The correction vector as the rotation camera shake correction parameter applied to the rotation camera shake correction process has also been described with reference to FIG.
In the examples described with reference to FIGS. 8A to 8E, all of the imaging apparatus side causes a counterclockwise rotational camera shake, and as a result, the rotation is performed when the image is captured in a clockwise tilt state. It is an example of setting of a camera shake correction vector.

As described above with reference to FIG. 8, the setting of the correction vector changes depending on where the center of rotation correction is provided in the image, but the center of rotation correction does not have to be the center of the actual rotation blur. This is virtually provided when the vector is set. The center of rotation correction is preferably not far from the image in order to maintain the accuracy of the set value of the correction vector, but may be set for convenience either inside or outside the image.
FIGS. 8A to 8E show vector setting examples based on the setting of rotation correction centers at different positions.

  The control unit reads the camera shake sensor value in the rotation direction from the camera shake sensor, and responds to each lens position based on the relationship data between the zoom lens position and the shake sensor value adjustment gain described above with reference to FIG. By adjusting the camera shake sensor value acquired in this way, an adjustment value of the camera shake sensor value is calculated. Based on this calculated value, for example, a rotation camera shake correction vector corresponding to the representative point shown in FIG. 8 is calculated.

  As described above with reference to FIG. 6, even when the camera shake sensor value is the same, the camera shake in the vertical and horizontal directions becomes larger in the image as the zoom lens position is on the telephoto side. The control unit reads the adjustment gain value shown in FIG. 6 from, for example, a lookup table previously stored in a memory, and multiplies the camera shake sensor read value to adjust the input value from the camera shake sensor.

(4-4. (D) Correction vector as parallel camera shake correction parameter applied to parallel camera shake correction process)
Next, correction vectors as parallel camera shake correction parameters applied to the parallel camera shake correction processing will be described.
This parallel camera shake correction vector has also been described with reference to FIG.
FIG. 7B is a diagram illustrating an example of setting a correction vector for an image in which translational camera shake has occurred. A vector (arrow) shown in the image is a correction vector. The example shown in FIG. 7 (2) shows an example of setting correction vectors for an image in which a captured image is taken so as to flow from the upper right to the lower left to cause translational hand movement.

  As described above with reference to FIG. 7 (2), when the imaging device is shaken vertically and horizontally due to camera shake or the like, the coordinates are shifted by the amount of shake from the entire image obtained by the image sensor. The shake is canceled by putting it out. The translational camera shake correction vector corresponds to a vector for determining the cutout position. The vector is set by obtaining a value indicating the amount of movement from each of the X-axis shake sensor and the Y-axis shake sensor described above with reference to FIG. 5, and calculating the shift direction and shift amount of the cutout position based on these values. , Based on this calculated value.

(4-5. (E) Correction vector as optical axis center correction parameter applied to optical axis center correction processing)
Next, a correction vector as an optical axis center correction parameter applied to the optical axis center correction process will be described with reference to FIG.

FIG. 17 shows a setting example of a correction vector as an optical axis center correction parameter applied to the optical axis center correction process.
The optical axis center correction vector shown in FIG. 17 adjusts the optical axis center coordinates slightly different among a plurality of images obtained from a plurality of imaging units to match (matches) the other image. This is an example in which a vector to be output is set. By the coordinate conversion process using this vector, the center coordinates of a plurality of images can be matched according to the state of the zoom or focus lens.

(4-6. (F) Correction vector as parallax correction parameter applied to parallax correction processing)
Next, a correction vector as a parallax correction parameter applied to the parallax correction process will be described.
The parallax correction vector is as described above with reference to FIG.
In FIG.
(A) Captured image of first imaging unit (b) Captured image of second imaging unit These are shown. (A) A vector (arrow) shown in the captured image of the first imaging unit is a parallax correction vector. For example, in the configuration of FIG. 1, the first imaging unit 111 and the second imaging unit 121 have different lens mounting positions, and the captured image of the first imaging unit 111 and the captured image of the second imaging unit 121 are slightly different images. It becomes.

  The parallax correction unit 157 obtains correction data from the optical system parallax data storage unit 131 so that a stereoscopic image composed of both has a preferable parallax, and the captured image of the first imaging unit 111 or the second imaging unit 121 It is necessary to perform parallax correction on at least one of the captured images. The vector applied to this correction process is, for example, the vector shown in FIG. 9i. Generally, parallax correction can be realized by parallel movement of an image, and enlargement / reduction processing is performed as necessary. Either the case where the parallax correction data is adjusted or the case where fixed data is used is applied according to the distance from the imaging device to each subject.

  As described above, in the apparatus configuration of FIG. 1 or FIG. 2, individual correction vectors corresponding to each correction purpose are calculated as correction parameters in each correction unit, and coordinate conversion using the correction vectors is sequentially performed in each correction unit. Run.

FIG. 18 shows a processing example in the case where the correction processing using the correction vector by each correction unit is executed in the apparatus configuration shown in FIG. 1 or FIG.
FIG. 18 shows an example of a correction vector and an output image when the following correction processing is sequentially executed according to the apparatus configuration shown in FIG. 1 or FIG.
(A) Zoom magnification correction processing (b) Distortion aberration correction processing (c) Rotational camera shake correction processing (d) Parallel camera shake correction processing (e) Optical axis center correction processing (f) Parallax correction processing

As shown in FIG.
Image series (1a) to (1f) to (1OUT) are correction processes for the output image of the first imaging unit 111 shown in FIGS.
Image series (2a) to (2f) to (2OUT) are correction processes for the output image of the second imaging unit 121 shown in FIG. 1 and FIG.
These are shown.
A vector (arrow) in each image is a correction vector of a representative point applied in each correction process.

First, zoom magnification correction in which a zoom magnification correction vector is applied to an output image of two imaging units, that is, the first imaging unit 111 and the second imaging unit 121 illustrated in FIG. 1 or 2. Execute the process.
This processing is shown by (1a) and (2a) in FIG.
(1a) shows an output image of the first imaging unit 111 and a zoom magnification correction vector for the image.
(2a) shows an output image of the second imaging unit 121.

The zoom magnification correction process is the process described above with reference to FIG. 15 and the like, and is a process for matching the zoom magnifications of the two output images. By executing the enlargement or reduction process for one of the images, It becomes possible to process.
In the example illustrated in FIG. 18, image conversion is performed by setting a correction vector only for the output image of the first imaging unit 111. The correction vector = 0 is set for the output image of the second imaging unit 121.

  The zoom magnification correction results obtained by applying the vectors shown in FIGS. 18A and 18A to the images shown in FIGS. 18A and 18B become (1b) and (2b), respectively, and the next distortion correction process is performed.

18 (1b) and (2b) show the following processes, respectively.
(1b) Setting example of distortion aberration correction vector for zoom magnification correction result image of output image of first imaging unit 111 (2b) Setting example of distortion aberration correction vector for zoom magnification correction result image of output image of second imaging unit 121 In this example, zoom magnification correction for the output image of the second imaging unit 121 is not substantially executed, so the images (2a) and (2b) are the same unprocessed images.

The distortion aberration correction is, for example, the correction described above with reference to FIG. For example, as shown in FIG. 7A, the correction vector for an image in which barrel distortion occurs is a set value that enlarges the image to the outside of the lens according to the distance from the lens center (near the image center). . Generally, the distance from the lens center increases the magnitude of the correction vector toward the outside of the lens.
The vectors shown in FIGS. 18 (1b) and (2b) have the same settings as the vectors shown in FIG. 7 (1), and show examples of correction vectors for an image in which barrel distortion has occurred.

  A distortion correction process is performed on the images shown in FIGS. 18 (1b) and (2b) by applying the vectors shown in the images. The correction results are (1c) and (2c), respectively. When the distortion is corrected, the curved portion returns to a straight line, and a lattice-like object is reproduced.

The following rotational camera shake correction processing is performed on the images (1c) and (2c) that are the distortion correction results.
18 (1c) and (2c) show the following processes, respectively.
(1c) Setting example of rotation camera shake correction vector for the corrected image on which the zoom magnification correction and the distortion aberration correction have been performed on the output image of the first imaging unit 111 (2c) on the output image of the second imaging unit 121 Example of setting a rotation image stabilization vector for a corrected image that has been subjected to zoom magnification correction and distortion correction

  The rotational camera shake correction executed by the vectors shown in FIGS. 18 (1c) and (2c) is, for example, the correction described above with reference to FIG. The examples shown in FIGS. 18 (1c) and (2c) are vector setting examples corresponding to the example in which the rotation center shown in FIG. 8 (e) is set as the image center. As described above with reference to FIG. 8, the setting of the correction vector changes depending on where the center of rotation correction is provided in the image, but the center of rotation correction does not have to be the center of the actual rotation blur. This is virtually provided when the correction vector is set, and is set at a predetermined position.

  For the images shown in FIGS. 18 (1c) and (2c), a rotational camera shake correction process is performed by applying the vectors shown in the images. The correction results are (1d) and (2d), respectively. (1d) and (2d) are images from which the rotational component has been removed. The following translational camera shake correction processing is performed on the images shown in (1d) and (2d).

18 (1d) and (2d) show the following processes, respectively.
(1d) Setting example of translation camera shake correction vector for a corrected image that has been subjected to zoom magnification correction, distortion aberration correction, and rotational camera shake correction processing for the output image of first image pickup unit 111 (2c) second image pickup unit 121 Example of setting of translational shake correction vector for a corrected image that has been subjected to zoom magnification correction, distortion aberration correction, and rotation shake correction processing for the output image of

  The translational shake correction performed by the vectors shown in FIGS. 18 (1d) and (2d) is, for example, the correction described above with reference to FIG. 7 (2). The example shown in FIG. 7 (2) shows an example of setting correction vectors for an image in which a captured image is taken so as to flow from the upper right to the lower left to cause translational camera shake. Unlike the example shown in FIG. 7 (2), the correction vectors shown in FIGS. 18 (1c) and (2c) are images that have been taken so that the taken image flows from the lower right to the upper left, and translational hand movement has occurred. This is an example of setting a correction vector for.

  For the images shown in FIGS. 18 (1d) and (2d), a parallel movement camera shake correction process is performed by applying the vectors shown in the images. The correction results are (1e) and (2e), respectively, and the translational camera shake component is removed, and the image becomes stable in the time axis direction. The following optical axis center correction processing is performed on the images shown in (1e) and (2e).

FIGS. 18 (1e) and (2e) show the following processes, respectively.
(1e) Setting example of optical axis center correction vector for a corrected image on which an output image of first imaging unit 111 has been subjected to zoom magnification correction, distortion aberration correction, rotation camera shake correction, and parallel movement camera shake correction processing (2e) Setting example of optical axis center correction vector for corrected image in which zoom magnification correction, distortion aberration correction, rotation camera shake correction, and parallel movement camera shake correction processing are performed on the output image of second imaging unit 121

  However, the optical axis center correction can be performed by performing correction on only one of the images. In this example, correction is performed by setting a vector only for the first image shown in FIG. Is going.

  The optical axis center correction processing executed by the vectors shown in FIGS. 18 (1e) and (2e) is, for example, the correction described above with reference to FIG. As described with reference to FIG. 17, the optical axis center correction vector adjusts (matches) the optical axis center coordinates slightly different between the plurality of images obtained from the plurality of imaging units to the other image. It is set as a vector that is translated and cut out as a whole.

  Optical axis center correction processing is performed on the images shown in FIGS. 18 (1e) and (2e) by applying the vectors shown in the images. The correction results are (1f) and (2f), respectively, and the next parallax correction process is performed.

FIGS. 18 (1f) and (2f) show the following processes, respectively.
(1f) Setting example of parallax correction vector for a corrected image on which an output image of the first imaging unit 111 is subjected to zoom magnification correction, distortion aberration correction, rotational camera shake correction, parallel movement camera shake correction, and optical axis center correction processing. 2f) An example of setting a parallax correction vector for a corrected image on which an output image of the second imaging unit 121 has been subjected to zoom magnification correction, distortion aberration correction, rotational camera shake correction, parallel movement camera shake correction, and optical axis center correction processing.

  However, the parallax correction can be performed by performing correction on only one of the images. In this example, correction is performed by setting a vector only for the first image shown in FIG. Yes.

The parallax correction processing executed by the vectors shown in FIGS. 18 (1f) and (2f) is, for example, the correction described above with reference to FIG. As described with reference to FIG. 9, the parallax correction is a correction process in which parallax adjustment is performed so that a stereoscopic image composed of both has a preferable parallax. The parallax correction based on the vector is performed on at least one of the captured image of the first imaging unit 111 and the captured image of the second imaging unit 121. In general, parallax correction can be realized by parallel movement of an image. Further, it may be executed as a process accompanied by an enlargement / reduction process as necessary.
In the example shown in FIG. 18, parallax correction is performed by setting a translation vector only for the first image shown in FIG.

  A parallax correction process is performed on the images shown in FIGS. 18 (1f) and (2f) by applying the vectors shown in the images. The correction results are (1out) and (2out), respectively. These images are output to a display device that performs 3D image display, for example, and 3D image display is executed.

When viewing stereoscopic images, it may be difficult to see or eye fatigue may be a problem. In general, these causes are considered to be a shift of two images, a difference in size, a double image, and the like. Of the correction processing in FIG.
(A) Optical zoom magnification correction,
(E) Optical axis center coordinate correction,
(F) Parallax correction These are these corrections. By performing a correction process for matching one with the other in two image processes, or a correction process for bringing both together as close as possible, the resulting stereoscopic image becomes a comfortable image that is much easier to watch.

further,
(B) distortion correction,
(C) rotational image stabilization,
(D) Parallel camera shake correction Each of these correction processes is a correction process necessary for creating a clearer, more faithful, and stable image even in a monocular camera. This is an indispensable correction process for any display of images.

[5. Specific examples of correction vector integration processing]
As described with reference to FIG. 18, all the correction processing in each correction unit can be executed as coordinate conversion processing using a vector.
That is, in each correction process, the following vectors are used.
(A) Correction vector as a zoom magnification correction parameter applied to zoom magnification correction processing (b) Correction vector as a distortion aberration correction parameter applied to distortion aberration correction processing (c) Rotation camera shake correction parameter applied to rotation camera shake correction processing (D) Correction vector as parallel camera shake correction parameter applied to parallel camera shake correction processing (e) Correction vector as optical axis center correction parameter applied to optical axis center correction processing (f) Application to parallax correction processing Correction vector as parallax correction parameter

  In the apparatus configuration of FIG. 1 or FIG. 2, the following individual vectors are calculated as correction parameters in each correction unit, and coordinate conversion using the correction vector is executed in each correction unit.

On the other hand, in the apparatus configuration of FIG. 3, the control unit 172 calculates one integrated correction vector by integrating the correction vectors applied to the correction processes (a) to (f) described above, and performs image conversion. The correction unit 251 executes processing to which the integrated correction vector is applied.
By performing such processing, the series of correction processing described with reference to FIG. 18 can be executed as one correction processing.

The process of calculating one integrated correction vector by integrating the correction vectors applied to the correction processes (a) to (f) described above can be executed as a vector addition process.
An example of vector addition will be described with reference to FIG.

FIG. 19 shows an example of vector addition.
(1) Distortion correction,
(2) Rotary camera shake correction,
It is a figure explaining the vector addition process applied to these two different corrections.

When integrating correction vectors, the set value is obtained by reversing the virtual correction processing order. For example, it is assumed that the point C shown in FIG. 19 is located at the intersection of the grid that should hold the correction parameters.
As the integrated correction vector calculation process, for example, when the point C is a corrected point obtained by performing distortion correction and rotational camera shake correction, it can be executed as a process for obtaining a corresponding point of the input image before correction. Here, it is the point B that is coordinate-converted to the point C by (2) rotational camera shake correction. Further, (1) the point A is coordinate-transformed to the point B by the distortion correction.

  That is, when (1) distortion correction, (2) rotational camera shake correction, and vector correction vectors applied to these two different corrections are integrated, the coordinate in the input image corresponding to the point C becomes the point A. At this time, since the rotational camera shake correction vector for the point B is different from the rotational camera shake correction vector for the point A, it is important to always obtain the vector at each stage by reversing the processing order.

  A vector as a correction parameter to be set at the point C is a vector from the point A to the point C. In this way, the integration of correction vectors is not a simple vector sum but can be obtained by obtaining the coordinates of the conversion source in the input image by following the order of each correction process.

  In FIG. 19, an example of integration of two correction vectors of (1) distortion aberration correction and (2) rotational camera shake correction has been described. Similarly, when the respective correction vectors are integrated, they can be calculated by tracing back to the original coordinates before correction in the reverse order of the respective correction processes. This integrated vector may be calculated for all the intersection points of the grid.

  In the case of having the “image conversion correction unit 251” in which a plurality of types of correction processes are integrated as shown in FIG. 3, the control unit considers the order of coordinate conversion in each correction process, and FIG. , (2f) to (1a), (2a) in reverse order, the integrated correction vector to be set for each representative point is obtained by tracing the original coordinates.

  Note that, in the process of calculating an integrated correction vector based on a plurality of correction vectors in the control unit, the correction vector to be integrated is decomposed into a horizontal vector and a vertical vector, and then integrated to increase calculation efficiency. It is possible.

However, when executing a vector integration process based on such a decomposition vector, for example, there are the following points of caution. FIG. 20 is a configuration example when the correction processing is divided in the order of horizontal and vertical.
Consider a case where a grid is set as shown in FIG. 20A and correction vectors are set for points A and C. Consider a 2Z vector for coordinate transformation of point B to point A and point D to point C. Considering normal vector decomposition, as shown in FIG.
For point A, horizontal vector VHA ′ and vertical vector VVA,
For point C, horizontal vector VHC ′ and vertical vector VVC,
It can be divided in this way.

  However, as described with reference to FIG. 19, the vector to be set at the lattice point A is the original coordinate to be moved to the point A, that is, the relative position from the point B. When interpolation is performed using VHA ′ and VHC ′ in FIG. 20B, the original coordinate of the point A becomes the point B ′, and correct coordinate conversion is not achieved. Therefore, FIG. 20B is not used.

In this case, as shown in FIG. 20C, the horizontal correction vector needs to be obtained in consideration of the vertical correction processing. If the horizontal vectors VHA and VHC are set so as to pass through the point B when interpolating in the vertical direction, the point B is correctly transformed into the point A.
As described above, when the control unit performs processing by dividing the correction vector into a horizontal vector and a vertical vector, the integrated correction vector is calculated by sequentially executing processing for obtaining original coordinates before conversion at each stage.

  The present invention has been described in detail above with reference to specific embodiments. However, it is obvious that those skilled in the art can make modifications and substitutions of the embodiments without departing from the gist of the present invention. In other words, the present invention has been disclosed in the form of exemplification, and should not be interpreted in a limited manner. In order to determine the gist of the present invention, the claims should be taken into consideration.

  The series of processing described in the specification can be executed by hardware, software, or a combined configuration of both. When executing processing by software, the program recording the processing sequence is installed in a memory in a computer incorporated in dedicated hardware and executed, or the program is executed on a general-purpose computer capable of executing various processing. It can be installed and run. For example, the program can be recorded in advance on a recording medium. In addition to being installed on a computer from a recording medium, the program can be received via a network such as a LAN (Local Area Network) or the Internet and can be installed on a recording medium such as a built-in hard disk.

  Note that the various processes described in the specification are not only executed in time series according to the description, but may be executed in parallel or individually according to the processing capability of the apparatus that executes the processes or as necessary. Further, in this specification, the system is a logical set configuration of a plurality of devices, and the devices of each configuration are not limited to being in the same casing.

  As described above, according to one embodiment of the present invention, it is possible to provide a configuration that reliably and efficiently implements a plurality of different image corrections on a captured image. Specifically, a zoom magnification correction, a distortion aberration correction, a rotation camera shake correction, a parallel camera shake correction, an optical axis center correction, a parallax correction, and a configuration that reliably executes all of these corrections are provided. In addition, a configuration is provided in which these correction processes are collectively executed to realize an efficient process. Especially for zoom magnification correction, optical axis center correction, and parallax correction required for a stereo camera that captures two images from two different viewpoints, the quality can be improved by correcting at least one of the images captured by the two imaging units. High 3D images can be provided. Furthermore, according to the configuration in which an integrated correction vector obtained by integrating correction vectors applied to each correction is calculated and corrected, efficient and reliable correction is realized.

111 First imaging unit 112 Lens group 113 Image sensor 114 Lens drive unit 115 Distortion aberration data storage unit 116 Camera shake sensor 121 Second imaging unit 122 Lens group 123 Image sensor 124 Lens drive unit 125 Distortion aberration data storage unit 126 Camera shake sensor 131 Optical Total parallax data storage unit 132 Zoom magnification correction value storage unit 133 Optical axis center correction value storage unit 141 Switching multiplexing unit 151 Camera signal processing unit 152 Zoom magnification correction unit 153 Distortion aberration correction unit 154 Rotation camera shake correction unit 155 Parallel movement camera shake correction Unit 156 optical axis center correction unit 157 parallax correction unit 158 data compression unit 159 image display unit 160 image storage medium or input / output terminal 171 image data storage unit 172 control unit 201 camera signal processing unit 202 zoom magnification correction unit 203 distortion Curvature aberration correction unit 204 Rotating camera shake correction unit 205 Parallel movement camera shake correction unit 206 Optical axis center correction unit 207 Parallax correction unit 208 Data compression unit

Claims (15)

A plurality of imaging units;
A correction unit that performs correction processing on the captured images of the plurality of imaging units;
A control unit that calculates a correction parameter to be applied to the correction process in the correction unit;
The correction unit is
While performing distortion correction and camera shake correction for each of the captured images,
An image characteristic adjustment correction process for matching characteristics of a plurality of images photographed by the plurality of imaging units;
The controller is
An integrated correction vector obtained by integrating correction vectors applied to a plurality of different correction processes executed in the correction unit is calculated for each of a plurality of representative points discretely set in an image, and provided to the correction unit.
Wherein the correction unit, Ri configuration der to perform collectively a plurality of different correction processing by the image conversion processing applying the integrated correction vector,
The control unit reverses the virtual execution order of the plurality of different correction processes, calculates the corresponding point of the input image whose representative point is the corrected point, and represents the vector from the corresponding point to the representative point. An imaging apparatus that calculates a point-corresponding integrated correction vector . The correction unit is
The imaging apparatus according to claim 1, wherein as the image characteristic adjustment correction process, a zoom magnification correction process for matching zoom magnifications of a plurality of images captured by the plurality of imaging units is performed. The correction unit is
The imaging apparatus according to claim 1, wherein as the image characteristic adjustment correction process, an optical axis center correction process for matching optical axis centers of a plurality of images captured by the plurality of imaging units is performed. The correction unit is
Furthermore, the imaging device of Claim 1 which performs the parallax correction process which adjusts the parallax of the several image image | photographed in these imaging parts. The controller is
The imaging apparatus according to claim 1, wherein calculation of a correction parameter applied to correction processing in the correction unit is executed and provided to the correction unit. The correction parameter is
The imaging apparatus according to claim 5, wherein the imaging device is a correction vector applied to image coordinate conversion in a correction unit. The correction unit is
A distortion correction unit that performs distortion correction on each of the captured images;
A rotational image stabilization unit that performs rotational image stabilization for each of the captured images;
A translational shake correction unit that performs translational shake correction for each of the captured images;
A zoom magnification correction unit that matches zoom magnifications that are characteristics of a plurality of images captured by the plurality of imaging units;
An optical axis center correcting unit that matches optical axis centers that are characteristics of a plurality of images captured by the plurality of imaging units;
The imaging apparatus according to claim 1, wherein the imaging apparatus includes a parallax correction unit that performs a parallax correction process for adjusting parallax of a plurality of images captured by the plurality of imaging units. The controller is
Distortion correction processing, rotational camera shake correction processing, translational camera shake correction processing, zoom magnification correction processing, optical axis center correction processing, parallax correction processing, and integrated correction that integrates correction vectors applied to each correction processing A vector is calculated and provided to the correction unit;
The correction unit includes a distortion aberration correction process, a rotation camera shake correction process, a parallel movement camera shake correction process, a zoom magnification correction process, an optical axis center correction process, a parallax, and an image conversion process using the integrated correction vector. The imaging apparatus according to claim 1, wherein the imaging apparatus is configured to execute correction processing all at once. The imaging device
A distortion aberration data storage unit that stores distortion aberration data corresponding to the imaging unit to be applied to distortion correction;
The imaging apparatus according to claim 1, wherein the control unit generates a distortion aberration correction parameter based on data acquired from the distortion aberration data storage unit. The imaging device
An optical axis center correction value storage unit that stores optical axis center correction data to be applied to optical axis center correction for matching the optical axis centers that are the characteristics of a plurality of images captured by a plurality of imaging units;
The imaging apparatus according to claim 1, wherein the control unit generates an optical axis center correction parameter based on data acquired from the optical axis center correction value storage unit. The imaging device
A zoom magnification correction value storage unit that stores zoom magnification correction data to be applied to zoom magnification correction for matching zoom magnifications that are characteristics of a plurality of images captured by a plurality of imaging units;
The imaging apparatus according to claim 1, wherein the control unit generates a zoom magnification correction parameter based on data acquired from the zoom magnification correction value storage unit. The imaging device
A parallax data storage unit that stores parallax data to be applied to parallax correction for adjusting parallax of a plurality of images captured by a plurality of imaging units;
The imaging apparatus according to claim 1, wherein the control unit generates a parallax correction parameter based on data acquired from the parallax data storage unit. A correction unit that executes correction processing on captured images of a plurality of imaging units;
A control unit that calculates a correction parameter to be applied to the correction process in the correction unit;
The correction unit is
While performing distortion correction and camera shake correction for each of the captured images,
An image characteristic adjustment correction process for matching characteristics of a plurality of images photographed by the plurality of imaging units;
The controller is
An integrated correction vector obtained by integrating correction vectors applied to a plurality of different correction processes executed in the correction unit is calculated for each of a plurality of representative points discretely set in an image, and provided to the correction unit.
Wherein the correction unit, Ri configuration der to perform collectively a plurality of different correction processing by the image conversion processing applying the integrated correction vector,
The control unit reverses the virtual execution order of the plurality of different correction processes, calculates the corresponding point of the input image whose representative point is the corrected point, and represents the vector from the corresponding point to the representative point. An image processing device that calculates a point-corresponding integrated correction vector . An image processing method executed in an image processing apparatus,
A parameter calculating step in which the control unit calculates a correction parameter to be applied to the correction processing in the correction unit;
The correction unit executes a correction step of executing correction processing to which the correction parameter is applied to the captured images of the plurality of imaging units,
The correction step includes
While performing distortion correction and camera shake correction for each of the captured images,
A step of executing image characteristic adjustment correction processing for matching the characteristics of a plurality of images photographed by the plurality of imaging units;
In the parameter calculation step, the control unit includes:
An integrated correction vector obtained by integrating correction vectors applied to a plurality of different correction processes executed in the correction unit is calculated for each of a plurality of representative points discretely set in an image, and provided to the correction unit.
In the correction step, the correction unit includes:
A plurality of different correction processes are collectively executed by the image conversion process using the integrated correction vector ,
In the parameter calculation step, the control unit calculates a corresponding point of the input image in which the representative point is a corrected point by tracing back the virtual execution order of the plurality of different correction processes, and represents the representative point from the corresponding point. An image processing method for calculating a vector directed to a point as an integrated correction vector corresponding to a representative point . A program for executing image processing in an image processing apparatus;
A parameter calculation step for causing the control unit to calculate a correction parameter to be applied to the correction process in the correction unit;
Causing the correction unit to execute a correction step of executing correction processing to which the correction parameter is applied to the captured images of the plurality of imaging units;
The correction step includes
While executing distortion correction and camera shake correction for each of the captured images,
A step of executing an image characteristic adjustment correction process for matching the characteristics of a plurality of images photographed by the plurality of imaging units;
The parameter calculating step includes:
A step of calculating an integrated correction vector obtained by integrating correction vectors applied to a plurality of different correction processes executed in the correction unit for each of a plurality of representative points set discretely in an image, and providing the calculated correction point to the correction unit; Yes,
The correction step includes
Ri step der to perform collectively a plurality of different correction processing by the image conversion processing applying the integrated correction vector,
The parameter calculating step reverses the virtual execution order of the plurality of different correction processes, calculates a corresponding point of the input image whose representative point is a corrected point, and calculates a vector from the corresponding point to the representative point. A program that is a step of calculating as an integrated correction vector corresponding to a representative point .