JP2013005392A - Image processing method having wide-angle distortion correction processing, image processing apparatus and imaging apparatus - Google Patents

Abstract

PROBLEM TO BE SOLVED: To accurately recognize a subject on distortion correction processing, using a relatively small scale circuit.SOLUTION: A distortion correction coefficient at least includes a first distortion correction coefficient calculated based on a physical characteristic of a lens in an optical system 110 and an incident angle of incident light from a virtual projection plane VP set to the optical system, and a second distortion correction coefficient which is an image height from an optical center at which an imaging device intersects with the optical axis of the optical system, calculated from a tangent function of the incident angle to the optical system as a variable. The tangent function is calculated by multiplying the incident angle by a predetermined coefficient smaller than 1 so as to avoid an incident angle range that saturates due to the value of the distortion correction coefficient exceeding a predetermined upper limit value.

Description

  The present invention relates to an image processing method, an image processing apparatus, and an imaging apparatus that perform distortion correction processing on an image captured by an imaging element via a wide-angle optical system including a condenser lens.

  In general, an image captured by an optical system having a short focal length lens or a large angle of view lens such as a wide-angle lens or a fish-eye lens is distorted, and image processing for correcting the distortion is performed.

  For such image processing, techniques related to the following patent documents have been proposed.

  Patent Document 1 discloses a correction method of the prior art that corrects distortion generated in a captured image captured using a lens with a short focal length using a lens correction parameter.

  In Patent Document 2, it is necessary to use an external information processing device to calculate the optical distortion correction parameter for each lens position from the wide end to the tele end of the optical zoom mechanism by using an interpolation operation. With the optical distortion correction parameter for the discrete lens position within the range to perform the optical zoom, the lens position at the time of zooming is limited to the lens position having the optical distortion correction parameter. The optical zoom between the positions is connected by electronic zoom.

  In Patent Document 3, distortion correction is performed depending on the difference in the angle of view switched by the angle-of-view switching means, such as performing distortion correction in the case of the angle of view on the wide angle side, and not performing distortion correction in the case of the angle of view other than the wide angle side. A video processing apparatus for changing the above is disclosed.

JP 2009-140066 A JP 2009-105546 A JP 2009-61969 A

  As disclosed in Patent Document 1, there is a problem that when a captured image obtained by a lens is hardwareized as an image processing device, the processing time becomes long, the circuit scale increases, and the cost increases. .

  In Patent Document 2, the lens position at the time of zooming is limited to a position corresponding to the discrete distortion correction parameter, and the zoom operation is omitted by interpolating the distortion correction parameter by connecting between them with an electronic zoom. In this way, the zoom operation of the image pickup device alone is realized. However, it can be applied only to a one-dimensional lens movement such as a zoom operation, and is difficult to apply to various movements such as panning and tilting. That is, there is a problem that it is not possible to cope with viewpoint conversion that changes the appearance by image processing.

  In addition, since the image after the distortion correction processing has a narrow viewing angle, it is difficult to recognize a wide area at a time. In an image that is not subjected to distortion correction processing, the viewing angle is widened, but on the other hand, it is difficult to recognize the sense of distance and size due to distortion of the subject.

  In Patent Document 3, the distortion correction amount is varied according to the angle of view for such a problem, but the distortion correction amount is not sequentially changed. It does not solve both of the two problems of accurate recognition.

  In addition, when generating and displaying an image after distortion correction from an entire image taken with a wide-angle lens and without distortion, it is clear which part of the entire image the final image after distortion correction displays. Therefore, it is desirable to display the distortion correction state while changing it little by little, but it has been found that each of the above patent documents is not suitable.

  In addition, when the inventor of the present application performs distortion correction of a subject by a method described later, which is different from the above-mentioned patent document, and performs processing so as to continuously change the distortion correction state little by little, the relationship between the incident angle and the parameter Thus, it became clear that the image after distortion correction was disturbed. That is, the present inventors have found that it is difficult to accurately recognize a subject because there is a disorder (image defect) in an image displaying a distortion correction state.

  In view of such a problem, the present invention provides an image processing method, an image processing apparatus, and an imaging device capable of accurately recognizing a subject and shortening a processing time with a relatively small circuit. The object is to provide a device.

  The above object is achieved by the invention described below.

  (1) In an image processing method for obtaining image data processed using a plurality of pixel data obtained by receiving an image sensor having a plurality of pixels via an optical system, the position of a virtual projection plane in the world coordinate system and A first step of setting a size, and a coordinate in the world coordinate system of each pixel of the virtual projection plane set in the first step is converted into a camera coordinate system using a distortion correction coefficient, and in the converted camera coordinate system A second step of calculating image data of the virtual projection plane set in the first step based on the coordinates and the plurality of pixel data, and a second step of outputting a display image based on the image data calculated in the second step. The distortion correction coefficient used in the second step includes at least the physical characteristics of the lens of the optical system and the setting to the optical system. The first distortion correction coefficient for correcting the distortion caused by the optical system calculated based on the incident angle of the incident light from the virtual projection plane and the tangent function of the incident angle to the optical system are calculated as variables. A second distortion correction coefficient that does not correct the distortion and includes an image height from an optical center at which the imaging element and the optical axis of the optical system intersect with each other. The second distortion is included in the second step. When there is a minimum incident angle at which the correction coefficient is equal to or greater than a predetermined value, the incident angle is multiplied by a predetermined coefficient less than 1 so as to avoid pixels in a region corresponding to an incident angle larger than the minimum incident angle, and the tangent function is calculated. It is characterized by calculating.

  An image processing apparatus or an imaging apparatus that obtains image data processed using a plurality of pixel data obtained by receiving an image sensor having a plurality of pixels via an optical system, and stores a distortion correction coefficient The coordinates in the world coordinate system of each pixel of the virtual projection plane whose position and size are set are converted into the camera coordinate system using the distortion correction coefficient stored in the storage unit, and converted into the camera coordinate system An image processing unit that calculates image data of the virtual projection plane based on the coordinates and the plurality of pixel data; an image signal output unit that outputs an image signal for display of the image data calculated by the image processing unit; And the storage unit stores at least the physical characteristics of the lens of the optical system and the incident light from the virtual projection plane set to the optical system as the stored distortion correction coefficient. A first distortion correction coefficient for correcting distortion caused by the optical system calculated based on an angle and a tangent function of an incident angle to the optical system and the optical axis of the optical system calculated as variables When the image processing unit has a second distortion correction coefficient that does not correct the distortion at the image height from the optical center where the second distortion correction coefficient is equal to or greater than a predetermined value, The tangent function is calculated by multiplying the incident angle by a predetermined coefficient less than 1 so as to avoid pixels in a region corresponding to an incident angle larger than the minimum incident angle.

  (2) In the above (1), the third step outputs a moving image for display of image data in which distortion is corrected stepwise, wherein the first distortion correction coefficient in the second step, The image data of the virtual projection plane is calculated by switching the second distortion correction coefficient and the third distortion correction coefficient obtained by interpolation from the first and second distortion correction coefficients in a stepwise manner. To do.

  In the above (1), the image signal output unit outputs a moving image for display of image data in which distortion is corrected stepwise, and the image processing unit includes the first distortion correction coefficient. The image data of the virtual projection plane is calculated using the second distortion correction coefficient and the third distortion correction coefficient obtained by interpolation from the first and second distortion correction coefficients in a stepwise manner. And

  (3) In the above (1) to (2), the third step outputs a moving image for display of image data in which distortion is corrected stepwise, and the second step includes the first step. Calculating the image data of the virtual projection plane using the distortion correction coefficient, the second distortion correction coefficient, and the third distortion correction coefficient obtained by interpolation from the first and second distortion correction coefficients in a stepwise manner; In the second step, when there is a minimum incident angle at which the second distortion correction coefficient is equal to or greater than a predetermined value, the incident angle is less than 1 so as to avoid pixels in a region corresponding to an incident angle larger than the minimum incident angle. When calculating the second distortion correction coefficient by multiplying by a predetermined coefficient, a coefficient less than 1 is determined according to an interpolation ratio of the second distortion correction coefficient included in the third distortion correction coefficient. And

  In the above (1) to (2), the image signal output unit outputs a moving image for display of image data in which distortion is corrected stepwise, and the image processing unit One distortion correction coefficient, the second distortion correction coefficient, and the third distortion correction coefficient obtained by interpolation from the first and second distortion correction coefficients are used in a stepwise manner to calculate image data of the virtual projection plane. In the second step, when there is a minimum incident angle at which the second distortion correction coefficient is greater than or equal to a predetermined value, the incident angle is less than 1 so as to avoid pixels in a region corresponding to an incident angle larger than the minimum incident angle. When the second distortion correction coefficient is calculated by multiplying by the predetermined coefficient, the coefficient less than 1 is determined according to the interpolation ratio of the second distortion correction coefficient included in the third distortion correction coefficient. Features.

  (4) In the above (1) to (3), in the first step, the first position and the second position of the virtual projection plane of the world coordinate system are set, and in the second step, the initial second A position and the second distortion correction coefficient, a final first position and the first distortion correction coefficient, a third position obtained by interpolation between the second position and the first position in the middle, and the third distortion correction coefficient The image data of the virtual projection plane is calculated by switching in a stepwise manner.

  Further, in the above (1) to (3), the virtual projection plane of the world coordinate system has a first position and a second position set, and the image processing unit performs the initial second position and the second position. 2 stepwise correction coefficients, the final first position and the first distortion correction coefficient, the third position obtained by interpolation of the second position and the first position in the middle, and the third distortion correction coefficient The image data of the virtual projection plane is calculated by switching to the above.

(5) In the above (1) to (4), the second distortion correction coefficient is expressed by the following conditional expression (1), and in consideration of saturation of the second distortion correction coefficient, It is expressed by conditional expression (1 ′).
L2 = ((ExportImageSize / 2) / (tan (focal / 2))) × tanθ Equation (1)
L2 = ((ExportImageSize / 2) / (tan (focal / 2))) × tan (αθ) Expression (1 ′)
here,
ExportImageSize: the length of the long side of the output image to be displayed on the display,
focal: Angle of view in the long side direction of the set virtual projection plane,
α: a predetermined coefficient less than 1.

  In the present invention, at least one of the first distortion correction coefficient calculated based on the physical characteristics of the lens of the optical system and the second distortion correction coefficient calculated by the relational expression using the incident angle to the optical system as a variable. Is used to calculate image data using a simple method of erroneously recognizing a subject due to image defects caused by image processing limitations caused by incidence on a wide-angle lens at a very large angle. Can be reduced.

It is a schematic diagram explaining the distortion correction which concerns on this embodiment. It is explanatory drawing which shows the example which moved the position of the virtual projection surface VP. It is a block diagram which shows schematic structure of an imaging device. It is a flowchart which shows the control flow of this embodiment. FIG. 6 is a characteristic diagram showing a relationship between an incident angle θ and an image height h on the image sensor surface IA. It is explanatory drawing which shows the state which set the virtual projection surface VP in the optical range. It is a schematic diagram which shows the relationship between the image height h in the 1st distortion correction coefficient L1 and the 2nd distortion correction coefficient L2. It is explanatory drawing which shows the example of an input-output image. It is explanatory drawing which shows the example which changes a distortion correction rate or a distortion correction rate and a viewpoint conversion rate with a predetermined period. This is an example in which the position of the virtual projection plane VP is changed with the image center o of the camera coordinate system as the rotation center. This is an example in which the position of the virtual projection plane VP is changed with the center ov of the virtual projection plane VP0 as the rotation center. It is a characteristic view which shows the saturation and discontinuous state of a distortion correction coefficient. It is explanatory drawing which shows the image defect by the influence of the saturation and discontinuity of a distortion correction coefficient with a photograph image. It is a characteristic view which shows the blackout point of this embodiment. It is a characteristic view which shows the blackout point of this embodiment. It is a figure which shows the correspondence of camera coordinate system xy and image pick-up element surface IA. It is a figure which shows the correspondence of camera coordinate system xy and image pick-up element surface IA. It is a schematic diagram explaining a distortion correction process. It is explanatory drawing which shows the example of a display of the process image to which this embodiment is applied with a photograph image.

  Hereinafter, the present invention will be described based on embodiments, but the present invention is not limited to the specific examples shown in the embodiments.

[A] Coordinate system:
FIG. 1 is a schematic diagram showing a coordinate system for explaining distortion correction according to the present embodiment.

  In FIG. 1, X, Y, and Z are world coordinate systems as coordinate systems for indicating the position of an object in space, and the origin O is the lens center of the optical system. Here, Z includes the optical axis, and the XY plane includes a lens center plane LC passing through the lens center O. Point P is an object point of the subject in the world coordinate system XYZ. θ is an incident angle with respect to the optical axis (coincident with the Z axis).

  On the other hand, x and y are camera coordinate systems based on the camera, and the xy plane corresponds to the image sensor surface IA. o is the optical center, which is the intersection of the optical axis Z and the image sensor surface. The point p is a point on the image sensor surface in the camera coordinate system, and the object point P is a distortion correction coefficient (a first distortion correction coefficient described later) based on a parameter based on the physical characteristics of the lens (hereinafter referred to as “lens parameter”). (Corresponding to L1) and converted to the camera coordinate system.

  In FIG. 1, VP is a virtual projection plane. The virtual projection plane VP is set on the opposite side of the imaging element (and imaging element surface IA) with respect to the lens position (lens center plane LC) of the optical system. The virtual projection plane VP can be moved and changed in size based on an instruction from the user to the operation unit 130 (see the block diagram of FIG. 3). In the specification of the present application, “position change” is a concept including not only the case where the virtual projection plane VP is translated on the XY plane, but also an angle change (also referred to as an attitude change) with respect to the XY plane.

  In an initial state (initial position setting, the same applies hereinafter), the virtual projection plane VP is arranged at a predetermined position (Z direction) parallel to the lens center plane LC (XY direction) with a predetermined size, and the center of the virtual projection plane VP. ov is located on the Z axis. Gv is a point where the object point P is projected onto the virtual projection plane VP, and is an intersection of the object point P and a straight line passing through the lens center O and the virtual projection plane VP.

  FIG. 2 shows a state in which the virtual projection plane VP0 is rotated on the XZ plane based on the input from the operation unit 130 to obtain the virtual projection plane VP1.

[B] Configuration:
FIG. 3 is a block diagram illustrating a schematic configuration of the imaging apparatus used in the present embodiment. The imaging device includes an imaging unit 110, a control device 100, a display unit 120, and an operation unit 130.

  The imaging unit 110 includes a short focus lens, an imaging element, and the like. In the present embodiment, examples of the short focus lens include a wide angle lens and a fisheye lens.

  The control device 100 includes an image processing unit 101, a setting unit 102, and a storage unit 103.

  The setting unit 102 sets the position and size of the virtual projection plane VP based on an input instruction to the operation unit 130.

  The image processing unit 101 creates a conversion table of each coordinate on the virtual projection plane into the camera coordinate system based on the set position and size of the virtual projection plane VP, and shoots with the imaging unit 110 using the conversion table. The processed pixel data is processed to create image data to be displayed on the display unit 120. That is, the image processing unit 101 performs coordinate transformation that applies to which coordinate position of the camera coordinates the world coordinate position corresponds to based on the lens data. At this time, distortion correction and rotation, translation, enlargement, and reduction associated with calculation of the position of the virtual projection plane VP are performed together. The image processing unit 101 also functions as an image signal output unit that outputs an image signal for display.

  The storage unit 103 stores coefficients calculated based on parameters relating to physical characteristics of the lens based on the physical characteristics of the lens (real lens) of the imaging unit 110 and the incident angle of incident light from the set virtual projection plane, that is, optical. A first distortion correction coefficient L1 that is a coefficient for correcting distortion generated in the system, and an incident angle of incident light that enters the lens from the set virtual projection plane, which corresponds to a coefficient that does not correct the distortion. Is stored as a second distortion correction coefficient L2, which is an image height from the optical center at which the image sensor and the optical axis of the optical system intersect, which is calculated using the tangent function of

  The storage unit 103 also stores the position and size of the virtual projection plane VP and the created conversion table. The distortion correction coefficient can be obtained by obtaining lens calibration data, calculating based on the fθ characteristic of the lens, or the like.

  Note that “do not perform distortion correction” in the present embodiment means that the distortion as it is captured is substantially generated, and is slightly inferior from that in which distortion correction is performed satisfactorily. It does not point.

  The display unit 120 includes a display screen such as a liquid crystal display, and sequentially displays a display image based on the image data created by the image processing unit 101 based on the pixel data captured by the imaging unit 110 on the display screen.

  The operation unit 130 includes a keyboard, a mouse, or a touch panel arranged so as to be superimposed on the liquid crystal display of the display unit, and receives a user's input operation.

[C] Control:
Hereinafter, the control of this embodiment will be described with reference to the flowchart of FIG. According to this control flowchart, a moving image is displayed by processing input images that are continuously input and continuously displaying output images on the display unit 120.

[C-1] Setting and input:
First, setting of a virtual projection plane, setting of viewpoint conversion, setting of distortion correction conditions, and the like are input (step S101 in FIG. 4). This is the first step in the claims. These various settings are performed by setting the position and size of the virtual projection plane VP in the world coordinate system by the setting unit 102 based on an input instruction to the operation unit 130 by the user. Also, parameters related to the physical characteristics of the lens used in the imaging unit 110 are read from the storage unit 103.

  Then, an image signal is input from the imaging unit 110, and the input image is taken into the control device 100 at a frame rate of 60 fps, for example (step S102 in FIG. 4).

[C-2] Virtual projection plane calculation:
Here, the image processing unit 101 calculates the position of the virtual projection plane VP by calculating the translation vector and the rotation vector based on the set condition and performing the Eugrid conversion (step in FIG. 4). S103). Note that this virtual projection plane calculation, distortion correction coefficient acquisition described later, and distortion correction processing are the second step in the claims.

[C-3] Acquisition of distortion correction coefficient:
Then, the image processing unit 101 performs first coordinate correction coefficient L1 and second distortion correction coefficient L2 for performing coordinate conversion that applies to which coordinate position of the camera coordinate the world coordinate position corresponds to based on the lens data. Is acquired (step S104 in FIG. 4).

  Here, the distortion correction coefficient acquired by the image processing unit 101 includes parameters relating to physical characteristics of the lens, which are coefficients for correcting distortion generated in the optical system, and the incident angle of incident light from the set virtual projection plane. The imaging is calculated using as a variable the first distortion correction coefficient L1 determined on the basis of the tangent function of the incident angle θ (see FIG. 1) of incident light incident from the virtual projection plane set to the optical system. A second distortion correction coefficient L2 that is an image height from the optical center where the element and the optical axis of the optical system intersect is included. The second distortion correction coefficient L2 is a distortion correction coefficient corresponding to not performing distortion correction, and will be described in detail later.

  Here, the first distortion correction coefficient L1 may be rewritten at any time, for example, every frame as required by the user. That is, in addition to acquiring the first distortion correction coefficient L1 by inputting the position and screen size of the virtual projection plane at the first step in the setting unit 102, the value of the first distortion correction coefficient L1 is directly changed. A configuration in which parameters can be input may be employed, whereby a relationship between pixels on camera coordinates and corresponding pixels on world coordinates may be arbitrarily changed.

  In the case of the present invention, it is also possible to change the image height position from the optical axis center where the incident angle and the optical axis intersect from the rewritten first distortion correction coefficient L1, the coordinates on the virtual projection plane, and the screen size. is there.

  This makes it possible to respond flexibly to various optical systems as compared to the case where the corresponding pixel position (region) on the camera coordinates is fixed in advance, and various distortions that do not depend on the pixel position (region) on the camera coordinates. There is an effect that the correction condition can be adopted.

  Further, the image processing unit 101 acquires a plurality of third distortion correction coefficients L3 so as to smoothly perform coordinate conversion by interpolating between the first distortion correction coefficient L1 and the second distortion correction coefficient L2. (Step S105 in FIG. 4).

[C-3-1] Description of distortion correction coefficient:
Hereinafter, the distortion correction coefficient will be described.

[C-3-1-1] Description of the first distortion correction coefficient L1 and the second distortion correction coefficient L2:
Here, the first distortion correction coefficient L1 and the second distortion correction coefficient L2 will be described with reference to FIG.

  FIG. 5 is a characteristic diagram showing a relationship in which the incident angle θ is the horizontal axis and the image height h (w2c value) from the optical center o intersecting the optical axis Z on the image sensor surface IA is the vertical axis.

  5, each of the first distortion correction coefficient L1 (solid line in FIG. 5), the second distortion correction coefficient L2 (one-dot chain line), and the third distortion correction coefficient L3 (two-dot chain line) by interpolation between L1 and L2. The relationship between the incident angle θ and the image height h is displayed.

  FIG. 6 shows a state in which the virtual projection plane VP is set in the optical range determined by the optical system. FIG. 7 is a characteristic diagram showing the relationship between the image heights h (indicated by h1 and h2) in the first distortion correction coefficient L1 and the second distortion correction coefficient L2. In FIGS. 6 and 7, the same reference numerals are given to the portions common to FIG.

  As shown in the characteristic diagram of FIG. 5, with the second distortion correction coefficient L2, distortion aberration corresponding to the incident angle θ occurs, and as the incident angle θ increases, the image height h (w2c value) increases and the amount of distortion aberration increases. Also grows. That is, the distortion correction rate = 0%, and by using the coefficient, image data in which distortion corresponding to the optical system is not corrected as it is can be obtained.

  On the other hand, in the characteristic diagram of FIG. 5, the first distortion correction coefficient L1 is a distortion correction coefficient set as a distortion correction rate = 100% so as to cancel the influence of the distortion aberration caused by the lens. Is a coefficient determined by the incident characteristics of the incident light entering the optical system from the set virtual projection plane.

  Further, the third distortion correction coefficient L3 obtained by interpolation between the first distortion correction coefficient L1 and the second distortion correction coefficient L2 has a distortion correction rate according to the interpolation ratio between the first distortion correction coefficient L1 and the second distortion correction coefficient L2. It is determined. The third distortion correction coefficient L3 and the interpolation ratio will be described later.

Here, the second distortion correction coefficient L2 is a distortion correction coefficient related to the image height from the optical center where the imaging element and the optical axis of the optical system intersect, which is calculated using the incident angle θ to the optical system as a variable. As the relational expression, for example, the following expression (1) is used. Note that there is an advantage that the output image can be easily scaled by multiplying the expression (1) several times.
L2 = ((ExportImageSize / 2) / (tan (focal / 2))) × tanθ ... Formula (1),
In this equation (1), “ExportImageSize” is the length of the long side (usually the horizontal direction, x) of the output image displayed on the display unit, and the unit is pixel. This coincides with the size of the long side of the virtual projection plane VP.

  In the formula (1), “focal” means the angle of view in the long side direction of the set virtual projection plane. Here, “focal” will be described with reference to FIG. In FIG. 6, ω is an optical angle of view corresponding to the optical range, and “focal” is an angle of view in the long side direction of the virtual projection plane VP. Note that a relational expression of orthographic projection and equidistant projection may be used instead of the expression (1). Further, “focal” corresponds to a display range within an optical range arbitrarily set by the user, and does not necessarily have to be in the long side direction of the angle of view.

  Note that the incident angle may exceed 90 ° in a super-wide-angle lens such as a fish-eye lens. In this case, in the above-described equation (1), when the incident angle exceeds 90 °, the sign of tanθ is inverted, so that it deviates from the original value of the second distortion correction coefficient L2. For this reason, in the present embodiment, the calculation is performed assuming that the image height (w2c value) is infinite in the region where the incident angle exceeds 90 ° regardless of the above-described equation (1).

  In the present embodiment, the conditional expression (1) has been described, but the conditional expression is not necessarily limited to this. In short, the image height with the incident angle θ as a variable, that is, on the camera coordinates from the optical center. Any conditional expression that calculates the distance may be used.

  FIG. 7 is a schematic diagram showing the relationship between the first distortion correction coefficient L1 and the image height h at the second distortion correction coefficient L2. The image height h is determined by the incident angle θ and the correction coefficient as shown in FIG. In the second distortion correction coefficient L2, the image height (distance from the optical axis Z) of the subject (object point P) is proportional to the image height h on the image sensor surface IA. Further, the image height h2 at the second distortion correction coefficient L2 is longer than the image height h1 of the first distortion correction coefficient L1, as shown in FIG.

  Here, in order to clarify the meaning of the first distortion correction coefficient L1 and the second distortion correction coefficient L2, the input image, the image processed with the first distortion correction coefficient L1, and the second distortion correction coefficient L2 are used. An example of the processed image is shown in FIG.

  FIG. 8 shows an example in which the virtual projection plane VP is set to the initial state. FIG. 8A shows an input image obtained by an imaging unit 110 having a wide-angle lens on a rectangular chart installed on a rectangular wall surface. 8B shows an output image when a distortion correction process described later is performed using the first distortion correction coefficient L1, and FIG. 8C shows a distortion correction process described later using the second distortion correction coefficient L2. It is an example of the output image at the time of performing.

  In the example shown in FIG. 8B, it can be seen that the distorted curve in the input image is corrected to be a straight line. On the other hand, when the distortion correction processing is performed with the second distortion correction coefficient L2 as shown in FIG. 8C, the output image is almost the same as the input image shown in FIG. That is, it can be seen that an output image equivalent to the case where distortion correction is not performed is obtained.

  When the distortion correction process is performed using the third distortion correction coefficient L3, an image in a distortion correction state intermediate between FIGS. 8B and 8C is obtained as will be described later.

[C-3-1-2] Description of distortion correction coefficient L3:
Here, the third distortion correction coefficient L3 will be described. In calculating the distortion correction coefficient, the third distortion correction coefficient L3 calculated by interpolation between the first distortion correction coefficient L1 and the second distortion correction coefficient L2 can be switched stepwise by changing the distortion correction rate (DistRatio). Is calculated.

  Further, in this embodiment, the position of the virtual projection plane VP and the third position V3 calculated by interpolation between the first position V1 and the second position V2 are also switched stepwise in accordance with the change of the viewpoint conversion rate (VCRatio). ing. A specific example of the viewpoint conversion will be described later.

  The third distortion correction coefficient L3 and the third position V3 are calculated by the following equations (2) and (3).

L3 = DistRatio / 100 × L1 + (1−DistRatio / 100) × L2 Equation (2),
However, 0 ≦ DistRatio / 100 ≦ 1.

V3 = VCRatio / 100 × V1 + (1−VCRatio / 100) × V2 Equation (3),
However, 0 ≦ VCRatio / 100 ≦ 1.

  Here, the distortion correction rate and the viewpoint conversion rate are calculated in accordance with the processing time set in the condition setting (step S101 in FIG. 1). For example, when 10 sec is set to 1 cycle, if the time is 0.0 sec, the distortion correction rate and the viewpoint conversion rate are both 0%, and if the time is 5.0 sec, the distortion correction rate and the viewpoint conversion are set. If the rate is 50% and the time is 10.0 sec, the distortion correction rate and the viewpoint conversion rate are both 100%.

  The distortion correction coefficient L3 is calculated from the distortion correction factor calculated in this way based on the above-described equation (2). Note that when the distortion correction rate is 0%, the second distortion correction coefficient L2 itself, and when 100%, the first distortion correction coefficient L1 itself.

  Further, the third position of the virtual projection plane VP is calculated from the viewpoint conversion rate. The second position, which is the initial position of the virtual projection plane VP corresponding to the second distortion correction coefficient L2, and the first position, which is the final position of the virtual projection plane VP corresponding to the first distortion correction coefficient L1, are: This is set in S101. In other words, the calculation of the third position is performed by using the viewpoint conversion rate for the coordinates of the first position and the second position of the virtual projection plane VP, using Equation (3).

  Here, FIG. 9 shows an example in which the distortion correction rate or the distortion correction rate and the viewpoint conversion rate are changed at a predetermined cycle.

  FIG. 9A shows an example in which the third distortion correction coefficient L3 is changed stepwise by 10 points every second by changing the distortion correction rate with 10 seconds as one cycle. Note that viewpoint conversion is not performed in the example of FIG.

  FIG. 9B shows an example in which the third distortion correction coefficient L3 is changed stepwise by 10 points every second by changing the distortion correction rate with 10 seconds as one cycle. Further, the third position V3 is also changed stepwise by changing the viewpoint conversion rate. In the example of FIG. 9B, the distortion correction rate is also changed as the viewpoint conversion rate is changed. 9 (a) and 9 (b) show an example in which the period is changed in 10 seconds and 11 steps in all, but it is only an example, and the cycle can be changed arbitrarily, or in more detailed steps than this. It may be changed.

[C-3-1-3] Change of virtual projection plane VP and viewpoint conversion:
Here, viewpoint conversion will be described. FIG. 10 is an example in which the position of the virtual projection plane VP0 is changed with the image center o in the camera coordinate system as the rotation center (or movement center). As shown in FIG. 10, the rotation about the x axis with the image center o as the rotation center is the actual pitch (also referred to as tilt), the rotation about the y axis is the actual yaw (also referred to as pan), and the Z axis. The rotation around is real roll.

  FIG. 11 is an example in which the position of the virtual projection plane VP is changed with the center ov of the virtual projection plane VP0 as the center of rotation based on the input rotation amount setting value. One of the two axes that are orthogonal to each other on the virtual projection plane VP0 is set as Yaw-axis, and the other is set as P-axis. Both are axes passing through the center ov, and rotation around the Yaw-axis with the center ov as the rotation center is called virtual yaw rotation, and rotation around the P-axis is called virtual pitch rotation. In the following, viewpoint conversion corresponding to rotating or changing the position of the virtual camera Ca0 is performed.

  In FIG. 9B, the second position V2 is set to the position in the initial state. The virtual projection plane VP at this time is parallel to the XY plane and the center ov is on the optical axis Z (corresponding to the position of VP0 in FIGS. 10 and 11). The first position V1 is an example in which the virtual pitch and the real pitch are both 45 degrees, and both are set to 90 degrees. In this case, the viewpoint is changed so as to look down at the subject. Note that the combination of position changes is not limited to this, and any one of parallel movement, real picth, real yaw, real roll, virtual pitch, virtual yaw, or a combination thereof may be used.

[C-4] Analysis and countermeasure of discontinuity of distortion correction coefficient:
Hereinafter, image defects that may occur by using the distortion correction coefficient shown in FIG. 5 will be described with reference to FIG. Also in FIG. 12, in consideration of a super-wide-angle lens having an incident angle exceeding 90 °, the original image height (w2c value) is obtained in the region where the incident angle exceeds 90 ° regardless of the above-described formula (1). The calculation is performed assuming that is infinite.

[C-4-1] Saturation of the second distortion correction coefficient L2:
Although the ideal distortion correction coefficient has been described with reference to FIG. 5, the second distortion correction coefficient L2 is a function of tan θ, and the w2c value as the image height increases rapidly as the incident angle θ approaches 90 °.

  That is, the second distortion correction coefficient L2 is a very large value even if the incident angle does not exceed 90 degrees and does not become infinite. For this reason, the image processing unit 101 may be limited by the arithmetic processing capability. For example, when the data size secured per w2c value in the image processing unit 101 is 9-bit processing capacity, the maximum processable value is 511, and values exceeding 511 are processed as 511. The For this reason, the image height h (w2c value) based on the second distortion correction coefficient L2 is saturated as shown in FIG. That is, since the second distortion correction coefficient L2 is a function of tan θ, when the incident angle θ is 90 ° or more or approaches 90 ° and the w2c value rapidly increases, the second distortion correction coefficient L2 is saturated due to the influence of the processing capability. L2 is a discontinuous curve.

[C-4-2] Effect on third distortion correction coefficient L3:
The saturation characteristic is similarly affected by the third distortion correction coefficient L3 generated by interpolation between the first distortion correction coefficient L1 and the second distortion correction coefficient L2. That is, as shown in FIG. 12, the image height h based on the third distortion correction coefficient L3 is similarly saturated in the region (b) that is the interpolation value due to the influence of the saturation of the second distortion correction coefficient L2, and becomes unsatisfactory. It becomes continuous and deforms.

[C-4-3] Effect on image:
When the second distortion correction coefficient L2 is saturated and the third distortion correction coefficient L3 becomes discontinuous, when distortion correction processing is performed using such a distortion correction coefficient, the distortion correction coefficient is saturated or discontinuous. As a result of the incorrect calculation, an image defect as shown in FIG. 13A occurs within the range of the corresponding incident angle. This image defect appears as an originally nonexistent image due to color bleeding or the like. Due to this image defect, it has been difficult to accurately recognize the subject. Hereinafter, image defects will be described.

[C-4-3-1] Image defect (1):
For example, as shown in the range surrounded by the one-dot chain line (a) in FIG. 13, an image that does not originally exist due to band-like color bleeding or the like within the range of the incident angle corresponding to saturation or discontinuity of the distortion correction coefficient. Appears as a (fake image). Due to this image defect, it has been difficult to accurately recognize the subject.

  The image defect due to the false image is treated as 0 because the w2c value saturated in the middle of the calculation overflows, and as a result, the pixel near the optical axis center coordinate (image height = 0) corresponding to w2c = 0 is saturated. It is a phenomenon that appears at the position of the incident angle.

  Although FIG. 13 shows a state in which the viewpoint conversion is performed together with the distortion correction process, even when only the distortion correction process is performed without performing the viewpoint conversion, the saturation of the second distortion correction coefficient L2 and the third distortion correction coefficient are performed. Due to the discontinuity of L3, a similar image defect occurs in any region of the corresponding incident angle.

  Further, since this image defect occurs in a region where the incident angle is a certain level or more, it may appear in a curved shape that matches the distortion of the image, instead of the straight belt shape as shown in FIG.

[C-4-3-3] Image defect (2):
In addition, in the region (b) of FIG. 12, due to the fact that the slope of the third distortion correction coefficient L3 is smaller than the original state, a value smaller than the original value is designated, and from the center to the outer side. There is also an image defect that can be observed as “smear” so as to spread. In this case, since “smear” spreads thinly, a specific portion is not shown in FIG.

[C-4-3-3] Image defect (3):
Further, pixels that originally do not exist appear as a black region like a region surrounded by a broken line in FIG. 13B by image processing such as distortion correction processing and viewpoint conversion processing. Similarly, the area outside the angle of view of the lens of the image pickup unit 110 is displayed as a gray area, such as an area surrounded by a broken line in FIG.

  These black areas and gray areas change in shape and size in accordance with the distortion correction rate and the degree of viewpoint conversion, and become an obstacle to the recognition of the subject. In addition, the black region and the gray region are generated in a region having an incident angle of a certain level or more as in the case of the above-described band-shaped image defect or image defect due to bleeding, and can be handled as an image defect.

[C-4-4] Blackout point:
In this embodiment, as shown in FIG. 12, when there is pixel data in a saturated region where there is a high possibility that an image defect will occur due to the acquisition of the second distortion correction coefficient L2 as described above, this will be described later. By taking countermeasures against image defects, image defects are prevented from being visualized. The content of the image defect countermeasure will be described later. Then, a blackout point is set as a threshold for determining whether or not the pixel data is a target for image defect countermeasures described later.

  The blackout point bop is a point at which the w2c value is saturated to a predetermined value or more in the second distortion correction coefficient because of the above-described image processing restriction, and the minimum incident angle corresponding to this bop is θbop. The image defect countermeasure is applied when there is a pixel corresponding to an incident angle larger than θbop. In the case where the degree of distortion correction is changed step by step using the third distortion correction coefficient obtained by interpolation between the first distortion correction coefficient and the second distortion correction coefficient, a predetermined boundary line passing through bop is defined. A pixel corresponding to an incident angle larger than the incident angle is set, with the predetermined incident angle of the third distortion correction coefficient determined based on the predetermined boundary line passing through the bop as the second blackout point. In the case where it exists, image defect countermeasures are taken similarly. Note that a predetermined boundary line passing through bop is also set as a saturated region, and is a threshold value for determining whether or not to take a countermeasure against an image defect with a saturated w2c value. For example, in the present embodiment, the point (c) in the second distortion correction coefficient L2 in FIGS. 14 and 15 is bop, and the boundary line is set to a straight line orthogonal to the axis of the incident angle θ in FIG. Similarly, in FIG. 15, the point (c) is bop, and a straight line that passes through this bop and is set with a predetermined inclination is a boundary line.

  Here, when the second distortion correction coefficient L2 is saturated as shown in FIG. 14A, the image processing unit 101 uses the second distortion correction coefficient L2 at θbop (FIG. 14) as a boundary line passing through the blackout point bop. The w2c value in (c)), the w2c value in the third distortion correction coefficient L3 (FIG. 14 (d1) to (d4)) at θbop as the second blackout point, and the first distortion at θbop as the third blackout point The w2c value in the correction coefficient L1 (FIG. 14 (d5)) is acquired (step S106 in FIG. 4). However, the first distortion correction coefficient L1 is not affected by the saturation of the second distortion correction coefficient L2.

  Note that the boundary line passing through the bop as shown in FIG. 14 is L2 and L1 when the optical system specification has a relatively narrow angle of view so that the incident angle does not exceed 90 degrees, or when there is no viewpoint conversion. This is useful when the time between changes is short or when the user places importance on not generating error pixels due to saturation.

  As another form of the boundary line passing through the blackout point bop, θbop and the maximum incident angle θbop2 of the lens of the imaging unit 110 are obtained as shown in FIG. The line connecting the point c) and the first distortion correction coefficient L1 at θbop2 (point (f) in FIG. 15) can be a boundary line line_bop (broken line in FIG. 15) passing through the blackout. Then, the image processing unit 101 calculates the w2c value at the bop, the w2c value at the third distortion correction coefficient L3 (points (e1) to (e4) in FIG. 15) intersecting the line_bop as the second blackout point, the third black As an out point, the w2c value at the first distortion correction coefficient L1 (point (f) in FIG. 14) at θbop2 is acquired (step S106 in FIG. 4). In this case, as the third distortion correction coefficient L3 approaches the first distortion correction coefficient L1, it is not affected by the saturation of the second distortion correction coefficient L2 due to the relationship of the interpolation ratio. is doing. Such a boundary line is used when the incident angle uses a partial value in the saturation region, such as in the case of an optical system specification with a wide incident angle exceeding 90 degrees, or when the angle of viewpoint conversion is large. This is useful when sudden changes are not desirable.

  Then, by appropriately setting the boundary line passing through the blackout point in this way, the threshold value for determining whether or not to deal with the above-described image defect is set, so that each distortion correction coefficient in the camera coordinate system is used in the subsequent distortion correction processing. It is determined whether to use each pixel data of the world coordinate system associated with the pixel data or to take countermeasures against image defects, and to apply image defect countermeasures if applicable.

[C-4-5] Image defect countermeasure:
[C-4-5-1] Countermeasure (1):
Here, θb is a value where the incident angle is close to 90 ° as shown in FIG. 14, and is generated at the corner of the image as shown in the image specific example of FIG.

Therefore, tan θ in the equation (1) for obtaining L2 is replaced with tan (αθ) as in the following equation (1 ′), and a predetermined coefficient less than 1 is assigned to α.
L2 = ((ExportImageSize / 2) / (tan (focal / 2))) × tan (αθ) Expression (1 ′)
For example, in the case of FIG. 15 with the above-mentioned specific (1), since the θb that is the blackout point is around 76 °, the actual incident angle θ = 90 ° that is the actual incident angle is Α is set to about 0.85 so that the corrected incident angle αθ = 76 °. In addition, the numerical value shown here is an example, and various changes are possible.

  That is, if the practical maximum value of the actual incident angle is θc, α can be obtained by α = θb / θc.

  As a result, image defects due to incorrect calculation results do not occur until the actual incident angle θ is about 90 °. Even in the case of dealing with an actual incident angle exceeding 90 °, the image defect occurs on the wide angle side before the countermeasure, so that it is difficult to influence the accurate recognition of the subject.

[C-4-5-2] Countermeasure (2):
Here, in the case of FIG. 15, in view of the influence of the second distortion correction coefficient L2 on the third distortion correction coefficient L3, θb1 to θb2 are set to 86 ° to 100 ° with respect to the incident angle serving as the blackout point.

  Also in this case, tan θ in equation (1) for obtaining L2 is replaced with tan (αθ) as in equation (1 ′), and a predetermined coefficient less than 1 is assigned to α.

  For example, in the case of FIG. 15 with the above-mentioned specific (1), θb1 that is the blackout point is around 76 °, so that the actual incident angle θ = 90 ° that is the actual incident angle is Α is set to about 0.85 so that the corrected incident angle αθ = 76 °. Also, θb2 of the first distortion correction coefficient L1 may be set to α = 1 because it is not affected by the saturation of the second distortion correction coefficient L2. In addition, the numerical value shown here is an example, and various changes are possible.

That is, assuming that the practical maximum value of the actual incident angle is θc, for each α of the third distortion correction coefficient L3 interpolated from the first distortion correction coefficient L1 and the second distortion correction coefficient L2, α corresponding to θb1 Is α1, and α corresponding to θb2 is α2,
α = DistRatio / 100 × α2 + (1−DistRatio / 100) × α1 Equation (4),
Α can be obtained in detail according to the interpolation ratio.

  Thereby, even when the third distortion correction coefficient L3 is gradually changed, image defects due to incorrect calculation results do not occur until the actual incident angle θ is about 90 °. Even in the case of dealing with an actual incident angle exceeding 90 °, the image defect occurs on the wide angle side before the countermeasure, so that it is difficult to influence the accurate recognition of the subject.

  That is, the image processing unit 101 calculates a predetermined coefficient α less than 1 when tan θ is tan (αθ) based on the above-described θb and θb1 · θb2 (step S107 in FIG. 4). Note that when θb is one type as shown in FIG. 14, the predetermined coefficient α is also one type. However, when it changes from θb1 to θb2 according to the distortion correction coefficient as shown in FIG. α also has a plurality of types according to the value of θb. That is, the image processing unit 101 determines a predetermined coefficient α less than 1 when tan θ is tan (αθ) according to the ratio of the second distortion correction coefficient L2 included in the third distortion correction coefficient L3.

[C-5] Creation of distortion correction processing LUT:
Then, the image processing unit 101 determines each correction factor and incident angle θ for the first distortion correction coefficient L1, the second distortion correction coefficient L2, the third distortion correction coefficient L3, and the predetermined coefficient α obtained as described above. From the w2c value according to the above, an LUT used for distortion correction processing when converting the coordinates of the virtual projection plane VP from the world coordinate system to the camera coordinate system is created (step S108 in FIG. 4). The generated LUT is stored in the storage unit 103.

  A different LUT may be provided for each distortion correction rate, or data corresponding to a plurality of distortion correction rates may be stored in one LUT.

  Here, FIG. 16 is a schematic diagram for explaining the world coordinate system. As shown in FIG. 16, point A (0, 0, Za), point B (0, 479, Zb), point C (639, 479, Zc), point D of the virtual projection plane VP in the world coordinate system. A plane surrounded by (639, 0, Zd) is divided into 640 × 480 pixel pixels Gv (total number of pixels: 307,000) at equal intervals, and coordinates of all the pixels Gv in the world coordinate system are obtained. Note that the values of the X and Y coordinates in FIG. 16 are examples, and the X and Y coordinates of the point A are displayed as zero for easy understanding.

  From the coordinates of the pixel Gv in the world coordinate system and the distortion correction coefficient acquired in steps S104 to S105, the coordinates Gi (x ′, y ′) in the corresponding camera coordinate system on the image sensor surface IA are calculated. Specifically, it is calculated from the incident angle θ with respect to the optical axis Z obtained from the distortion correction coefficient and the coordinates of each pixel Gv (reference document: International Publication No. 2010/032720).

  FIG. 17 is an explanatory diagram illustrating a correspondence relationship between the camera coordinate system xy and the imaging element surface IA. In FIG. 17, points a to d are obtained by converting the points A to D in FIG. 16 into the camera coordinate system by the LUT generated using the first distortion correction coefficient L1.

  In FIG. 16, the virtual projection plane VP surrounded by the points A to D is a rectangular plane. In FIG. 17, the area surrounded by the points a to d after the coordinate conversion to the camera coordinate system is (the virtual projection plane VP The shape is distorted (corresponding to position). FIG. 17 shows an example in which the barrel shape is distorted. However, depending on the characteristics of the optical system, the pincushion type and the Jinkasa type (a shape that changes into a barrel shape at the center and a straight line or a pincushion shape at the end) are distorted. In some cases.

[C-6] Distortion correction processing execution:
The image processing unit 101 generates an image subjected to distortion correction processing using the LUT while changing the distortion correction rate according to time as shown in FIG. 9 (step S109 in FIG. 4).

  FIG. 18 is a schematic diagram for explaining the distortion correction processing. FIG. 18 is an enlarged view of a part of the image data, and the illustrated grids indicate pixel divisions.

  The input image data in FIG. 18A is an enlarged display of a part of FIG. 17, and as described in FIGS. 16 to 17, the subject on the straight line is affected by the aberration on the image sensor surface IA. It is input on the curve (FIG. 18A, thick frames Gi1 to Gi3). The output image data is generated by using each pixel data of the output image data using the LUT and corresponding pixel data of the input image data.

  Since the LUT is generated in consideration of the distortion correction coefficient corresponding to each distortion correction factor, the LUT data generated based on the first distortion correction factor L1 of the distortion correction factor = 100% is finally used. In this case, the data of the pixel Gv1 (for example, luminance signal) refers to the data of Gi1 at the coordinates (100, 200) in FIG. Similarly, Gv2 and Gv3 refer to the data of Gi2 and Gi3, respectively.

  In addition, the image processing unit 101 performs processing such as changing the virtual projection plane based on the predetermined coefficient α in accordance with the LUT in the above-described image defect area at each distortion correction rate.

[C-7] Image output:
The image processing unit 101 outputs an image generated by distortion correction processing using the LUT while changing the distortion correction rate according to time as shown in FIG. 9 to the display unit 120 and displays the image (step S110 in FIG. 4). ). For example, the output image generated by the distortion correction process is output to the display unit 120 at, for example, a frame rate of 30 fps or 60 fps, and the entire process ends. This image output is the third step.

  As a result, the distortion is corrected in accordance with the distortion correction rate. When the distortion correction rate is 100%, as shown in FIG. The subject is generated (displayed) as the original straight line.

  Although not shown in FIG. 18, if the distortion correction rate = 0%, an image having a shape with the distortion shown in FIG. 18A as it is is output, and the distortion correction rate is larger than 0% and 100%. If it is smaller, an image having a shape with an intermediate distortion between FIGS. 18A and 18B is output.

  At this time, if the viewpoint conversion is set by setting the virtual projection plane VP in step S101, such processing is also performed by the image processing unit 101.

[D] Display example by distortion correction processing:
Hereinafter, a display example when the distortion correction processing of the present embodiment is applied will be described.

  19 (a) to 19 (f) show the display displayed on the display unit 120 by performing distortion correction processing by the image processing unit 101 using the above-described respective distortion correction coefficients, with the start point conversion looking down at 90 ° as described above. It is an example of an image.

  Here, FIG. 19A shows a distortion correction rate of 0%, FIG. 19F shows a distortion correction rate of 100%, and FIGS. 19B to 19E show the third distortion correction coefficient L3. Distortion correction rate is an intermediate stage of about 20% to 80%.

  In these drawings, tan θ at the time of calculating the second distortion correction coefficient L2 is tan (αθ), and a predetermined coefficient less than 1 is assigned to α, so that an incorrect calculation result can be avoided, and an image defect (FIG. 19). The band noise (b) is generated outside the display area. Although it is not clear in this drawing, “smudge” that occurs in the peripheral portion is effectively eliminated by taking measures against the image defect so as not to include the image defect in the peripheral portion as in this aspect. In addition, the display of gray areas and black areas has also decreased.

  Further, as described above, since the predetermined coefficient α is calculated based on the incident angle θb that becomes the blackout point, it is only necessary to change the minimum processing during processing so as not to include image defects. Changes to the output image are minimal.

  With the above processing, the display unit 120 displays the processed image in the middle stage in which distortion correction is performed based on the middle distortion correction rate and no image defect is included, with FIG. 19A as the initial image. At the same time, the display up to the final image shown in FIG. 19F is displayed while sequentially updating the display image at a constant speed. In the specific example of FIG. 9B, images with a low correction rate and position movement rate are displayed in order from a low image to a high image, and this is repeated at a cycle of 10 seconds.

  According to this embodiment, the virtual projection plane whose position is set is converted into a camera coordinate system using a correction coefficient, and the image data of the virtual projection plane is converted based on the coordinates of the converted camera coordinate system and the pixel data of the pixel. This is a case where the position of the virtual projection plane (camera viewpoint change) is changed in parallel with the distortion correction rate change by calculating the image data while changing the correction coefficient and the position data. However, it is not necessary to provide a separate process. It is possible to change the position and change the distortion correction rate at the same time without requiring new resource enhancement.

  In addition, according to the present embodiment, it is easy to accurately recognize an output image without distortion correction by changing the distortion correction rate stepwise from an output image without distortion correction to an output image with distortion correction. At this time, it is possible to accurately recognize the subject by not displaying the image defect caused by the saturation of the distortion correction coefficient.

  That is, erroneous recognition of a subject due to image defects caused by image processing limitations caused by entering a wide-angle lens at a very large angle can be effectively reduced.

[E] Other embodiments:
In the above embodiment, the distortion correction rate has been shown as a specific example of several steps, but the present invention is not limited to this. When displaying 10 seconds from before correction to after correction at a display rate of 30 fps, the distortion correction coefficient L3 is set to 300 levels, thereby enabling smooth display in which distortion correction is gradually changed for each frame. become. At this time, it is easier to accurately recognize the subject by setting the predetermined coefficient α so as not to cause an image defect for each frame.

DESCRIPTION OF SYMBOLS 100 Control apparatus 101 Image processing part 102 Setting part 103 Storage part 110 Imaging unit 120 Display part 130 Operation part VP Virtual projection surface LC Lens center plane IA Image sensor surface O Lens center o Image center L1 1st distortion correction coefficient L2 2nd distortion Correction coefficient L3 Third distortion correction coefficient

Claims (15)

An image processing method for obtaining image data processed using a plurality of pixel data obtained by receiving an image sensor having a plurality of pixels via an optical system,
A first step of setting a position and size of a virtual projection plane in the world coordinate system;
The coordinates in the world coordinate system of each pixel of the virtual projection plane set in the first step are converted into a camera coordinate system using a distortion correction coefficient, and based on the converted coordinates in the camera coordinate system and the plurality of pixel data A second step of calculating image data of the virtual projection plane set in the first step;
A third step of outputting a display image based on the image data calculated in the second step,
The distortion correction coefficient used in the second step is calculated based on at least the physical characteristics of the lens of the optical system and the incident angle of the incident light from the virtual projection plane set to the optical system. From the optical center at which the first distortion correction coefficient for correcting distortion caused by the optical system and the optical element of the optical system intersect with the image sensor calculated using the tangent function of the incident angle to the optical system as a variable And a second distortion correction coefficient that does not correct the distortion is included,
In the second step, when there is a minimum incident angle at which the second distortion correction coefficient is equal to or greater than a predetermined value, the incident angle is less than 1 so as to avoid pixels in a region corresponding to an incident angle larger than the minimum incident angle. Calculating the second distortion correction coefficient by multiplying by a predetermined coefficient of
An image processing method. The third step is to output a moving image for display of image data in which distortion is corrected in stages,
In the second step, the first distortion correction coefficient, the second distortion correction coefficient, and the third distortion correction coefficient obtained by interpolation from the first and second distortion correction coefficients are switched in stages and used for virtual projection. Calculate the image data of the surface,
The image processing method according to claim 1. The third step is to output a moving image for display of image data in which distortion is corrected in stages,
In the second step, the first distortion correction coefficient, the second distortion correction coefficient, and the third distortion correction coefficient obtained by interpolation from the first and second distortion correction coefficients are switched in stages and used for virtual projection. Calculate the image data of the surface,
In the second step, when there is a minimum incident angle at which the second distortion correction coefficient is equal to or greater than a predetermined value, the incident angle is less than 1 so as to avoid pixels in a region corresponding to an incident angle larger than the minimum incident angle. When calculating the second distortion correction coefficient by multiplying by a predetermined coefficient, a coefficient less than 1 is determined according to an interpolation ratio of the second distortion correction coefficient included in the third distortion correction coefficient;
The image processing method according to claim 1, wherein: In the first step, the first position and the second position of the virtual projection plane of the world coordinate system are set,
In the second step, the initial second position and the second distortion correction coefficient, the final first position and the first distortion correction coefficient, and interpolation between the second position and the first position in the middle are obtained. Calculating the image data of the virtual projection plane using the third position and the third distortion correction coefficient by switching in stages.
The image processing method according to any one of claims 1 to 3. The second distortion correction coefficient is expressed by the following conditional expression (1):
When considering the saturation of the second distortion correction coefficient, it is expressed by the following conditional expression (1 ′).
The image processing method according to claim 1, wherein the image processing method is an image processing method.
L2 = ((ExportImageSize / 2) / (tan (focal / 2))) × tanθ Equation (1)
L2 = ((ExportImageSize / 2) / (tan (focal / 2))) × tan (αθ) Expression (1 ′)
here,
ExportImageSize: the length of the long side of the output image to be displayed on the display,
focal: Angle of view in the long side direction of the set virtual projection plane,
α: a predetermined coefficient less than 1,
It is. An image processing apparatus that obtains image data processed using a plurality of pixel data obtained by receiving an image sensor having a plurality of pixels via an optical system,
A storage unit for storing distortion correction coefficients;
The coordinates in the world coordinate system of each pixel of the virtual projection plane whose position and size are set are converted into the camera coordinate system using the distortion correction coefficient stored in the storage unit, and the coordinates converted into the camera coordinate system and the plurality of the coordinates An image processing unit that calculates image data of the virtual projection plane based on the pixel data;
An image signal output unit that outputs an image signal for display of the image data calculated by the image processing unit,
In the storage unit, the stored distortion correction coefficient is calculated based on at least the physical characteristics of the lens of the optical system and the incident angle of the incident light from the virtual projection plane set on the optical system. A first distortion correction coefficient for correcting distortion caused by the optical system and a tangent function of an incident angle to the optical system are used as variables, and the optical element from the optical center where the optical axis of the optical system intersects. A second distortion correction coefficient which is an image height and does not correct the distortion;
When there is a minimum incident angle at which the second distortion correction coefficient is equal to or greater than a predetermined value, the image processing unit is less than 1 in the incident angle so as to avoid pixels in a region corresponding to an incident angle larger than the minimum incident angle. Multiplying a predetermined coefficient to calculate the tangent function,
An image processing apparatus. The image signal output unit outputs a moving image for display of image data in which distortion is corrected stepwise,
The image processing unit virtually switches the first distortion correction coefficient, the second distortion correction coefficient, and the third distortion correction coefficient obtained by interpolation from the first and second distortion correction coefficients in a stepwise manner. Calculating image data of the projection plane,
The image processing apparatus according to claim 6. The image signal output unit outputs a moving image for display of image data in which distortion is corrected stepwise,
The image processing unit virtually switches the first distortion correction coefficient, the second distortion correction coefficient, and the third distortion correction coefficient obtained by interpolation from the first and second distortion correction coefficients in a stepwise manner. Calculate the image data of the projection plane,
In the second step, when there is a minimum incident angle at which the second distortion correction coefficient is equal to or greater than a predetermined value, the incident angle is less than 1 so as to avoid pixels in a region corresponding to an incident angle larger than the minimum incident angle. When calculating the second distortion correction coefficient by multiplying by a predetermined coefficient, a coefficient less than 1 is determined according to an interpolation ratio of the second distortion correction coefficient included in the third distortion correction coefficient;
The image processing apparatus according to claim 6, wherein the image processing apparatus is an image processing apparatus. The virtual projection plane of the world coordinate system has a first position and a second position,
The image processing unit is obtained by interpolation between the initial second position and the second distortion correction coefficient, the final first position and the first distortion correction coefficient, and the intermediate second position and first position. Calculating the image data of the virtual projection plane using the third position and the third distortion correction coefficient by switching in stages.
The image processing apparatus according to claim 6, wherein the image processing apparatus is an image processing apparatus. The second distortion correction coefficient is expressed by the following conditional expression (1):
When considering the saturation of the second distortion correction coefficient, it is expressed by the following conditional expression (1 ′).
The image processing apparatus according to claim 6, wherein the image processing apparatus is an image processing apparatus.
L2 = ((ExportImageSize / 2) / (tan (focal / 2))) × tanθ Equation (1)
L2 = ((ExportImageSize / 2) / (tan (focal / 2))) × tan (αθ) Expression (1 ′)
here,
ExportImageSize: the length of the long side of the output image to be displayed on the display,
focal: Angle of view in the long side direction of the set virtual projection plane,
α: a predetermined coefficient less than 1,
It is. Optical system,
An imaging device having a plurality of pixels;
A storage unit for storing distortion correction coefficients of the optical system;
A coordinate in the world coordinate system of each pixel of the virtual projection plane is converted into a camera coordinate system using the distortion correction coefficient of the optical system, and the coordinates converted into the camera coordinate system and a plurality of light received by the imaging element are received. An image processing unit that calculates image data of the virtual projection plane based on pixel data;
An image signal output unit that outputs an image signal for display of the image data calculated by the image processing unit,
In the storage unit, the stored distortion correction coefficient is calculated based on at least the physical characteristics of the lens of the optical system and the incident angle of the incident light from the virtual projection plane set on the optical system. A first distortion correction coefficient for correcting distortion caused by the optical system and a tangent function of an incident angle to the optical system are used as variables, and the optical element from the optical center where the optical axis of the optical system intersects. A second distortion correction coefficient which is an image height and does not correct the distortion;
When there is a minimum incident angle at which the second distortion correction coefficient is equal to or greater than a predetermined value, the image processing unit is less than 1 in the incident angle so as to avoid pixels in a region corresponding to an incident angle larger than the minimum incident angle. Multiplying a predetermined coefficient to calculate the tangent function,
An imaging apparatus characterized by that. The image signal output unit outputs a moving image for display of image data in which distortion is corrected stepwise,
The image processing unit virtually switches the first distortion correction coefficient, the second distortion correction coefficient, and the third distortion correction coefficient obtained by interpolation from the first and second distortion correction coefficients in a stepwise manner. Calculating image data of the projection plane,
The imaging apparatus according to claim 11. The image signal output unit outputs a moving image for display of image data in which distortion is corrected stepwise,
The image processing unit virtually switches the first distortion correction coefficient, the second distortion correction coefficient, and the third distortion correction coefficient obtained by interpolation from the first and second distortion correction coefficients in a stepwise manner. Calculate the image data of the projection plane,
In the second step, when there is a minimum incident angle at which the second distortion correction coefficient is equal to or greater than a predetermined value, the incident angle is less than 1 so as to avoid pixels in a region corresponding to an incident angle larger than the minimum incident angle. When calculating the second distortion correction coefficient by multiplying by a predetermined coefficient, a coefficient less than 1 is determined according to an interpolation ratio of the second distortion correction coefficient included in the third distortion correction coefficient;
The imaging apparatus according to claim 11 or 12, characterized in that: The virtual projection plane of the world coordinate system has a first position and a second position,
The image processing unit is obtained by interpolation between the initial second position and the second distortion correction coefficient, the final first position and the first distortion correction coefficient, and the intermediate second position and first position. Calculating the image data of the virtual projection plane using the third position and the third distortion correction coefficient by switching in stages.
The image pickup apparatus according to claim 11, wherein the image pickup apparatus is an image pickup apparatus. The second distortion correction coefficient is expressed by the following conditional expression (1):
When considering the saturation of the second distortion correction coefficient, it is expressed by the following conditional expression (1 ′).
The imaging device according to any one of claims 11 to 14, wherein
L2 = ((ExportImageSize / 2) / (tan (focal / 2))) × tanθ Equation (1)
L2 = ((ExportImageSize / 2) / (tan (focal / 2))) × tan (αθ) Expression (1 ′)
here,
ExportImageSize: the length of the long side of the output image to be displayed on the display,
focal: Angle of view in the long side direction of the set virtual projection plane,
α: a predetermined coefficient less than 1,
It is.