JP2013005394A - Image processing method having wide-angle distortion correction processing, image processing apparatus and imaging apparatus - Google Patents

Abstract

PROBLEM TO BE SOLVED: To accurately recognize a subject on distortion correction processing, using a relatively small scale circuit.SOLUTION: A distortion correction coefficient at least includes a first distortion correction coefficient calculated based on a physical characteristic of a lens in an optical system 110 and an incident angle of incident light from a virtual projection plane VP set to the optical system, and a second distortion correction coefficient which is an image height from an optical center at which an imaging device intersects with the optical axis of the optical system, calculated from a tangent function of the incident angle to the optical system as a variable. An image data, calculated in an incident angle range that saturates due to the value of the distortion correction coefficient exceeding a predetermined upper limit value, is converted into a preset predetermined value.

Description

  The present invention relates to an image processing method, an image processing apparatus, and an imaging apparatus that perform distortion correction processing on an image captured by an imaging element via a wide-angle optical system including a condenser lens.

  In general, an image captured by an optical system having a short focal length lens or a large angle of view lens such as a wide-angle lens or a fish-eye lens is distorted, and image processing for correcting the distortion is performed.

  For such image processing, techniques related to the following patent documents have been proposed.

  Patent Document 1 discloses a correction method of the prior art that corrects distortion generated in a captured image captured using a lens with a short focal length using a lens correction parameter.

  In Patent Document 2, it is necessary to use an external information processing device to calculate the optical distortion correction parameter for each lens position from the wide end to the tele end of the optical zoom mechanism by using an interpolation operation. With the optical distortion correction parameter for the discrete lens position within the range to perform the optical zoom, the lens position at the time of zooming is limited to the lens position having the optical distortion correction parameter. The optical zoom between the positions is connected by electronic zoom.

  In Patent Document 3, distortion correction is performed depending on the difference in the angle of view switched by the angle-of-view switching means, such as performing distortion correction in the case of the angle of view on the wide angle side, and not performing distortion correction in the case of the angle of view other than the wide angle side. A video processing apparatus for changing the above is disclosed.

JP 2009-140066 A JP 2009-105546 A JP 2009-61969 A

  As disclosed in Patent Document 1, there is a problem that when a captured image obtained by a lens is hardwareized as an image processing device, the processing time becomes long, the circuit scale increases, and the cost increases. .

  In Patent Document 2, the lens position at the time of zooming is limited to a position corresponding to the discrete distortion correction parameter, and the zoom operation is omitted by interpolating the distortion correction parameter by connecting between them with an electronic zoom. In this way, the zoom operation of the image pickup device alone is realized. However, it can be applied only to a one-dimensional lens movement such as a zoom operation, and is difficult to apply to various movements such as panning and tilting. That is, there is a problem that it is not possible to cope with viewpoint conversion that changes the appearance by image processing.

  In addition, since the image after the distortion correction processing has a narrow viewing angle, it is difficult to recognize a wide area at a time. In an image that is not subjected to distortion correction processing, the viewing angle is widened, but on the other hand, it is difficult to recognize the sense of distance and size due to distortion of the subject.

  In Patent Document 3, the distortion correction amount is varied according to the angle of view for such a problem, but the distortion correction amount is not sequentially changed. It does not solve both of the two problems of accurate recognition.

  In addition, when generating and displaying an image after distortion correction from an entire image taken with a wide-angle lens and without distortion, it is clear which part of the entire image the final image after distortion correction displays. Therefore, it is desirable to display the distortion correction state while changing it little by little, but it has been found that each of the above patent documents is not suitable.

  In addition, when the inventor of the present application performs distortion correction on the subject by a method different from that of the above-described patent document and continuously changes the distortion correction state little by little, the incident angle is about 90 degrees or 90 degrees. It has been clarified that when the angle is very large, the distortion-corrected image is disturbed due to the number of processing bits of the image processing for calculating the distortion-corrected image data. That is, the present inventors have found that it is difficult to accurately recognize a subject because there is a disorder (image defect) in an image displaying a distortion correction state.

  In view of such a problem, the present invention provides an image processing method, an image processing apparatus, and an imaging device capable of accurately recognizing a subject and shortening a processing time with a relatively small circuit. The object is to provide a device.

  The above object is achieved by the invention described below.

  (1) In an image processing method for obtaining image data processed using a plurality of pixel data obtained by receiving an image sensor having a plurality of pixels via an optical system, the position of a virtual projection plane in the world coordinate system and A first step of setting a size, and a coordinate in the world coordinate system of each pixel of the virtual projection plane set in the first step is converted into a camera coordinate system using a distortion correction coefficient, and in the converted camera coordinate system A second step of calculating image data of the virtual projection plane set in the first step based on the coordinates and the plurality of pixel data, and a second step of outputting a display image based on the image data calculated in the second step. The distortion correction coefficient used in the second step includes at least the physical characteristics of the lens of the optical system and the setting to the optical system. The first distortion correction coefficient for correcting the distortion caused by the optical system calculated based on the incident angle of the incident light from the virtual projection plane and the tangent function of the incident angle to the optical system are calculated as variables. And a second distortion correction coefficient that does not correct the distortion and includes an image height from an optical center at which the imaging element and the optical axis of the optical system intersect. And a step of converting pixel data corresponding to an incident angle larger than the minimum incident angle to a predetermined value when there is a minimum incident angle at which a distortion correction coefficient is a predetermined value or more. To do.

  An image processing apparatus or an imaging apparatus that obtains image data processed using a plurality of pixel data obtained by receiving an image sensor having a plurality of pixels via an optical system, and stores a distortion correction coefficient The coordinates in the world coordinate system of each pixel of the virtual projection plane whose position and size are set are converted into the camera coordinate system using the distortion correction coefficient stored in the storage unit, and converted into the camera coordinate system An image processing unit that calculates image data of the virtual projection plane based on the coordinates and the plurality of pixel data; an image signal output unit that outputs an image signal for display of the image data calculated by the image processing unit; And the storage unit stores at least the physical characteristics of the lens of the optical system and the incident light from the virtual projection plane set to the optical system as the stored distortion correction coefficient. A first distortion correction coefficient for correcting distortion caused by the optical system calculated based on an angle and a tangent function of an incident angle to the optical system and the optical axis of the optical system calculated as variables And a second distortion correction coefficient that does not correct the distortion, and the image processing unit has a minimum incident angle at which the value of the second distortion correction coefficient is equal to or greater than a predetermined value. In some cases, pixel data corresponding to an incident angle larger than the minimum incident angle is converted into a predetermined value.

  (2) In the above (1), the third step outputs a moving image for display of image data in which distortion is corrected stepwise, wherein the first distortion correction coefficient in the second step, The image data of the virtual projection plane is calculated by switching the second distortion correction coefficient and the third distortion correction coefficient obtained by interpolation from the first and second distortion correction coefficients in a stepwise manner. .

  In the above (1), the image signal output unit outputs a moving image for display of image data in which distortion is corrected stepwise, and the image processing unit includes the first distortion correction coefficient. The image data of the virtual projection plane is calculated by stepwise switching and using the second distortion correction coefficient and the third distortion correction coefficient obtained by interpolation from the first and second distortion correction coefficients. To do.

  (3) In the above (2), when there is a minimum incident angle at which the value of the second distortion correction coefficient is not less than a predetermined value in the second step, the angle corresponding to an incident angle larger than the minimum incident angle. When the pixel data is converted into a predetermined value, the incident angle when the pixel data is converted into the predetermined value is changed according to the interpolation ratio of the second distortion correction coefficient included in the third distortion correction coefficient. It is characterized by.

  In the above (2), the image processing unit determines a minimum incident angle at which the second distortion correction coefficient is equal to or greater than a predetermined value, and the minimum incident angle at which the second distortion correction coefficient is equal to or greater than a predetermined value. When the pixel data corresponding to an incident angle larger than the minimum incident angle is converted into a predetermined value, the second distortion correction coefficient included in the third distortion correction coefficient The incident angle at the time of the conversion is changed in accordance with an interpolation ratio.

  (4) In the above (1) to (3), in the first step, the first position and the second position of the virtual projection plane of the world coordinate system are set, and in the second step, the initial second A position and the second distortion correction coefficient, a final first position and the first distortion correction coefficient, a third position obtained by interpolation between the second position and the first position in the middle, and the third distortion correction coefficient The image data of the virtual projection plane is calculated by switching in a stepwise manner.

  Further, in the above (1) to (3), the virtual projection plane of the world coordinate system has a first position and a second position set, and the image processing unit performs the initial second position and the second position. 2 stepwise correction coefficients, the final first position and the first distortion correction coefficient, the third position obtained by interpolation of the second position and the first position in the middle, and the third distortion correction coefficient The image data of the virtual projection plane is calculated by switching to the above.

(5) In the above (1) to (4), the second distortion correction coefficient is represented by the following conditional expression (1).
L2 = ((ExportImageSize / 2) / (tan (focal / 2))) × tanθ Equation (1)
here,
ExportImageSize: the length of the long side of the output image to be displayed on the display,
focal: Angle of view in the long side direction of the set virtual projection plane,
It is.

  In the present invention, at least one of the first distortion correction coefficient calculated based on the physical characteristics of the lens of the optical system and the second distortion correction coefficient calculated by the relational expression using the incident angle to the optical system as a variable. Is used to calculate image data using a simple method of erroneously recognizing a subject due to image defects caused by image processing limitations caused by incidence on a wide-angle lens at a very large angle. Can be reduced.

It is a schematic diagram explaining the distortion correction which concerns on this embodiment. It is explanatory drawing which shows the example which moved the position of the virtual projection surface VP. It is a block diagram which shows schematic structure of an imaging device. It is a flowchart which shows the control flow of this embodiment. FIG. 6 is a characteristic diagram showing a relationship between an incident angle θ and an image height h on the image sensor surface IA. It is explanatory drawing which shows the state which set the virtual projection surface VP in the optical range. It is a schematic diagram which shows the relationship between the image height h in the 1st distortion correction coefficient L1 and the 2nd distortion correction coefficient L2. It is explanatory drawing which shows the example of an input-output image. It is explanatory drawing which shows the example which changes a distortion correction rate or a distortion correction rate and a viewpoint conversion rate with a predetermined period. It is a characteristic view which shows the saturation and discontinuous state of a distortion correction coefficient. It is explanatory drawing which shows the image defect by the influence of the saturation and discontinuity of a distortion correction coefficient with a photograph image. It is a characteristic view which shows the blackout point of this embodiment. It is a characteristic view which shows the blackout point of this embodiment. It is a schematic diagram explaining a coordinate system. It is a figure which shows the correspondence of camera coordinate system xy and image pick-up element surface IA. It is a schematic diagram explaining a distortion correction process. This is an example in which the position of the virtual projection plane VP is changed with the image center o of the camera coordinate system as the rotation center. This is an example in which the position of the virtual projection plane VP is changed with the center ov of the virtual projection plane VP0 as the rotation center. It is explanatory drawing which shows the example of a display of the process image to which this embodiment is applied with a photograph image. It is explanatory drawing which shows the example of a display of the process image to which this embodiment is applied with a photograph image.

  Hereinafter, the present invention will be described based on embodiments, but the present invention is not limited to the specific examples shown in the embodiments.

[A] Coordinate system:
FIG. 1 is a schematic diagram showing a coordinate system for explaining distortion correction according to the present embodiment.

  In FIG. 1, X, Y, and Z are world coordinate systems as coordinate systems for indicating the position of an object in space, and the origin O is the lens center of the optical system. Here, Z includes the optical axis, and the XY plane includes a lens center plane LC passing through the lens center O. Point P is an object point of the subject in the world coordinate system XYZ. θ is an incident angle with respect to the optical axis (coincident with the Z axis).

  On the other hand, x and y are camera coordinate systems based on the camera, and the xy plane corresponds to the image sensor surface IA. o is the optical center, which is the intersection of the optical axis Z and the image sensor surface. The point p is a point on the image sensor surface in the camera coordinate system, and the object point P is a distortion correction coefficient (a first distortion correction coefficient described later) based on a parameter based on the physical characteristics of the lens (hereinafter referred to as “lens parameter”). (Corresponding to L1) and converted to the camera coordinate system.

  In FIG. 1, VP is a virtual projection plane. The virtual projection plane VP is set on the opposite side of the imaging element (and imaging element surface IA) with respect to the lens position (lens center plane LC) of the optical system. The virtual projection plane VP can be moved and changed in size based on an instruction from the user to the operation unit 130 (see the block diagram of FIG. 3). In the specification of the present application, “position change” is a concept including not only the case where the virtual projection plane VP is translated on the XY plane, but also an angle change (also referred to as an attitude change) with respect to the XY plane.

  In an initial state (initial position setting, the same applies hereinafter), the virtual projection plane VP is arranged at a predetermined position (Z direction) parallel to the lens center plane LC (XY direction) with a predetermined size, and the center of the virtual projection plane VP. ov is located on the Z axis. Gv is a point where the object point P is projected onto the virtual projection plane VP, and is an intersection of the object point P and a straight line passing through the lens center O and the virtual projection plane VP.

  FIG. 2 shows a state in which the virtual projection plane VP0 is rotated on the XZ plane based on the input from the operation unit 130 to obtain the virtual projection plane VP1.

[B] Configuration:
FIG. 3 is a block diagram illustrating a schematic configuration of the imaging apparatus used in the present embodiment. The imaging device includes an imaging unit 110, a control device 100, a display unit 120, and an operation unit 130.

  The imaging unit 110 includes a short focus lens, an imaging element, and the like. In the present embodiment, examples of the short focus lens include a wide angle lens and a fisheye lens.

  The control device 100 includes an image processing unit 101, a setting unit 102, and a storage unit 103.

  The setting unit 102 sets the position and size of the virtual projection plane VP based on an input instruction to the operation unit 130.

  The image processing unit 101 creates a conversion table of each coordinate on the virtual projection plane into the camera coordinate system based on the set position and size of the virtual projection plane VP, and shoots with the imaging unit 110 using the conversion table. The processed pixel data is processed to create image data to be displayed on the display unit 120. That is, the image processing unit 101 performs coordinate transformation that applies to which coordinate position of the camera coordinates the world coordinate position corresponds to based on the lens data. At this time, distortion correction and rotation, translation, enlargement, and reduction associated with calculation of the position of the virtual projection plane VP are performed together. The image processing unit 101 also functions as an image signal output unit that outputs an image signal for display.

  The storage unit 103 stores coefficients calculated based on parameters related to physical characteristics of the lens based on the physical characteristics of the lens (real lens) of the imaging unit 110 and the incident angle of incident light from the set virtual projection plane, that is, Incidence of incident light incident on the lens from the set virtual projection plane, which corresponds to a first distortion correction coefficient L1 that is a coefficient for correcting distortion generated in the optical system, and a coefficient for not correcting the distortion. A second distortion correction coefficient L2, which is an image height from the optical center at which the image sensor and the optical axis of the optical system intersect, calculated using a tangent function of the angle as a variable, is stored.

  The storage unit 103 also stores the position and size of the virtual projection plane VP and the created conversion table. The distortion correction coefficient can be obtained by obtaining lens calibration data, calculating based on the fθ characteristic of the lens, or the like.

  Note that “do not perform distortion correction” in the present embodiment means that the distortion as it is captured is substantially generated, and is slightly inferior from that in which distortion correction is performed satisfactorily. It does not point.

  The display unit 120 includes a display screen such as a liquid crystal display, and sequentially displays a display image based on the image data created by the image processing unit 101 based on the pixel data captured by the imaging unit 110 on the display screen.

  The operation unit 130 includes a keyboard, a mouse, or a touch panel arranged so as to be superimposed on the liquid crystal display of the display unit, and receives a user's input operation.

[C] Control:
Hereinafter, the control of this embodiment will be described with reference to the flowchart of FIG. According to this control flowchart, a moving image is displayed by processing input images that are continuously input and continuously displaying output images on the display unit 120.

[C-1] Setting and input:
First, setting of a virtual projection plane, setting of viewpoint conversion, setting of distortion correction conditions, and the like are input (step S101 in FIG. 4). This is the first step. These various settings are performed by setting the position and size of the virtual projection plane VP in the world coordinate system by the setting unit 102 based on an input instruction to the operation unit 130 by the user. Also, parameters relating to the physical characteristics of the lens used in the imaging unit 110 are read from the storage unit 103.

  Then, an image signal is input from the imaging unit 110, and the input image is taken into the control device 100 at a frame rate of 60 fps, for example (step S102 in FIG. 4).

[C-2] Virtual projection plane calculation:
Here, the image processing unit 101 calculates the position of the virtual projection plane VP by calculating the translation vector and the rotation vector based on the set condition and performing the Eugrid conversion (step in FIG. 4). S103). Note that the distortion correction coefficient acquisition and distortion correction processing described later from this virtual projection plane calculation are the second step.

[C-3] Acquisition of distortion correction coefficient:
Then, the image processing unit 101 uses a first distortion correction coefficient L1 for performing coordinate conversion that applies to which coordinate position of the camera coordinates the world coordinate position corresponds to based on data relating to the physical characteristics of the lens. The second distortion correction coefficient L2 is acquired (step S104 in FIG. 4).

  Here, the distortion correction coefficient acquired by the image processing unit 101 includes parameters relating to physical characteristics of the lens, which are coefficients for correcting distortion generated in the optical system, and the incident angle of incident light from the set virtual projection plane. The imaging is calculated using as a variable the first distortion correction coefficient L1 determined on the basis of the tangent function of the incident angle θ (see FIG. 1) of incident light incident from the virtual projection plane set to the optical system. A second distortion correction coefficient L2 that is an image height from the optical center where the element and the optical axis of the optical system intersect is included. The second distortion correction coefficient L2 is a distortion correction coefficient corresponding to not performing distortion correction, and will be described in detail later.

  Here, the first distortion correction coefficient L1 may be rewritten at any time, for example, every frame as required by the user. That is, in addition to acquiring the first distortion correction coefficient L1 by inputting the position and screen size of the virtual projection plane at the first step in the setting unit 102, the value of the first distortion correction coefficient L1 is directly changed. A configuration in which parameters can be input may be employed, whereby a relationship between pixels on camera coordinates and corresponding pixels on world coordinates may be arbitrarily changed.

  In the case of the present invention, it is also possible to change the image height position from the optical axis center where the incident angle and the optical axis intersect from the rewritten first distortion correction coefficient L1, the coordinates on the virtual projection plane, and the screen size. is there.

  This makes it possible to respond flexibly to various optical systems as compared to the case where the corresponding pixel position (region) on the camera coordinates is fixed in advance, and various distortions that do not depend on the pixel position (region) on the camera coordinates. There is an effect that the correction condition can be adopted.

  Further, the image processing unit 101 acquires a plurality of third distortion correction coefficients L3 so as to smoothly perform coordinate conversion by interpolating between the first distortion correction coefficient L1 and the second distortion correction coefficient L2. (Step S105 in FIG. 4).

[C-3-1] Description of distortion correction coefficient:
Hereinafter, the distortion correction coefficient will be described.

[C-3-1-1] Description of the first distortion correction coefficient L1 and the second distortion correction coefficient L2:
Here, the first distortion correction coefficient L1 and the second distortion correction coefficient L2 will be described with reference to FIG.

  FIG. 5 is a characteristic diagram showing a relationship in which the incident angle θ is the horizontal axis and the image height h (w2c value) from the optical center o intersecting the optical axis Z on the image sensor surface IA is the vertical axis.

  5, each of the first distortion correction coefficient L1 (solid line in FIG. 5), the second distortion correction coefficient L2 (one-dot chain line), and the third distortion correction coefficient L3 (two-dot chain line) by interpolation between L1 and L2. The relationship between the incident angle θ and the image height h is displayed.

  FIG. 6 shows a state in which the virtual projection plane VP is set in the optical range determined by the optical system. FIG. 7 is a characteristic diagram showing the relationship between the image heights h (indicated by h1 and h2) in the first distortion correction coefficient L1 and the second distortion correction coefficient L2. In FIGS. 6 and 7, the same reference numerals are given to the portions common to FIG.

  As shown in the characteristic diagram of FIG. 5, with the second distortion correction coefficient L2, distortion aberration corresponding to the incident angle θ occurs, and as the incident angle θ increases, the image height h (w2c value) increases and the amount of distortion aberration increases. Also grows. That is, the distortion correction rate = 0%, and by using the coefficient, image data in which distortion corresponding to the optical system is not corrected as it is can be obtained.

  On the other hand, in the characteristic diagram of FIG. 5, the first distortion correction coefficient L1 is a distortion correction coefficient set as a distortion correction rate = 100% so as to cancel the influence of the distortion aberration caused by the lens. Is a coefficient determined by the incident characteristics of the incident light entering the optical system from the set virtual projection plane.

  Further, the third distortion correction coefficient L3 obtained by interpolation between the first distortion correction coefficient L1 and the second distortion correction coefficient L2 has a distortion correction rate according to the interpolation ratio between the first distortion correction coefficient L1 and the second distortion correction coefficient L2. It is determined. The third distortion correction coefficient L3 and the interpolation ratio will be described later.

Here, the second distortion correction coefficient L2 is a distortion correction coefficient related to the image height from the optical center where the imaging element and the optical axis of the optical system intersect, which is calculated using the incident angle θ to the optical system as a variable. As the relational expression, for example, the following expression (1) is used. Note that there is an advantage that the output image can be easily scaled by multiplying the expression (1) several times.
L2 = ((ExportImageSize / 2) / (tan (focal / 2))) × tanθ ... Formula (1),
In this equation (1), “ExportImageSize” is the length of the long side (usually the horizontal direction, x) of the output image displayed on the display unit, and the unit is pixel. This coincides with the size of the long side of the virtual projection plane VP.

  In the formula (1), “focal” means the angle of view in the long side direction of the set virtual projection plane. Here, “focal” will be described with reference to FIG. In FIG. 6, ω is an optical angle of view corresponding to the optical range, and “focal” is an angle of view in the long side direction of the virtual projection plane VP. Note that a relational expression of orthographic projection and equidistant projection may be used instead of the expression (1). Further, “focal” corresponds to a display range within an optical range arbitrarily set by the user, and does not necessarily have to be in the long side direction of the angle of view.

  Note that the incident angle may exceed 90 ° in a super-wide-angle lens such as a fish-eye lens. In this case, in the above-described equation (1), when the incident angle exceeds 90 °, the sign of tanθ is inverted, so that it deviates from the original value of the second distortion correction coefficient L2. For this reason, in the present embodiment, the calculation is performed assuming that the image height (w2c value) is infinite in the region where the incident angle exceeds 90 ° regardless of the above-described equation (1).

  In the present embodiment, the conditional expression (1) has been described, but the conditional expression is not necessarily limited to this. In short, the image height with the incident angle θ as a variable, that is, on the camera coordinates from the optical center. Any conditional expression that calculates the distance may be used.

  FIG. 7 is a schematic diagram showing the relationship between the first distortion correction coefficient L1 and the image height h at the second distortion correction coefficient L2. The image height h is determined by the incident angle θ and the correction coefficient as shown in FIG. In the second distortion correction coefficient L2, the image height (distance from the optical axis Z) of the subject (object point P) is proportional to the image height h on the image sensor surface IA. Further, the image height h2 at the second distortion correction coefficient L2 is longer than the image height h1 of the first distortion correction coefficient L1, as shown in FIG.

  Here, in order to clarify the meaning of the first distortion correction coefficient L1 and the second distortion correction coefficient L2, the input image, the image processed with the first distortion correction coefficient L1, and the second distortion correction coefficient L2 are used. An example of the processed image is shown in FIG.

  FIG. 8 shows an example in which the virtual projection plane VP is set to the initial state. FIG. 8A shows an input image obtained by an imaging unit 110 having a wide-angle lens on a rectangular chart installed on a rectangular wall surface. 8B shows an output image when a distortion correction process described later is performed using the first distortion correction coefficient L1, and FIG. 8C shows a distortion correction process described later using the second distortion correction coefficient L2. It is an example of the output image at the time of performing.

  In the example shown in FIG. 8B, it can be seen that the distorted curve in the input image is corrected to be a straight line. On the other hand, when the distortion correction processing is performed with the second distortion correction coefficient L2 as shown in FIG. 8C, the output image is almost the same as the input image shown in FIG. That is, it can be seen that an output image equivalent to the case where distortion correction is not performed is obtained.

  Further, when distortion correction processing is performed using the third distortion correction coefficient L3, an image in a distortion correction state intermediate between FIGS. 8B and 8C is obtained as described later.

[C-3-1-2] Description of distortion correction coefficient L3:
Here, the third distortion correction coefficient L3 will be described. In calculating the distortion correction coefficient, the third distortion correction coefficient L3 calculated by interpolation between the first distortion correction coefficient L1 and the second distortion correction coefficient L2 can be switched stepwise by changing the distortion correction rate (DistRatio). Is calculated.

  Further, in this embodiment, the position of the virtual projection plane VP and the third position V3 calculated by interpolation between the first position V1 and the second position V2 are also switched stepwise in accordance with the change of the viewpoint conversion rate (VCRatio). ing. A specific example of the viewpoint conversion will be described later.

  The third distortion correction coefficient L3 and the third position V3 are calculated by the following equations (2) and (3).

L3 = DistRatio / 100 × L1 + (1−DistRatio / 100) × L2 Equation (2),
However, 0 ≦ DistRatio / 100 ≦ 1.

V3 = VCRatio / 100 × V1 + (1−VCRatio / 100) × V2 Equation (3),
However, 0 ≦ VCRatio / 100 ≦ 1.

  Here, the distortion correction rate and the viewpoint conversion rate are calculated in accordance with the processing time set in the condition setting (step S101 in FIG. 1). For example, when 10 sec is set to 1 cycle, if the time is 0.0 sec, the distortion correction rate and the viewpoint conversion rate are both 0%, and if the time is 5.0 sec, the distortion correction rate and the viewpoint conversion are set. If the rate is 50% and the time is 10.0 sec, the distortion correction rate and the viewpoint conversion rate are both 100%.

  The distortion correction coefficient L3 is calculated from the distortion correction factor calculated in this way based on the above-described equation (2). Note that when the distortion correction rate is 0%, the second distortion correction coefficient L2 itself, and when 100%, the first distortion correction coefficient L1 itself.

  Further, the third position of the virtual projection plane VP is calculated from the viewpoint conversion rate. The second position, which is the initial position of the virtual projection plane VP corresponding to the second distortion correction coefficient L2, and the first position, which is the final position of the virtual projection plane VP corresponding to the first distortion correction coefficient L1, are: This is set in S101. In other words, the calculation of the third position is performed by using the viewpoint conversion rate for the coordinates of the first position and the second position of the virtual projection plane VP, using Equation (3).

  Here, FIG. 9 shows an example in which the distortion correction rate or the distortion correction rate and the viewpoint conversion rate are changed at a predetermined cycle.

  FIG. 9A shows an example in which the third distortion correction coefficient L3 is changed stepwise by 10 points every second by changing the distortion correction rate with 10 seconds as one cycle. Note that viewpoint conversion is not performed in the example of FIG.

  FIG. 9B shows an example in which the third distortion correction coefficient L3 is changed stepwise by 10 points every second by changing the distortion correction rate with 10 seconds as one cycle. Further, the third position V3 is also changed stepwise by changing the viewpoint conversion rate. In the example of FIG. 9B, the distortion correction rate is also changed as the viewpoint conversion rate is changed. 9 (a) and 9 (b) show an example in which the period is changed in 10 seconds and 11 steps in all, but it is only an example, and the cycle can be changed arbitrarily, or in more detailed steps than this. It may be changed.

[C-4] Analysis and countermeasure of discontinuity of distortion correction coefficient:
Hereinafter, image defects that may occur by using the distortion correction coefficient shown in FIG. 5 will be described with reference to FIG. In FIG. 10 as well, in consideration of a super-wide-angle lens with an incident angle exceeding 90 °, the original image height (w2c value) is in the region where the incident angle is 90 ° or more regardless of the above-described equation (1). Calculate as if it is infinite.

[C-4-1] Saturation of the second distortion correction coefficient L2:
Although the ideal distortion correction coefficient has been described with reference to FIG. 5, since the second distortion correction coefficient L2 is a function of tan θ, the w2c value as the image height increases rapidly as the incident angle θ approaches 90 °. .

  That is, the second distortion correction coefficient L2 is a very large value even if the incident angle does not exceed 90 degrees and does not become infinite. For this reason, the image processing unit 101 may be limited by the arithmetic processing capability.

  For example, when the data size secured per w2c value in the image processing unit 101 is 9-bit processing capacity, the maximum processable value is 511, and values exceeding 511 are processed as 511. The For this reason, the image height h (w2c value) by the second distortion correction coefficient L2 has a saturated characteristic as shown in FIG. That is, since the second distortion correction coefficient L2 is a function of tan θ, when the incident angle θ is 90 ° or more or approaches 90 ° and the w2c value rapidly increases, the second distortion correction coefficient L2 is saturated due to the influence of the processing capability. L2 is a discontinuous curve.

[C-4-2] Effect on third distortion correction coefficient L3:
The saturation characteristic is similarly affected by the third distortion correction coefficient L3 generated by interpolation between the first distortion correction coefficient L1 and the second distortion correction coefficient L2. That is, as shown in FIG. 10, the image height h by the third distortion correction coefficient L3 is also saturated in the region (b) which is the interpolation value due to the influence of the saturation of the second distortion correction coefficient L2, and becomes unsatisfactory. It becomes continuous and deforms.

[C-4-3] Effect on image:
When the second distortion correction coefficient L2 is saturated and the third distortion correction coefficient L3 becomes discontinuous, when distortion correction processing is performed using such a distortion correction coefficient, the distortion correction coefficient is saturated or discontinuous. Thus, an image defect may occur within a corresponding incident angle range. Hereinafter, image defects will be described.

[C-4-3-1] Image defect (1):
For example, as shown in a range surrounded by a one-dot chain line (a) in FIG. 11, an image that does not originally exist due to a band-like color blur or the like within a range of incident angles corresponding to saturation or discontinuity of a distortion correction coefficient. Appears as a (fake image). Due to this image defect, it has been difficult to accurately recognize the subject.

  The image defect due to the false image is treated as 0 because the w2c value saturated in the middle of the calculation overflows, and as a result, the pixel near the optical axis center coordinate (image height = 0) corresponding to w2c = 0 is saturated. It is a phenomenon that appears at the position of the incident angle.

  FIG. 11 shows a state in which the viewpoint conversion is performed together with the distortion correction processing. However, even when only the distortion correction processing is performed without performing the viewpoint conversion, the saturation of the second distortion correction coefficient L2 and the third distortion correction coefficient are performed. Due to the discontinuity of L3, a similar image defect occurs in any region of the corresponding incident angle.

  Further, since this image defect occurs in a region where the incident angle is a certain level or more, it may appear in a curved shape that matches the distortion of the image, instead of the straight belt shape as shown in FIG.

[C-4-3-3] Image defect (2):
In addition, in the region (b) of FIG. 10, due to the fact that the slope of the third distortion correction coefficient L3 is smaller than the original state, a value smaller than the original value is designated, and from the center to the outer side. There is also an image defect that can be observed as “smear” so as to spread. In this case, since “smear” spreads thinly, it is difficult to show a specific portion in FIG.

[C-4-3-3] Image defect (3):
Further, pixels that do not originally exist appear as a black region like a region surrounded by a broken line in FIG. 11B by image processing such as distortion correction processing and viewpoint conversion processing. Similarly, a region outside the angle of view of the lens of the imaging unit 110 is displayed as a gray region like a region surrounded by a broken line in FIG.

  These black areas and gray areas change in shape and size in accordance with the distortion correction rate and the degree of viewpoint conversion, and become an obstacle to the recognition of the subject. In addition, the black region and the gray region are generated in a region having an incident angle of a certain level or more as in the case of the above-described band-shaped image defect or image defect due to bleeding, and can be handled as an image defect.

[C-4-4] Blackout point:
In the present embodiment, as shown in FIG. 10, pixel data in a saturated region that is highly likely to cause an image defect by acquiring the second distortion correction coefficient L2 as described above is determined in advance. By converting it to a value, image defects are prevented from being visualized. For example, for the broken line area (b) in FIG. 10, the image data is forcibly converted to a value such as black (R = 0, G = 0, B = 0). (Of course, the conversion value is not necessarily limited to this.)
Then, a blackout point is set as a threshold for determining whether or not the pixel data is the target of such conversion processing.

  The blackout point bop is a point at which the w2c value is saturated to a predetermined value or more in the second distortion correction coefficient due to the above-described image processing restriction, and the minimum incident angle corresponding to this bop is θbop. . The conversion process to a predetermined value as described above is applied to a pixel corresponding to an incident angle larger than such θbop.

  In the case where the degree of distortion correction is changed step by step using the third distortion correction coefficient obtained by interpolation between the first distortion correction coefficient and the second distortion correction coefficient, a predetermined boundary line passing through bop is defined. A pixel corresponding to an incident angle larger than the incident angle is set, with a predetermined incident angle of the third distortion correction coefficient determined based on a predetermined boundary line passing through the bop as a second blackout point. Similarly, the pixel is subjected to conversion processing to a predetermined value.

  Note that the predetermined boundary line passing through bop is also set as a saturated region, and is a threshold value for determining whether or not to convert the saturated w2c value into a predetermined value. For example, in the present embodiment, point (c) in the second distortion correction coefficient L2 in FIGS. 12 and 13 is bop, and the boundary line is set to a straight line orthogonal to the axis of the incident angle θ in FIG.

  Similarly, in FIG. 13, the point (c) is bop, and a straight line that passes through this bop and is set with a predetermined inclination is a boundary line.

  Here, when the second distortion correction coefficient L2 is saturated as shown in FIG. 12A, the image processing unit 101 uses the second distortion correction coefficient L2 at θbop (FIG. 12) as a boundary line passing through the blackout point bop. The w2c value in (c)), the w2c value in the third distortion correction coefficient L3 ((d1) to (d4) in FIG. 12) at θbop as the second blackout point, and the first distortion at θbop as the third blackout point The w2c value in the correction coefficient L1 (FIG. 12 (d5)) is acquired (step S106 in FIG. 4). However, the first distortion correction coefficient L1 is not affected by the saturation of the second distortion correction coefficient L2.

  Note that the boundary line passing through the bop as shown in FIG. 12 is L2 and L1 when the optical system specification has a relatively narrow angle of view so that the incident angle does not exceed 90 degrees, or when there is no viewpoint conversion. This is useful when the time between changes is short or when the user places importance on not generating error pixels due to saturation.

  Further, as another aspect of the boundary line passing through the blackout point bop, θbop and the maximum incident angle θbop2 of the lens of the imaging unit 110 are obtained as shown in FIG. 13A, and the blackout point bop (( A line connecting the point c) and the first distortion correction coefficient L1 at θbop2 (point (f) in FIG. 13) can be a boundary line_bop (broken line in FIG. 13) passing through the blackout. Then, the image processing unit 101 calculates the w2c value at the bop, the w2c value at the third distortion correction coefficient L3 (points (e1) to (e4) in FIG. 13) intersecting the line_bop as the second blackout point, the third black The w2c value at the first distortion correction coefficient L1 (point (f) in FIG. 12) at θbop2 is acquired as an out point (step S106 in FIG. 4).

  In this case, as the third distortion correction coefficient L3 approaches the first distortion correction coefficient L1, it is not affected by the saturation of the second distortion correction coefficient L2 due to the interpolation ratio, so the incident angle for conversion processing to a predetermined value is set. It is gradually increasing.

  Such a boundary line may cause a sudden change in the image when the incident angle uses a partial value in the saturation region, such as in an optical system specification with an incident angle exceeding 90 degrees, or when the angle of viewpoint conversion is large. This is useful when it is not preferable.

In this way, by appropriately setting the boundary line passing through the blackout point, the threshold value for determining whether or not the pixel is to be converted to the predetermined value is set. Therefore, in the subsequent distortion correction processing, the camera coordinate system is set by each distortion correction coefficient. It is determined whether each pixel data in the world coordinate system associated with each pixel data is used or converted into a predetermined value, and a conversion process is applied to the corresponding pixel data.

[C-5] Creation of distortion correction processing LUT:
The image processing unit 101 determines the first distortion correction coefficient L1, the second distortion correction coefficient L2, and the third distortion correction coefficient independently of whether or not each pixel corresponds to a pixel that is converted to a predetermined value based on the threshold value described above. For L3, an LUT used for distortion correction processing when converting the coordinates of the virtual projection plane VP from the world coordinate system to the camera coordinate system is created from the w2c value corresponding to each correction factor and the incident angle θ (FIG. 4). Middle step S107). The generated LUT is stored in the storage unit 103.

  A different LUT may be provided for each distortion correction rate, or data corresponding to a plurality of distortion correction rates may be stored in one LUT.

  Here, FIG. 14 is a schematic diagram for explaining the world coordinate system. As shown in FIG. 14, point A (0, 0, Za), point B (0, 479, Zb), point C (639, 479, Zc), point D of the virtual projection plane VP in the world coordinate system. A plane surrounded by (639, 0, Zd) is divided into 640 × 480 pixel pixels Gv (total number of pixels: 307,000) at equal intervals, and coordinates of all the pixels Gv in the world coordinate system are obtained. Note that the values of the X and Y coordinates in FIG. 14 are examples, and the X and Y coordinates of the point A are displayed as zero for easy understanding.

  From the coordinates of the pixel Gv in the world coordinate system and the distortion correction coefficient acquired in steps S104 to S105, the coordinate Gi (x ′, y ′) in the corresponding camera coordinate system on the image sensor surface IA is calculated to calculate the LUT. Create Specifically, it is calculated from the incident angle θ with respect to the optical axis Z obtained from the distortion correction coefficient and the coordinates of each pixel Gv (reference document: International Publication No. 2010/032720).

  FIG. 15 is an explanatory diagram illustrating a correspondence relationship between the camera coordinate system xy and the imaging element surface IA. In FIG. 15, points a to d are coordinate positions when the points A to D in FIG. 14 are converted into the camera coordinate system using the first distortion correction coefficient L1.

  Note that the pixel of the image sensor to be referred to is determined from the coordinates Gi (x ′, y ′) shown in FIG. 15. At this time, x and y in the coordinates (x, y) of each pixel of the image sensor are integers. However, x ′ and y ′ of the coordinates Gi (x ′, y ′) used in the LUT generation process in step S107 in FIG. 4 are not limited to integers and can take real values having a decimal part. When x ′ and y ′ are integers and the coordinates Gi (x ′, y ′) and the position of the pixel of the image sensor coincide with each other, the pixel data of the pixel of the corresponding image sensor is used as a pixel on the virtual projection plane VP. It can be used as pixel data corresponding to Gv (X, Y, Z). On the other hand, when x ′ and y ′ are not integers and x ′ and y ′ do not match x and y, the calculated coordinates Gi are used as pixel data of the output image corresponding to the pixel Gv as four-point interpolation. Using the pixel data of the pixels around (x ′, y ′) and the top four pixels close to the position of the coordinates Gi (x ′, y ′), these simple average values or coordinates Gi (x ′, y Pixel data calculated by weighting four pixels closer to the distance to y ′) are used. In addition, it is not restricted to 4 point | piece interpolation, As a surrounding location, the simple interpolation of 1 location and the multipoint interpolation using 16 locations or more may be sufficient.

  In FIG. 14, the virtual projection plane VP surrounded by the points A to D is a rectangular plane. In FIG. 15, the area surrounded by the points a to d after the coordinate conversion to the camera coordinate system is (the virtual projection plane VP The shape is distorted (corresponding to the position). Although FIG. 15 shows an example in which the barrel shape is distorted, the distortion is a pincushion type or a Jinkasa type (a shape that changes into a barrel type at the center and straight or pincushion at the end) due to the characteristics of the optical system. In some cases.

[C-6] Distortion correction processing:
The image processing unit 101 generates an image subjected to distortion correction processing using the LUT while changing the distortion correction rate according to time as shown in FIG. 9 (step S108 in FIG. 4).

  FIG. 16 is a schematic diagram for explaining the distortion correction processing. FIG. 16 is an enlarged view of a part of the image data, and the lattices shown in the figure indicate pixel divisions.

  The input image data in FIG. 16A is an enlarged display of a part of FIG. 15. As described with reference to FIGS. 14 to 15, the subject on the straight line is affected by the aberration on the image sensor surface IA. It is input on the curve (FIG. 16A, thick frames Gi1 to Gi3). The output image data is generated by using each pixel data of the output image data using the LUT and corresponding pixel data of the input image data.

  Since the LUT is generated in consideration of the distortion correction coefficient corresponding to each distortion correction factor, the LUT data generated based on the first distortion correction factor L1 of the distortion correction factor = 100% is finally used. In such a case, the data of the pixel Gv1 (for example, the luminance signal) refers to the data of Gi1 at the coordinates (100, 200) in FIG. Similarly, Gv2 and Gv3 refer to the data of Gi2 and Gi3, respectively.

  At this time, as described above, the pixel data converted by the LUT is invalid for the pixel determined to be the pixel corresponding to the incident angle equal to or larger than the threshold set by the boundary line passing through the blackout point, that is, by the LUT. Regardless of the value to be converted, it is converted to a predetermined value, for example, a value that is painted black.

[C-7] Image output:
The image processing unit 101 outputs an image generated by distortion correction processing using the LUT while changing the distortion correction rate according to time as shown in FIG. 9 to the display unit 120 and displays the image (step S109 in FIG. 4). ). For example, the output image generated by the distortion correction process is output to the display unit 120 at, for example, a frame rate of 30 fps or 60 fps, and the entire process ends. This image output is the third step.

  As a result, the distortion is corrected in accordance with the distortion correction rate. When the distortion correction rate is 100%, as shown in FIG. The subject is generated (displayed) as the original straight line.

  Although not shown in FIG. 16, if the distortion correction rate = 0%, an image having a shape with the distortion shown in FIG. 16A is output as it is, and the distortion correction rate is greater than 0% and 100%. If it is smaller, an image having a shape with an intermediate distortion between FIGS. 16A and 16B is output.

  At this time, if the viewpoint conversion is set by setting the virtual projection plane VP in step S101, such processing is also performed by the image processing unit 101.

[D] Display example by distortion correction processing:
Hereinafter, a display example when the distortion correction processing of the present embodiment is applied will be described.

[D-1] Viewpoint conversion by changing the position of the virtual projection plane VP:
FIG. 17 shows an example in which the position of the virtual projection plane VP0 is changed with the image center o in the camera coordinate system as the rotation center (or movement center). As shown in FIG. 17, the rotation about the x-axis with the image center o as the rotation center is an actual pitch (also referred to as tilt), the rotation about the y-axis is the actual yaw (also referred to as pan), and the Z-axis. The rotation around is real roll.

  FIG. 18 is an example in which the position of the virtual projection plane VP is changed with the center ov of the virtual projection plane VP0 as the center of rotation based on the input rotation amount setting value. One of the two axes that are orthogonal to each other on the virtual projection plane VP0 is set as Yaw-axis, and the other is set as P-axis. Both are axes passing through the center ov, and rotation around the Yaw-axis with the center ov as the rotation center is called virtual yaw rotation, and rotation around the P-axis is called virtual pitch rotation. In the following, viewpoint conversion corresponding to rotating or changing the position of the virtual camera Ca0 is performed.

  In FIG. 9B, the second position V2 is set to the position in the initial state. The virtual projection plane VP at this time is parallel to the XY plane and the center ov is on the optical axis Z (corresponding to the position of VP0 in FIGS. 17 and 18). The first position V1 is an example in which the virtual pitch and the real pitch are both 45 degrees, and both are set to 90 degrees. In this case, the viewpoint is changed so as to look down at the subject. Note that the combination of position changes is not limited to this, and any one of parallel movement, real picth, real yaw, real roll, virtual pitch, virtual yaw, or a combination thereof may be used.

[D-2] Image processing example of viewpoint conversion by changing the position of the virtual projection plane VP:
19 (a) to 19 (f) are accompanied by the start point conversion looking down at 90 ° as in the above-described D-1, and by performing the filling at the blackout point bop in each of the above distortion correction coefficients, This is an example of a display image that has been subjected to distortion correction processing by the image processing unit 101 and displayed on the display unit 120 so as not to be displayed. In the example of FIG. 19, the position of the virtual projection plane VP is also changed corresponding to the change of the distortion correction rate as shown in FIG. 9B.

  FIG. 19A shows an example in which image data calculated under the conditions of a distortion correction rate of 0% (second distortion correction coefficient L2) and a position movement rate of 0% (second position) are displayed as initial values.

  FIG. 19B to FIG. 19E show examples in which image data calculated under conditions in which both the distortion correction rate and the position movement rate are changed to 20%, 40%, 60%, and 80% are displayed. It is. Note that the b-filled area described above is indicated by hatching. Actually, it is painted with a constant value such as black, so that the image defect is covered and inconspicuous.

  Although it is not clear in this drawing, by performing conversion processing to a predetermined value as in this aspect, “smudge” is effectively eliminated as compared with the case where a saturated region is simply displayed as it is.

  Here, a specific example is given of a case where the boundary line passing through the blackout point bop is gradually changed according to the correction rate as shown in FIG.

  In FIG. 19F, the distortion of the image is eliminated by setting the distortion correction rate to 100%, and the virtual pitch and the actual pitch are both changed by −45 degrees due to the movement of the position, so that the subject is looked down. The viewpoint has been changed.

  With the above processing, the display unit 120 displays the initial image shown in FIG. 19A and the final image shown in FIG. 19F while sequentially updating the display image at a constant speed. In the specific example of FIG. 9B, images having a low correction rate and position movement rate are displayed in order from a low image to a high image, and this is repeated at a cycle of 10 seconds.

  Further, FIG. 20 shows a display example when the boundary line passing through the blackout point bop is determined vertically as shown in FIG. In this case, the image defects in the gray area and black area described in FIG. 11 are also filled with a constant value (indicated by hatching in FIG. 20), and thus the [C-4-3-1] image Pixel output of defect (1), [C-4-3-3] image defect (2), [C-4-3-3] image defect (3) can be effectively reduced.

  According to this embodiment, the virtual projection plane whose position is set is converted into a camera coordinate system using a correction coefficient, and the image data of the virtual projection plane is converted based on the coordinates of the converted camera coordinate system and the pixel data of the pixel. This is a case where the position of the virtual projection plane (camera viewpoint change) is changed in parallel with the distortion correction rate change by calculating the image data while changing the correction coefficient and the position data. However, it is not necessary to provide a separate process. It is possible to change the position and change the distortion correction rate at the same time without requiring new resource enhancement.

  In addition, according to the present embodiment, it is easy to accurately recognize an output image without distortion correction by changing the distortion correction rate stepwise from an output image without distortion correction to an output image with distortion correction. At this time, it is possible to accurately recognize the subject by covering up the image defect that occurs due to the saturation of the distortion correction coefficient.

  That is, erroneous recognition of a subject due to image defects caused by image processing limitations caused by entering a wide-angle lens at a very large angle can be effectively reduced.

DESCRIPTION OF SYMBOLS 100 Control apparatus 101 Image processing part 102 Setting part 103 Storage part 110 Imaging unit 120 Display part 130 Operation part VP Virtual projection surface LC Lens center plane IA Image sensor surface O Lens center o Image center L1 1st distortion correction coefficient L2 2nd distortion Correction coefficient L3 Third distortion correction coefficient

Claims (15)

An image processing method for obtaining image data processed using a plurality of pixel data obtained by receiving an image sensor having a plurality of pixels via an optical system,
A first step of setting a position and size of a virtual projection plane in the world coordinate system;
The coordinates in the world coordinate system of each pixel of the virtual projection plane set in the first step are converted into a camera coordinate system using a distortion correction coefficient, and based on the converted coordinates in the camera coordinate system and the plurality of pixel data A second step of calculating image data of the virtual projection plane set in the first step;
A third step of outputting a display image based on the image data calculated in the second step,
The distortion correction coefficient used in the second step is calculated based on at least the physical characteristics of the lens of the optical system and the incident angle of the incident light from the virtual projection plane set to the optical system. From the optical center at which the first distortion correction coefficient for correcting distortion caused by the optical system and the optical element of the optical system intersect with the image sensor calculated using the tangent function of the incident angle to the optical system as a variable And a second distortion correction coefficient that does not correct the distortion,
In the second step, when there is a minimum incident angle at which the second distortion correction coefficient is equal to or greater than a predetermined value, pixel data corresponding to an incident angle larger than the minimum incident angle is converted into a predetermined value. An image processing method comprising the step of: The three steps output a moving image for display of image data in which distortion is corrected in stages,
In the second step, the first distortion correction coefficient, the second distortion correction coefficient, and the third distortion correction coefficient obtained by interpolation from the first and second distortion correction coefficients are switched in stages and used for virtual projection. Calculate the image data of the surface,
The image processing method according to claim 1. In the second step, when there is a minimum incident angle at which the value of the second distortion correction coefficient is equal to or greater than a predetermined value, the pixel data corresponding to an incident angle larger than the minimum incident angle is set to a predetermined value Changing the incident angle when converting to the predetermined value according to the interpolation ratio of the second distortion correction coefficient included in the third distortion correction coefficient,
The image processing method according to claim 2. In the first step, the first position and the second position of the virtual projection plane of the world coordinate system are set,
In the second step, the initial second position and the second distortion correction coefficient, the final first position and the first distortion correction coefficient, and interpolation between the second position and the first position in the middle are obtained. Calculating the image data of the virtual projection plane using the third position and the third distortion correction coefficient by switching in stages.
The image processing method according to any one of claims 1 to 3. The second distortion correction coefficient is expressed by the following conditional expression (1).
The image processing method according to claim 1, wherein the image processing method is an image processing method.
L2 = ((ExportImageSize / 2) / (tan (focal / 2))) × tanθ Equation (1)
here,
ExportImageSize: the length of the long side of the output image to be displayed on the display,
focal: Angle of view in the long side direction of the set virtual projection plane,
It is. An image processing apparatus that obtains image data processed using a plurality of pixel data obtained by receiving an image sensor having a plurality of pixels via an optical system,
A storage unit for storing distortion correction coefficients;
The coordinates in the world coordinate system of each pixel of the virtual projection plane whose position and size are set are converted into the camera coordinate system using the distortion correction coefficient stored in the storage unit, and the coordinates converted into the camera coordinate system and the plurality of the coordinates An image processing unit that calculates image data of the virtual projection plane based on the pixel data;
An image signal output unit that outputs an image signal for display of the image data calculated by the image processing unit,
In the storage unit, the stored distortion correction coefficient is calculated based on at least the physical characteristics of the lens of the optical system and the incident angle of the incident light from the virtual projection plane set on the optical system. A first distortion correction coefficient for correcting distortion caused by the optical system and a tangent function of an incident angle to the optical system are used as variables, and the optical element from the optical center where the optical axis of the optical system intersects. A second distortion correction coefficient which is an image height and does not correct the distortion;
The image processing unit sets pixel data corresponding to an incident angle larger than the minimum incident angle to a predetermined value when there is a minimum incident angle at which the second distortion correction coefficient is equal to or greater than a predetermined value. An image processing apparatus for converting. The image signal output unit outputs a moving image for display of image data in which distortion is corrected stepwise,
The image processing unit virtually switches the first distortion correction coefficient, the second distortion correction coefficient, and the third distortion correction coefficient obtained by interpolation from the first and second distortion correction coefficients in a stepwise manner. Calculating image data of the projection plane,
The image processing apparatus according to claim 6. The image processing unit determines a minimum incident angle at which the second distortion correction coefficient is equal to or greater than a predetermined value, and has a minimum incident angle at which the second distortion correction coefficient is equal to or greater than a predetermined value. When the pixel data corresponding to the incident angle larger than the angle is converted into a predetermined value, the conversion is performed according to the interpolation ratio of the second distortion correction coefficient included in the third distortion correction coefficient. Changing the incident angle of
The image processing apparatus according to claim 7. The virtual projection plane of the world coordinate system has a first position and a second position,
The image processing unit is obtained by interpolation between the initial second position and the second distortion correction coefficient, the final first position and the first distortion correction coefficient, and the intermediate second position and first position. Calculating the image data of the virtual projection plane using the third position and the third distortion correction coefficient by switching in stages.
The image processing apparatus according to claim 6, wherein the image processing apparatus is an image processing apparatus. The second distortion correction coefficient is expressed by the following conditional expression (1).
The image processing apparatus according to claim 6, wherein the image processing apparatus is an image processing apparatus.
L2 = ((ExportImageSize / 2) / (tan (focal / 2))) × tanθ Equation (1)
here,
ExportImageSize: the length of the long side of the output image to be displayed on the display,
focal: Angle of view in the long side direction of the set virtual projection plane,
It is. Optical system,
An imaging device having a plurality of pixels;
A storage unit for storing distortion correction coefficients of the optical system;
A coordinate in the world coordinate system of each pixel of the virtual projection plane is converted into a camera coordinate system using the distortion correction coefficient of the optical system, and the coordinates converted into the camera coordinate system and a plurality of light received by the imaging element are received. An image processing unit that calculates image data of the virtual projection plane based on pixel data;
An image signal output unit that outputs an image signal for display of the image data calculated by the image processing unit,
In the storage unit, the stored distortion correction coefficient is calculated based on at least the physical characteristics of the lens of the optical system and the incident angle of the incident light from the virtual projection plane set on the optical system. A first distortion correction coefficient for correcting distortion caused by the optical system and a tangent function of an incident angle to the optical system are used as variables, and the optical element from the optical center where the optical axis of the optical system intersects. A second distortion correction coefficient which is an image height and does not correct the distortion;
The image processing unit sets pixel data corresponding to an incident angle larger than the minimum incident angle to a predetermined value when there is a minimum incident angle at which the second distortion correction coefficient is equal to or greater than a predetermined value. An imaging apparatus characterized by converting. The image signal output unit outputs a moving image for display of image data in which distortion is corrected stepwise,
The image processing unit virtually switches the first distortion correction coefficient, the second distortion correction coefficient, and the third distortion correction coefficient obtained by interpolation from the first and second distortion correction coefficients in a stepwise manner. Calculating image data of the projection plane,
The imaging apparatus according to claim 11. The previous image processing unit determines a minimum incident angle at which the second distortion correction coefficient is equal to or greater than a predetermined value, and there is a minimum incident angle at which the second distortion correction coefficient is equal to or greater than a predetermined value. When converting pixel data corresponding to an incident angle larger than an angle into a predetermined value, the conversion is performed according to the interpolation ratio of the second distortion correction coefficient included in the third distortion correction coefficient. Changing the incident angle,
The imaging apparatus according to claim 12. The virtual projection plane of the world coordinate system has a first position and a second position,
The image processing unit is obtained by interpolation between the initial second position and the second distortion correction coefficient, the final first position and the first distortion correction coefficient, and the intermediate second position and first position. Calculating the image data of the virtual projection plane using the third position and the third distortion correction coefficient by switching in stages.
The image pickup apparatus according to claim 11, wherein the image pickup apparatus is an image pickup apparatus. The second distortion correction coefficient is expressed by the following conditional expression (1).
The imaging device according to any one of claims 11 to 14, wherein
L2 = ((ExportImageSize / 2) / (tan (focal / 2))) × tanθ Equation (1)
here,
ExportImageSize: the length of the long side of the output image to be displayed on the display,
focal: Angle of view in the long side direction of the set virtual projection plane,
It is.