JP5067336B2 - Image conversion apparatus and image conversion method - Google Patents

Description

  The present invention relates to an image conversion apparatus and an image conversion method, and more particularly, to a technique for performing processing for cutting out a part of a distorted circular image obtained by photographing using a fisheye lens and converting the image into a planar regular image.

  When a fisheye lens is used, a circular image showing all directions of a hemisphere can be obtained without a mechanical operation mechanism. For this reason, it is widely used when taking a landscape photograph aiming at an unusual effect. However, since an image obtained by photographing with a fisheye lens is a distorted circular image, it may be usable as it is for art photography or the like, but is not suitable for general use.

In view of this, there has been proposed an apparatus that performs processing for cutting out a part of a distorted circular image obtained by photographing using a fisheye lens and converting it into a planar regular image. For example, Patent Document 1 below discloses a technique for converting a part of a distorted circular image into a planar regular image in real time using a computer. By using such a conversion technique, it becomes possible to observe a moving image composed of a distorted circular image taken using a fisheye lens in real time as a moving image composed of a planar regular image, and has a 180 ° field of view. Application to systems etc. can be expected.
Japanese Patent No. 3051173

  In order to extract a part of a distorted circular image obtained by photographing using a fisheye lens and convert it into a planar regular image, which part of the distorted circular image is the part to be converted (the position of the extraction) It is necessary to set parameters such as the plane regular image in which the portion is displayed (direction of clipping) and the extent to which the portion is to be converted (conversion magnification). Of course, these parameters can be arbitrarily set according to the user's wishes.

  However, in the conventional image conversion devices disclosed in the above-mentioned patent documents and the like, it is necessary to specify the clipping position and orientation by three angles called Euler angles, and to specify the conversion magnification separately. There is a problem of poor operability from the standpoint.

  The Euler angle defines a dome-like virtual spherical surface on a distorted circular image, and when a planar regular image forming surface that touches a desired contact point on the virtual spherical surface is defined, the position of the contact point is represented by an azimuth angle α and a zenith angle. This is represented by β, and the orientation of the planar regular image is represented by a plane inclination angle ψ. While these three angles α, β, and ψ are angles defined in the three-dimensional space, the distorted circular image that the user actually sees is an image displayed on the two-dimensional plane. It is. Therefore, for the user, the operation of specifying the three angles α, β, and ψ as parameters cannot be said to be an intuitive operation.

  Another problem with the conventional image conversion apparatus is the computational burden. When the above-described Euler angles α, β, and ψ are specified as parameters, it is naturally necessary to perform an operation using a conversion equation that includes these Euler angles α, β, and ψ as parameters. However, such a conversion arithmetic expression becomes a complicated expression including a trigonometric function with respect to three angles α, β, and ψ, which imposes a considerably large calculation burden on the system. In particular, when used in applications that require real-time conversion processing, high-performance hardware capable of performing trigonometric function operations at high speed is required, which is an important problem that hinders reduction in product cost.

  Therefore, the present invention makes it possible to set necessary parameters by a user's intuitive operation when converting a distorted circular image obtained by photographing using a fisheye lens into a planar regular image, and to reduce the calculation burden. An object of the present invention is to provide an image conversion apparatus and an image conversion method capable of performing the above.

(1) A first aspect of the present invention is an image conversion apparatus that performs a process of cutting out a part of a distorted circular image obtained by photographing using a fisheye lens and converting the image into a planar regular image.
A distorted circular image having a radius R centered on the origin O of the XY coordinate system, which is composed of a collection of a large number of pixels arranged at positions indicated by coordinates (x, y) on a two-dimensional XY orthogonal coordinate system. A distorted circular image storage unit for storing;
A planar regular image storage unit for storing a planar regular image configured by an aggregate of a large number of pixels arranged at a position indicated by coordinates (u, v) on a two-dimensional UV orthogonal coordinate system;
A distorted circular image display unit for displaying the distorted circular image stored in the distorted circular image storage unit on a display;
An instruction input unit for inputting the position and cutting direction of the cut-out center point P on the distorted circular image displayed on the display based on a user's instruction;
In a three-dimensional XYZ Cartesian coordinate system including a two-dimensional XY Cartesian coordinate system, when a virtual sphere having a radius R with the origin O as the center is defined, a straight line and a virtual sphere passing through the cut center point P and parallel to the Z axis An intersection calculation unit for obtaining the position coordinates (x ₀ , y ₀ , z ₀ ) of the intersection G with
A vector U oriented in the U-axis direction of the two-dimensional UV orthogonal coordinate system to be defined on a tangent plane that is in contact with the phantom sphere at the intersection G, and a vector X oriented in the X-axis direction of the two-dimensional XY orthogonal coordinate system An angle determination unit that determines the plane inclination angle φ given as an angle based on the cutout direction;
The coordinates (u, v) and the coordinates (x, y) are associated with each other by using an orthographic projection conversion equation including the position coordinates (x ₀ , y ₀ , z ₀ ) and the plane inclination angle φ as parameters. , The pixel value of the pixel on the planar regular image placed at the position indicated by the coordinates (u, v), and the reference pixel on the distorted circular image placed at the position indicated by the corresponding coordinates (x, y). By determining based on the pixel value, an operation is performed to generate a plane regular image for a partial image cut out in a direction indicated by the plane inclination angle φ around the cut center point P from the distorted circular image. A conversion operation unit that stores the planar regular image in the planar regular image storage unit;
Is provided.

(2) According to a second aspect of the present invention, in the image conversion device according to the first aspect described above,
The conversion calculation unit is an orthographic conversion calculation expression.
x = R [(u−x ₀ ) A + (v−y ₀ ) B + (w−z ₀ ) E] /
√ ((u−x ₀ ) ² + (v−y ₀ ) ² + (w−z ₀ ) ² )
y = R [(u−x ₀ ) C + (v−y ₀ ) D + (w−z ₀ ) F] /
√ ((u−x ₀ ) ² + (v−y ₀ ) ² + (w−z ₀ ) ² )
here,
A = 1- (1-cosφ) (y ₀ ² + z ₀ ² )
B = −z ₀ sinφ + x ₀ y ₀ (1-cosφ)
C = z ₀ sinφ + x ₀ y ₀ (1-cosφ)
D = 1- (1-cosφ) (z ₀ ² + x ₀ ² )
E = y ₀ sinφ + z ₀ x ₀ (1-cosφ)
F = −x ₀ sinφ + y ₀ x ₀ (1-cosφ)
w = mR (where m is a predetermined conversion magnification)
This formula is used.

(3) According to a third aspect of the present invention, in the image conversion device according to the first or second aspect described above,
The instruction input unit has a function of inputting an instruction for defining the reference straight line J drawn on the distorted circular image,
The angle determination unit determines the plane inclination angle φ based on the angle θ formed by the reference straight line J and the X axis (however, when both are parallel, θ = 0 °).

(4) According to a fourth aspect of the present invention, in the image conversion device according to the third aspect described above,
The instruction input unit has a function of inputting an instruction for designating the positions of two points of the cut center point P and the auxiliary point Q on the distorted circular image. Is a reference straight line J.

(5) According to a fifth aspect of the present invention, in the image conversion device according to the third aspect described above,
An instruction input unit inputs a numerical value indicating an angle θ formed by the reference straight line J and the X axis on a predetermined input screen, and an instruction for specifying the position of the cutting center point P on the distorted circular image. And an input function.

(6) According to a sixth aspect of the present invention, in the image conversion device according to the third to fifth aspects described above,
The angle determination unit approximates the angle θ formed by the reference straight line J and the X axis to the plane inclination angle φ.

(7) According to a seventh aspect of the present invention, in the image conversion device according to the second aspect described above,
The instruction input unit has a function of inputting the conversion magnification m based on a user instruction,
The conversion operation unit performs an operation using the conversion magnification m input by the instruction input unit.

(8) According to an eighth aspect of the present invention, in the image conversion device according to the seventh aspect described above,
The instruction input unit has a function of inputting an instruction for designating the positions of two points of the cut center point P and the auxiliary point Q on the distorted circular image. Based on the distance d, a numerical value given as m = k / d (k is a predetermined proportional constant) is set as the conversion magnification m.

(9) According to a ninth aspect of the present invention, in the image conversion device according to the first to eighth aspects described above,
When the transformation calculation unit determines the pixel value of the pixel on the planar regular image arranged at the position indicated by the coordinates (u, v), it is arranged near the position indicated by the corresponding coordinates (x, y). The interpolation calculation is performed on the pixel values of a plurality of reference pixels on the distorted circular image.

(10) According to a tenth aspect of the present invention, in the image conversion device according to the first to ninth aspects described above,
If the image stored in the distorted circular image storage unit is not an orthographic image taken with a fisheye lens of orthographic projection but a non-orthographic image taken with a fisheye lens of non-projecting scheme, Using a first coordinate conversion formula for converting coordinates on the image into coordinates on an orthographic image and a second coordinate conversion formula for converting coordinates on the orthographic image into coordinates on a non-orthographic image. And
The intersection calculation unit converts the coordinates of the cut-out center point P using the first coordinate conversion formula, and obtains the position coordinates (x ₀ , y ₀ , z ₀ ) of the intersection using the converted coordinates. Done
The conversion calculation unit obtains the coordinates (x, y) corresponding to the coordinates (u, v) using the orthographic projection calculation expression, and then uses the second coordinate conversion expression for the coordinates (x, y). The position of the reference pixel on the distorted circular image is specified using the converted coordinates.

(11) According to an eleventh aspect of the present invention, in the image conversion device according to the tenth aspect described above,
When the image stored in the distorted circular image storage unit is an equidistant projection image taken by an equidistant projection fisheye lens,
As the first coordinate conversion formula, the intersection calculation unit converts the coordinates (x ′, y ′) on the equidistant projection image into the coordinates (x, y) on the orthographic projection image.
x = sinc (π / 2 · √ (x ′ ² + y ′ ² )) × π / 2 · x ′
y = sinc (π / 2 · √ (x ′ ² + y ′ ² )) × π / 2 · y ′
Using the formula
As a second coordinate conversion formula for converting the coordinate (x, y) on the orthographic projection image into the coordinate (x ′, y ′) on the equidistant projection image,
x ′ = 2 / π · x / sinc (π / 2 · √ (x ² + y ² ))
y ′ = 2 / π · y / sinc (π / 2 · √ (x ² + y ² ))
This formula is used.

  (12) According to a twelfth aspect of the present invention, the image conversion apparatus according to the first to eleventh aspects is configured by incorporating a program into a computer.

  (13) According to a thirteenth aspect of the present invention, an electronic circuit that functions as a conversion arithmetic unit that is a component of the image conversion apparatus according to the first to eleventh aspects described above is incorporated in a semiconductor integrated circuit. It is.

  (14) A fourteenth aspect of the present invention is a combination of the image conversion apparatus according to the first to eleventh aspects described above, a camera using a fisheye lens, and a monitor device that displays a planar regular image on a screen. The distortion circular image obtained by photographing using the camera is stored in the distortion circular image storage unit, and the planar regular image obtained in the planar regular image storage unit is configured to be displayed by the monitor device. The fish-eye monitoring system is realized.

(15) According to a fifteenth aspect of the present invention, in the image conversion device according to the second aspect described above,
The conversion operation part
A first function table in which the value of “function f (c) = 1 / c” is associated with the value of various variables c, and the value of “function f (ξ) = 1 / √ξ” A second function table associated with the values of
a = u−x ₀ , b = v−y ₀ , c = w−z ₀ ,
ξ = (a / c) ² + (b / c) ² +1
To obtain the values of c and ξ,
By referring to the first function table and the second function table, the values of the functions f (c) and f (ξ) corresponding to the obtained values of c and ξ are obtained,
In the transformation formula of orthographic projection
√ ((u−x ₀ ) ² + (v−y ₀ ) ² + (w−z ₀ ) ² )
Is obtained by a calculation of f (c) × f (ξ).

(16) According to a sixteenth aspect of the present invention, in the image conversion device according to the first to eleventh aspects described above,
The conversion operation part
An even function table in which a value of a predetermined function f (t) is associated with an even variable t among variables t taking discrete values of the interval W according to a predetermined number of significant digits;
An odd function table in which a value of a predetermined function f (t) is associated with an odd variable t among the variables t taking discrete values of the interval W according to the number of significant digits;
A T register for storing a variable t composed of an upper bit consisting of the number of significant digits and a lower bit indicating a digit lower than the number of significant digits;
When the upper bits are even, the value of the function f (t) associated with the even variable t indicated by the upper bits is read from the even function table, and when the upper bits are odd, the upper bits An even number reading unit for reading out the value of the function f (t) associated with the smallest even number variable t larger than the odd number variable t shown in FIG.
When the upper bit is odd, the value of the function f (t) associated with the odd variable t indicated by the upper bit is read from the odd function table. When the upper bit is even, the upper bit An odd reading unit that reads the value of the function f (t) associated with the smallest odd variable t larger than the even variable t shown in FIG.
An A register for storing the value of the function f (t) read from the even function table or the odd function table;
A B register for storing the value of the function f (t) read from the even function table or the odd function table;
When the upper bits are even, the value of the function f (t) read by the even reading unit is stored in the A register, and the value of the function f (t) read by the odd reading unit is stored in the B register. When the upper bits are odd, the value of the function f (t) read by the odd reading unit is stored in the A register, and the value of the function f (t) read by the even reading unit is stored in the B register. An even-odd selector to
The value of the function f (t) after interpolation, where f (A) is the value stored in the A register, f (B) is the value stored in the B register, and δ is the value indicated by the lower bits. An interpolation calculation unit for calculating f (t) = ((W−δ) / W) × f (A) + (δ / W) × f (B),
And the calculation is performed using the value of the interpolated function f (t) obtained by the interpolation calculation unit.

(17) According to a seventeenth aspect of the present invention, in the image conversion device according to the eleventh aspect described above,
The intersection calculation unit calculates the function sinc (t) in the first coordinate conversion formula,
^{sinc (t) = 1-t} 2/3! + ^T 4/5! -T ^6/7! + ^T 8/9! − ....
Calculate based on the expression of the Taylor expansion form
The transformation calculation unit uses the function 1 / sinc (t) in the second coordinate transformation formula as 1 / sinc (t) using predetermined coefficient values a ₂ , a ₄ , a ₆ , a ₈ ,. = 1 + a ₂ t ² + a ₄ t ⁴ + a ₆ t ⁶ + a ₈ t ⁸ +.
The calculation is based on an expression of the form

(18) According to an eighteenth aspect of the present invention, in the image conversion device according to the second aspect described above,
The conversion operation unit obtains the value of cosφ by dividing the inner product of the vector U and the vector X by the product of the magnitude of the vector U and the magnitude of the vector X.

(19) According to a nineteenth aspect of the present invention, in the image conversion device according to the second aspect described above,
The instruction input unit has a function of inputting an instruction for defining the reference straight line J drawn on the distorted circular image,
The transformation calculation unit defines a vector J facing the direction of the reference straight line J, and divides the value of cosφ by the inner product of the vector J and the vector X by the product of the magnitude of the vector J and the magnitude of the vector X. It is what you ask for.

(20) According to a twentieth aspect of the present invention, in the image conversion device according to the second aspect described above,
The conversion operation unit obtains the value of sinφ by an operation of sinφ = √ (1-cos ² φ).

(21) According to a twenty-first aspect of the present invention, in an image conversion apparatus that performs processing of cutting out a part of a distorted circular image obtained by photographing using a fisheye lens and converting the image into a planar regular image,
A distorted circular image storage unit for storing a distorted circular image configured by an aggregate of a large number of pixels arranged at a position indicated by coordinates (x, y) on a two-dimensional XY orthogonal coordinate system;
A planar regular image storage unit for storing a planar regular image configured by an aggregate of a large number of pixels arranged at a position indicated by coordinates (u, v) on a two-dimensional UV orthogonal coordinate system;
A distorted circular image display unit for displaying the distorted circular image stored in the distorted circular image storage unit on a display;
A straight line connecting the cut-out center point P and the auxiliary point Q on the distorted circular image displayed on the display by inputting the positions of the cut-out center point P and the auxiliary point Q based on a user instruction. Is recognized as a reference straight line J, and based on the distance d between the cutting center point P and the auxiliary point Q, an instruction for recognizing the numerical value given as m = k / d (k is a predetermined proportional constant) as the conversion magnification m An input section;
The pixel value of the pixel on the planar regular image arranged at the position indicated by the coordinates (u, v) is calculated using a conversion operation expression that associates the coordinates (u, v) with the coordinates (x, y). By determining based on the pixel value of the reference pixel on the distorted circular image arranged at the position indicated by the corresponding coordinates (x, y), the reference straight line J is centered on the cut-out center point P from the distorted circular image. A conversion operation unit that performs an operation for generating a planar regular image scaled based on the conversion magnification m for the partial image cut out in a corresponding direction, and stores the generated planar regular image in the planar regular image storage unit;
Is provided.

(22) According to a twenty-second aspect of the present invention, in an image conversion apparatus that performs a process of cutting out a part of a distorted circular image obtained by photographing using a fisheye lens and converting the image into a planar regular image.
A distorted circular image storage unit for storing a distorted circular image configured by an aggregate of a large number of pixels arranged at a position indicated by coordinates (x, y) on a two-dimensional XY orthogonal coordinate system;
A planar regular image storage unit for storing a planar regular image configured by an aggregate of a large number of pixels arranged at a position indicated by coordinates (u, v) on a two-dimensional UV orthogonal coordinate system;
A distorted circular image display unit for displaying the distorted circular image stored in the distorted circular image storage unit on a display;
On a predetermined input screen, an angle θ defined as an angle formed by the reference straight line J and the X axis and a conversion magnification m are input based on a user instruction, and the distorted circle displayed on the display On the image, an instruction input unit for inputting the position of the cutting center point P based on a user instruction;
The pixel value of the pixel on the planar regular image arranged at the position indicated by the coordinates (u, v) is calculated using a conversion operation expression that associates the coordinates (u, v) with the coordinates (x, y). By determining based on the pixel value of the reference pixel on the distorted circular image arranged at the position indicated by the corresponding coordinates (x, y), the reference straight line J is centered on the cut-out center point P from the distorted circular image. A conversion operation unit that performs an operation for generating a planar regular image scaled based on the conversion magnification m for the partial image cut out in a corresponding direction, and stores the generated planar regular image in the planar regular image storage unit;
Is provided.

(23) According to a twenty-third aspect of the present invention, in an image conversion method for performing processing of cutting out a part of a distorted circular image obtained by photographing using a fisheye lens and converting the image into a planar regular image,
A distorted circular image having a radius R centered on the origin O of the XY coordinate system, which is composed of a collection of a large number of pixels arranged at positions indicated by coordinates (x, y) on a two-dimensional XY orthogonal coordinate system. Storing in the distorted circular image storage unit;
Displaying the distorted circular image stored in the distorted circular image storage unit on a display;
A step of inputting the position of the cut-out center point P and the cut-out direction on the distorted circular image displayed on the display based on a user instruction;
In a three-dimensional XYZ Cartesian coordinate system including a two-dimensional XY Cartesian coordinate system, when a virtual sphere having a radius R with the origin O as the center is defined, a straight line and a virtual sphere passing through the cut center point P and parallel to the Z axis Obtaining the position coordinates (x ₀ , y ₀ , z ₀ ) of the intersection G with
A vector U oriented in the U-axis direction of the two-dimensional UV orthogonal coordinate system to be defined on a tangent plane that is in contact with the phantom sphere at the intersection G, and a vector X oriented in the X-axis direction of the two-dimensional XY orthogonal coordinate system Determining a plane inclination angle φ given as an angle based on the cutout direction;
The coordinates (u, v) and the coordinates (x, y) are associated with each other by using an orthographic projection conversion equation including the position coordinates (x ₀ , y ₀ , z ₀ ) and the plane inclination angle φ as parameters. , The pixel value of each pixel on the planar regular image formed by the aggregate of a large number of pixels arranged at the position indicated by the coordinates (u, v) at the position indicated by the corresponding coordinates (x, y). By determining based on the pixel value of the reference pixel on the arranged distorted circular image, the plane of the partial image cut out from the distorted circular image in the direction indicated by the plane inclination angle φ with the cut center point P as the center Performing a computation to generate a regular image;
Is executed by a computer or an electronic circuit.

(24) According to a twenty-fourth aspect of the present invention, in the image conversion method according to the twenty-third aspect described above,
As a transformation formula of orthographic projection method,
x = R [(u−x ₀ ) A + (v−y ₀ ) B + (w−z ₀ ) E] /
√ ((u−x ₀ ) ² + (v−y ₀ ) ² + (w−z ₀ ) ² )
y = R [(u−x ₀ ) C + (v−y ₀ ) D + (w−z ₀ ) F] /
√ ((u−x ₀ ) ² + (v−y ₀ ) ² + (w−z ₀ ) ² )
here,
A = 1- (1-cosφ) (y ₀ ² + z ₀ ² )
B = −z ₀ sinφ + x ₀ y ₀ (1-cosφ)
C = z ₀ sinφ + x ₀ y ₀ (1-cosφ)
D = 1- (1-cosφ) (z ₀ ² + x ₀ ² )
E = y ₀ sinφ + z ₀ x ₀ (1-cosφ)
F = −x ₀ sinφ + y ₀ x ₀ (1-cosφ)
w = mR (where m is a predetermined conversion magnification)
This formula is used.

(25) According to a twenty-fifth aspect of the present invention, in the image conversion method according to the twenty-third or twenty-fourth aspect described above,
On the distorted circular image displayed on the display, an instruction for designating the positions of two points of the cut center point P and the auxiliary point Q is input, and a straight line connecting the cut center point P and the auxiliary point Q Is a reference straight line J, and the plane inclination angle φ is determined on the basis of an angle θ formed by the reference straight line J and the X axis (however, when both are parallel, θ = 0 °).

(26) According to a twenty-sixth aspect of the present invention, in the image conversion method according to the twenty-third or twenty-fourth aspect described above,
On the distorted circular image displayed on the display, an instruction for designating the position of the cut-out center point P is input, and on the predetermined input screen, the reference straight line J on the distorted circular image and the X axis The formed angle θ is input, and the plane inclination angle φ is determined based on the angle θ.

Since the user using the image conversion apparatus according to the present invention only has to indicate the position and the cutout direction of the cutout center point P on the distorted circular image obtained by shooting using the fisheye lens, Parameter settings necessary for conversion can be performed. Moreover, the conversion calculation is a relatively simple conversion calculation formula that uses the coordinate values (x ₀ , y ₀ , z ₀ ) of the contact point on the virtual sphere derived based on the user's instruction input and the plane inclination angle φ as parameters. Therefore, the calculation burden can be reduced.

  Hereinafter, the present invention will be described based on the illustrated embodiments.

<<< §1. Basic Principle of Image Conversion Processing >>>
First, general characteristics of a distorted circular image obtained by photographing using a fisheye lens and a basic principle of processing for cutting out a part of the image and converting it into a planar regular image will be described. FIG. 1 is a perspective view showing a basic model for forming a distorted circular image S by photographing using an orthographic fisheye lens. In general, fish-eye lenses are classified into a plurality of types depending on the projection method, but the model shown in FIG. 1 is for a fish-eye lens of an orthographic projection method (a method for applying the present invention to a fish-eye lens other than an orthographic projection method). , Described in §5).

  FIG. 1 shows an example in which a distorted circular image S is formed on an XY plane in a three-dimensional XYZ orthogonal coordinate system. Here, in order to ensure consistency with the conversion equation described later, as shown in the figure, the Z axis is taken downward in the figure, and a dome-shaped virtual spherical surface H (hemisphere) is provided on the negative region side of the Z axis. Let's show a defined example.

  The distorted circular image S formed on the XY plane is an image forming a circle with a radius R centered on the origin O of the coordinate system, and is an area having an angle of view of 180 ° on the negative area side of the Z axis. Is equivalent to the image recorded with distortion. FIG. 2 is a plan view showing an example of a distorted circular image S obtained by photographing using a fisheye lens. As described above, in the distorted circular image S, all the images existing on the negative region side of the Z-axis are recorded, but the scale magnification of the image is different between the central portion and the peripheral portion. The shape of the recorded image is distorted. Note that the distorted circular image S shown in FIG. 2 shows a general image of a distorted circular image obtained by photographing using a fisheye lens, and does not show an accurate image obtained using an actual fisheye lens. Absent.

  An actual fisheye lens is constituted by an optical system that combines a plurality of convex lenses and concave lenses, and it is known that its optical characteristics can be modeled by a virtual spherical surface H as shown in FIG. In other words, considering a model in which a dome-shaped virtual spherical surface H (hemisphere) having a radius R is arranged on the upper surface of the distorted circular image S, the optical characteristics of the orthographic fisheye lens are arbitrary on the virtual spherical surface H. The incident light beam L1 incident from the normal direction to the point H (x, y, z) behaves toward the point S (x, y) on the XY plane as an incident light beam L2 parallel to the Z axis. You may think. In other words, the pixel located at the point S (x, y) on the distorted circular image S in FIG. 2 indicates one point on the object existing on the extension line of the incident light beam L1 shown in FIG. become.

  Of course, an optical phenomenon occurring in an actual fisheye lens is that a specific point of an object to be imaged is connected to a specific point S (x, y) on the XY plane due to refraction by a plurality of convex lenses and concave lenses. Although this is an image phenomenon, there is no problem in performing an image conversion process or the like even if the argument is replaced with a model using a virtual spherical surface H as shown in FIG. Therefore, even in the image conversion processing disclosed in the above-mentioned patent document, a technique based on such a model is shown. In the following description of the present invention, description based on such a model is also given. To do.

An object of the image conversion apparatus according to the present invention is to perform a process of cutting out a part on the distorted circular image S and converting it into a planar regular image. For example, assume that a user who looks at the distorted circular image S shown in FIG. 2 wants to observe the image of a woman drawn on the lower left of the image with a correct image without distortion. In such a case, the user needs to specify which part of the distorted circular image S should be cut out and converted. For example, if the cutout area E as shown by hatching in FIG. 3 is designated as the area to be converted, the most intuitive designation method is the center point P (x ₀ , y ₀ ). It would be a way to specify the position. In the present invention, the point P designated by the user in this way is referred to as a cutting center point P.

Here, the following model is considered in order to convert an image in the cut-out area E centered at the cut-out center point P (x ₀ , y ₀ ) into a planar regular image. FIG. 4 is a perspective view showing the relationship between the XY coordinate system including the distorted circular image S and the UV coordinate system including the planar regular image T in this model. As illustrated, since the distorted circular image S is defined on the XY plane of the three-dimensional XYZ orthogonal coordinate system, the distorted circular image S itself is an image defined on the two-dimensional XY orthogonal coordinate system. Therefore, consider an intersection G of a virtual sphere H with a straight line passing through the cut center point P (x ₀ , y ₀ ) defined on the distorted circular image S and parallel to the Z axis. This intersection point G is, so to speak, a point immediately above the cut-out center point P (x ₀ , y ₀ ), and its position coordinates are (x ₀ , y ₀ , z ₀ ).

Next, a tangent plane that is in contact with the phantom spherical surface H is defined at the intersection point G (x ₀ , y ₀ , z ₀ ), and a two-dimensional UV orthogonal coordinate system is defined on the tangent plane. Then, the planar regular image T is obtained as an image on the two-dimensional UV orthogonal coordinate system. In the example shown in FIG. 4, the UV coordinate system is defined so that the intersection point G (x ₀ , y ₀ , z ₀ ) is the origin. Eventually, the origin of the UV coordinate system in this model is set anywhere on the virtual spherical surface H, and the UV plane constituting the UV coordinate system coincides with the tangential plane with respect to the virtual spherical surface H at this origin position.

The position of the intersection point G (x ₀ , y ₀ , z ₀ ) serving as the origin of the UV coordinate system can be specified by the azimuth angle α and the zenith angle β as shown in the figure. Here, the azimuth angle α (0 ≦ α <360 °) is an angle formed by a straight line connecting the cutting center point P (x ₀ , y ₀ ) and the origin O of the XY coordinate system and the X axis, and the zenith The angle β (0 ≦ β ≦ 90 °) is an angle between the Z axis and a straight line connecting the point G (x ₀ , y ₀ , z ₀ ) serving as the origin of the UV coordinate system and the origin O of the XY coordinate system. is there.

As described above, the UV plane can be specified by specifying the azimuth angle α and the zenith angle β. However, in order to determine the UV coordinate system, another parameter needs to be specified. This parameter indicates a rotation factor (direction of the U axis) on the tangent plane with the point G as the center, and is generally called a plane inclination angle. Normally, this plane inclination angle is defined as an angle ψ formed by the reference axis and the U axis, with “an axis passing through the point G, being parallel to the XY plane and orthogonal to the straight line OG” as a reference axis ( The reference axis varies depending on the azimuth angle α and the zenith angle β). On the other hand, in the present invention, instead of the angle ψ, an angle φ indicating the direction of the U axis with respect to the X axis is used (the reference axis is always fixed to the X axis). . Specifically, the angle φ used as a parameter in the present invention is defined as an angle formed by the U axis and the X ′ axis, as shown in FIG. Here, the X ′ axis is an axis that passes through the intersection point G (x ₀ , y ₀ , z ₀ ) and is parallel to the X axis. In short, the angle φ is an angle formed between the vector U and the vector X when the vector U facing the U-axis direction in the UV coordinate system and the vector X facing the X-axis direction in the XY coordinate system are defined. Becomes a defined corner. As described above, the “angle φ used in the present invention” is an angle defined by a method different from the “conventional general plane inclination angle ψ”, and both are parameters indicating the rotation factor of the UV coordinate system. Therefore, the term “plane inclination angle” is also used here for the angle φ.

FIG. 5 is a plan view showing the relationship between the planar regular image T defined on the UV coordinate system and the planar tilt angle φ. In the case of the example shown here, the planar regular image T is defined as a rectangle on the UV plane centered at the origin G (x ₀ , y ₀ , z ₀ ) of the UV coordinate system, and its long side is the U axis. The short side is parallel to the V-axis. Since the plane inclination angle φ is an angle formed by the U axis and the X ′ axis as described above, in the example shown in FIG. 5, the plane inclination angle φ is a parameter indicating the rotation factor of the plane regular image T on the UV plane. Become.

  As a result, the position and orientation of the UV coordinate system for forming the planar regular image T shown in FIG. 4 are uniquely set by setting parameters composed of three angles, the azimuth angle α, the zenith angle β, and the plane tilt angle φ. To be determined. These three angles are generally called Euler angles (as described above, the definition method of the plane inclination angle φ is slightly different from the conventional general plane inclination angle ψ).

  Now, the image conversion process executed in the present invention is the coordinate conversion from the XY coordinate system to the UV coordinate system. Let's take a closer look at the geometric positional relationship between the XY coordinate system and the UV coordinate system. As shown in the perspective view of FIG. 6, when the distorted circular image S on the XY plane is inclined by the zenith angle β with respect to the direction indicated by the azimuth angle α, an inclined surface S1 is obtained. Here, as shown in the figure, a normal vector n is defined in the direction from the origin O of the XY coordinate system to the origin G of the UV coordinate system, and the inclined plane S1 is translated in the direction of the normal vector n by a distance R. Then, a tangential plane S2 is obtained. The moving distance R is the radius of the distorted circular image S and the radius of the phantom spherical surface H.

The tangent plane S2 is a plane in contact with the virtual sphere H at the point G, and the normal vector n is a vector indicating the normal direction of the virtual sphere H at the point G. The UV coordinate system is a coordinate system defined on the tangent plane S2, and the angle between the U axis and the X ′ axis (the axis obtained by translating the X axis) with the point G as the origin is the plane tilt angle. A two-dimensional orthogonal coordinate system defined to be φ. FIG. 7 is a diagram of each component shown in the perspective view of FIG. 6 viewed from the horizontal direction. As described above, the point G (x ₀ , y ₀ , z ₀ ) passes through the cut center point P (x ₀ , y ₀ ) defined on the distorted circular image S and is a straight line and virtual sphere that are parallel to the Z axis. This is a point determined as an intersection with H, and its position is determined by the azimuth angle α and the zenith angle β. On the other hand, FIG. 8 is a view of each component shown in the perspective view of FIG. 6 as viewed from above. The intersection point G (x ₀ , y ₀ , z ₀ ) shown in the figure is a point on the phantom spherical surface H, and is located above the XY plane. Then, a UV coordinate system is defined on the tangent plane with respect to the virtual spherical surface H at the intersection point G (x ₀ , y ₀ , z ₀ ). At this time, the direction of the U axis is determined so that the angle formed by the U axis and the X ′ axis is φ.

FIG. 9 is a plan view showing a planar regular image T defined on the UV coordinate system. Here, an arbitrary point on the planar regular image T is expressed as T (u, v) using the coordinate values u and v of the UV coordinate system. As described above, in the model shown here, the position of the origin T (0, 0) of the UV coordinate system coincides with the point G (x ₀ , y ₀ , z ₀ ). On the other hand, FIG. 10 is a plan view showing a distorted circular image S defined on the XY coordinate system. Here, an arbitrary point on the distorted circular image S is expressed as S (x, y) using the coordinate values x and y in the XY coordinate system.

In the image conversion apparatus according to the present invention, as shown in FIG. 10, when the user designates the position of one point P (x ₀ , y ₀ ) on the distorted circular image S as a cut-out center point (actually described later). The distorted image in the cutout area E in the vicinity thereof is converted into a regular image and output as a planar regular image T defined on the UV coordinate system shown in FIG. It has a function. In order to perform such image conversion, a one-to-one correspondence relationship is defined between one point T (u, v) on the UV coordinate system and one point S (x, y) on the XY coordinate system. It is necessary to keep it. Such a correspondence relationship can actually be expressed as a conversion equation.

  Such a conversion equation can be uniquely defined if the position and orientation of the UV coordinate system arranged in the space of the three-dimensional XYZ coordinate system are determined. For example, it is assumed that a specific UV coordinate system as shown in FIG. 4 is defined, and some planar regular image T is arranged thereon. In this case, if the planar regular image T is considered as an object to be photographed and the object is photographed using a fisheye lens corresponding to the model shown in the drawing, a distorted circular image S is obtained on the XY plane. At this time, the imaging position S (x, y) on the distorted circular image S of an arbitrary point T (u, v) on the planar regular image T that is the object to be photographed is the optical property of the fisheye lens described in FIG. Can be determined based on basic characteristics. That is, if the point H (x, y, z) on the phantom spherical surface H where the light ray L1 from the point T (u, v) is incident from the normal direction is obtained, the imaging position is S (x, y). Will be given as.

  Eventually, if the position and orientation of the UV coordinate system are determined, there is a one-to-one correspondence between one point T (u, v) on the UV coordinate system and one point S (x, y) on the XY coordinate system. Relationships can be defined, and such correspondences can be expressed as conversion equations. Image conversion processing for converting a part of the distorted circular image S into the planar regular image T can be performed by coordinate conversion calculation using this conversion calculation formula. For example, the pixel value of a pixel located at an arbitrary point T (u, v) in the planar regular image T shown in FIG. 9 is located at the corresponding point S (x, y) on the distorted circular image S shown in FIG. What is necessary is just to determine based on the pixel value of the pixel to perform. This is the basic principle of image conversion processing for converting a part of a distorted circular image into a planar regular image.

<<< §2. Comparison of conversion formulas >>
As described above, the conversion equation for associating one point T (u, v) on the UV coordinate system with one point S (x, y) on the XY coordinate system is arranged in the space of the three-dimensional XYZ coordinate system. If the position and orientation of the coordinated UV coordinate system are not determined, it cannot be uniquely defined. However, it is possible to define any conversion arithmetic expression that includes the position and orientation of the UV coordinate system as parameter values. .

  For example, the conversion arithmetic expressions shown as the expressions (1) to (9) in FIG. 11 are expressions disclosed in the above-mentioned patent documents. This conversion arithmetic expression is an expression including parameters α and β indicating the position of the UV coordinate system and a parameter ψ indicating the direction of the UV coordinate system. These three parameters are the Euler angles described in §1, that is, the azimuth angle α, the zenith angle β, and the plane tilt angle ψ. Here, the plane inclination angle ψ is a parameter indicating a rotation factor defined by a conventional general method. In FIG. 4, “the plane tilt angle ψ passes through the point G, is parallel to the XY plane, and is orthogonal to the straight line OG. "Axis (not shown)" is an angle defined as an angle ψ formed by the reference axis and the U axis (as described above, in the present invention, instead of the angle ψ, as shown in FIG. Thus, the angle φ formed between the U axis and the X ′ axis is used).

In particular,
x = R (uA + vB + wE) /
√ (u ² + v ² + w ² ) Formula (1)
Is an equation for obtaining the x coordinate value of the corresponding point S (x, y) on the XY coordinate system using the coordinate value u, v of one point T (u, v) on the UV coordinate system. , A, B, E are respectively
A = cosψcosα-sinψsinαcosβ Equation (3)
B = −sinψcosα-cosψsinαcosβ Formula (4)
E = sinβsinα Formula (7)
And is determined by calculation using a trigonometric function of Euler angles α, β, and ψ.

Similarly,
y = R (uC + vD + wF) /
√ (u ² + v ² + w ² ) Formula (2)
Is an expression for obtaining the y coordinate value of the corresponding point S (x, y) on the XY coordinate system using the coordinate values u, v of one point T (u, v) on the UV coordinate system. , C, D and F are respectively
C = cosψsinα + sinψcosαcosβ Equation (5)
D = −sinψsinα + cosψcosαcosβ Equation (6)
F = −sin βcos α Formula (8)
And is determined by calculation using a trigonometric function of Euler angles α, β, and ψ.

In the formulas (1) and (2), w is
w = mR Formula (9)
The value given by. Here, R is the radius of the distorted circular image S, and m is the conversion magnification. The conversion magnification m indicates the relationship between the scaling of the coordinate values u and v and the scaling of the coordinate values x and y, and the larger the conversion magnification m, the larger the plane regular image T is enlarged. An image is required. Actually, since the size (for example, the number of vertical and horizontal pixels) of the planar regular image T is limited, the cut region E of the distorted circular image S becomes smaller as the conversion magnification m is set larger.

  After all, in the equation shown in FIG. 11, the value of R is known as the radius of the distorted circular image S, and the value of m is known as the conversion magnification value designated (or fixed in advance) by the user. If the user specifies the Euler angles α, β, and ψ to determine the position and orientation of the UV coordinate system, the unknowns for calculating the coordinate values x and y in the conversion equation shown in FIG. v only. Therefore, by using this conversion arithmetic expression, it is possible to determine the corresponding point S (x, y) on the distorted circular image S corresponding to an arbitrary point T (u, v) in the planar regular image T.

  The image conversion method disclosed in the above-mentioned patent document converts a part of the distorted circular image S into a planar regular image T using the conversion calculation formula shown in FIG. However, the conventional image conversion apparatus that performs the conversion operation based on such a conversion operation formula has two problems as described above.

  The first problem is an operability problem seen from the user's standpoint. In order to execute the calculation based on the conversion calculation formula shown in FIG. 11, the user needs to perform an operation of designating the Euler angles α, β, and ψ. However, Euler angles α, β, and ψ are angles defined in a three-dimensional space, whereas a distorted circular image S that a user actually sees is an image displayed on a two-dimensional plane. is there. Therefore, for the user, the operation of specifying the three angles α, β, and ψ as parameters cannot be said to be an intuitive operation. From this point of view, the conventional image conversion device disclosed in the above-mentioned patent document is not necessarily an easy-to-use device from the user's standpoint.

  The second problem is that the calculation based on the conversion equation shown in FIG. Expressions (3) to (8) shown in FIG. 11 include a large number of trigonometric function terms. In general, the calculation of trigonometric functions is a considerably heavy operation for an arithmetic unit and a computer. In particular, when real-time conversion processing is required, high-performance hardware capable of performing trigonometric function operations at high speed is required, and the product cost is inevitably increased.

  The most important feature of the present invention is that instead of the conventional conversion operation expression shown in FIG. 11, a new conversion operation expression shown in FIG. 12 (hereinafter referred to as the conversion operation expression of the present invention. (It will be described in §7). Each arithmetic expression is an expression for determining a corresponding point S (x, y) on the distorted circular image S corresponding to an arbitrary point T (u, v) in the planar regular image T. This is common in that a set of function values x and y can be obtained by substituting the variable values u and v. However, the conversion calculation formulas of the present invention shown as equations (11) to (19) in FIG. 12 are compared with the conventional conversion calculation equations shown as equations (1) to (9) in FIG. It can be seen that the frequency of appearance of trigonometric functions has decreased considerably. The reason is that the azimuth angle and zenith angle are not used explicitly among the three angles, azimuth angle, zenith angle, and plane tilt angle.

In particular,
x = R [(u−x ₀ ) A + (v−y ₀ ) B + (w−z ₀ ) E] /
√ ((u−x ₀ ) ² + (v−y ₀ ) ² + (w−z ₀ ) ² ) Formula (11)
Is an equation for obtaining the x coordinate value of the corresponding point S (x, y) on the XY coordinate system using the coordinate values u, v of one point T (u, v) on the UV coordinate system. Here, A, B, and E are respectively
A = 1- (1-cosφ) (y ₀ ² + z ₀ ² ) Formula (13)
B = −z ₀ sinφ + x ₀ y ₀ (1-cosφ) Equation (14)
E = y ₀ sinφ + z ₀ x ₀ (1-cosφ) Equation (17)
It is a value obtained by the following formula. The only trigonometric functions included in this equation are sinφ and cosφ for the plane tilt angle φ.

Similarly,
y = R [(u−x ₀ ) C + (v−y ₀ ) D + (w−z ₀ ) F] /
√ ((u−x ₀ ) ² + (v−y ₀ ) ² + (w−z ₀ ) ² ) (12)
Is an equation for obtaining the y coordinate value of the corresponding point S (x, y) on the XY coordinate system using the coordinate values u, v of one point T (u, v) on the UV coordinate system. Here, C, D, and F are respectively
C = z ₀ sinφ + x ₀ y ₀ (1-cosφ) Equation (15)
D = 1- (1-cosφ) (z ₀ ² + x ₀ ² ) Formula (16)
F = −x ₀ sinφ + y ₀ x ₀ (1−cos φ) Equation (18)
It is a value obtained by the following formula. The trigonometric functions included in this equation are only sinφ and cosφ for the plane tilt angle φ.

In the equations (11) and (12), w is
w = mR Formula (19)
Is the same as that of the conventional conversion calculation formula in that R is the radius of the distorted circular image S and m is the conversion magnification.

As a result, the trigonometric functions related to the azimuth angle α and the zenith angle β included in the equation shown in FIG. 11 are not included in the equation shown in FIG. Instead, the equation shown in FIG. 12 includes the coordinate value (x ₀ , y ₀ , z ₀ ) of the point G. This is because the azimuth angle α and zenith angle β are used in the conventional equation shown in FIG. 11 as parameters indicating the position of the UV coordinate system, whereas in the equation of the present invention shown in FIG. This is because the coordinate values (x ₀ , y ₀ , z ₀ ) are used.

As shown in FIG. 4, a point G (x ₀ , y ₀ , z ₀ ) serving as the origin of the UV coordinate system passes through the cut center point P (x ₀ , y ₀ ) and is a straight line parallel to the Z axis. It is a point defined as an intersection with the spherical surface H. Here, the position of the cut-out center point P (x ₀ , y ₀ ) is set based on the user's specification (request for which part of the distorted circular image S should be cut out and converted into a planar regular image). Therefore, the intersection point G (x ₀ , y ₀ , z ₀ ) is eventually set according to the user's request. As a method for specifying the intersection point G, the conventional equation shown in FIG. 11 is an equation using two angles, an azimuth angle α and a zenith angle β, whereas the equation of the present invention shown in FIG. Is an expression using coordinate values of (x ₀ , y ₀ , z ₀ ). Further, the plane inclination angle ψ used in the conventional equation shown in FIG. 11 is “an axis (not shown) passing through the point G, parallel to the XY plane, and orthogonal to the straight line OG”. Is defined as an angle ψ formed by the reference axis and the U axis, whereas the plane inclination angle φ used in the formula of the present invention shown in FIG. 12 is as shown in FIG. , Defined as the angle between the U axis and the X ′ axis.

In this way, the position of the intersection G is expressed using the azimuth angle α and the zenith angle β, and the conventional transformation equation using the plane inclination angle ψ (FIG. 11) and the position of the intersection G are expressed as (x ₀ , y The conversion operation expression (FIG. 12) of the present invention using the coordinate value of ( ₀ , z ₀ ) and using the plane inclination angle φ can be said to be a geometrically equivalent expression. However, there is a great difference between these two conversion arithmetic expressions from the viewpoint of performing actual arithmetic processing by hardware configured by combining arithmetic units or computer software. That is, in order to perform an operation based on the conventional conversion operation expression (FIG. 11), it is necessary to perform a large number of trigonometric function operations (expressions (3) to (8)), whereas the conversion operation of the present invention. If the equation (FIG. 12) is used, the trigonometric function calculation is greatly reduced (equations (13) to (18)).

Further, from the viewpoint of user operability, the operation of setting the coordinate values (x ₀ , y ₀ , z ₀ ) of the point G as parameters compared to the operation of setting the azimuth angle α and the zenith angle β as parameters. This is much more intuitive and provides better operability. As described above, the point G is a point defined as an intersection of a straight line passing through the cut-out center point P (x ₀ , y ₀ ) and parallel to the Z axis and the virtual spherical surface H. Therefore, the user can distort the circular image. If an operation for designating the position of the cut-out center point P (x ₀ , y ₀ ) on S is performed, the coordinate value z ₀ can be automatically obtained by an operation for obtaining the intersection. Of course, the operation of designating the position of the cut-out center point P (x ₀ , y ₀ ) can be performed by an intuitive operation such as clicking one point on the distorted circular image S displayed on the display. Furthermore, compared with the conventional plane tilt angle ψ, the plane tilt angle φ of the present invention is an angle that is always defined with reference to a fixed axis called the X axis. The angle φ can be intuitively designated on the distorted circular image S.

  Thus, in the image conversion apparatus according to the present invention, instead of the calculation based on the conventional conversion calculation formula (FIG. 11), the calculation based on the conversion calculation formula (FIG. 12) of the present invention is performed. When the distorted circular image S obtained by the used photographing is converted into the planar regular image T, necessary parameters can be set by a user's intuitive operation, and the calculation burden can be reduced.

<<< §3. Parameter setting method in the present invention >>>
In order to perform the calculation based on the conversion calculation formula of the present invention, as shown in each formula of FIG. 12, the plane inclination angle φ, the coordinate value (x ₀ , y ₀ , z ₀ ) of the intersection G, and the distorted circle It is necessary to set the radius R of the image S and the conversion magnification m as parameters. In other words, if the values of these parameters are known, the function values x and y can be calculated by giving predetermined variable values u and v to the equations (11) and (12). The corresponding point S (x, y) on the XY coordinate system corresponding to an arbitrary point T (u, v) on the UV coordinate system can be specified, and the distorted circular image S is converted into a planar regular image T. Can be processed.

Here, among the parameters φ, x ₀ , y ₀ , z ₀ , R, and m, the value of z ₀ can be automatically obtained by a geometric operation for obtaining an intersection, and therefore needs to be input by the user. There is no. Further, since the radius R is a numerical value specific to the fisheye lens used for photographing, it is not necessary for the user to input if the calculation using the specific value is performed in advance. Of course, if it is possible to appropriately deal with distorted circular images taken with a plurality of types of fisheye lenses having different radius R values, different R values may be input to the user depending on the fisheye lens used. . Even in such a case, since the operation is merely an input of a numerical value, the operation burden on the user is light.

Therefore, here, a unique embodiment is described in which the remaining parameters φ, x ₀ , y ₀ , m are set efficiently by the user. This embodiment proposes an ideal man-machine interface between a user and an image conversion apparatus.

The feature of the parameter setting method described here is that, as shown in FIG. 13, a distorted circular image S defined on the XY coordinate system is presented to the user, and two points P and Q are designated on this image. is there. In this method, the values of the parameters φ, x ₀ , y ₀ , m can be set based on the position information of the two points P (x ₀ , y ₀ ), Q (x ₁ , y ₁ ). Specifically, the coordinate values (x ₀ , y ₀ ) of the point P (x ₀ , y ₀ ) are used as they are as the values of the parameters x ₀ and y ₀ . The value of the distance d between the two points P and Q is used to determine the conversion magnification m. That is, the conversion magnification m is determined by a value of m = k / d using a predetermined proportional constant k. 2 points P, the distance d between the Q coordinate values of the point P _(x 0, _{y 0)} and the coordinate values of the point Q _{(x 1, y} 1) and _{using, d = √ ((x 0} -x ₁ ) ² + (y ₀ −y ₁ ) ² ) Furthermore, if a straight line connecting the two points P and Q is a reference straight line J and an angle θ formed by the reference straight line J and the X axis is obtained (however, if both are parallel, θ = 0 °), this angle θ Can determine the plane inclination angle φ (a specific method will be described later).

In this way, when the user performs an operation of designating the two points P and Q by a method such as a mouse click on the image while looking at the distorted circular image S displayed on the display screen, The parameters φ, x ₀ , y ₀ and m are determined. Let us consider the meaning of the two points P and Q. First, the point P is nothing but the cutting center point P (x ₀ , y ₀ ) described so far. As in the example illustrated in FIG. 10, the distorted image in the cutout area E in the vicinity of the cutout center point P (x ₀ , y ₀ ) designated by the user is converted into a planar regular image. Therefore, when the user performs an operation of inputting a request to convert the “part centered here” of the distorted circular image S into a regular image as the position of the cut-out center point P (x ₀ , y ₀ ). Good.

On the other hand, the point Q is a point for designating the “cutout direction” and the “conversion magnification”, and is referred to as an auxiliary point Q here. The “cutting direction” is determined based on the positional relationship between the cutting center point P and the auxiliary point Q. In other words, the reference straight line J connecting the two points P and Q indicates the “cutting direction”. As will be described later, since the plane inclination angle φ is determined based on the angle θ formed by the reference straight line J and the X axis, the orientation of the reference straight line J shown in FIG. This is a parameter that affects the direction. If the reference straight line J rotates clockwise on the XY plane shown in FIG. 13, the U axis also rotates clockwise on the tangential plane shown in FIG. 5, which is shown in FIG. The cut-out area E that is present also rotates clockwise. Thus, the user, after the input that specifies the location of the cutting center point _{P (x} 0, _{y 0),} the desired relative position with respect to the cutting center point _{P (x} 0, _{y 0)} auxiliary By performing an input operation for designating the point Q (x ₁ , y ₁ ), it is possible to transmit a request for obtaining a regular image of the distorted circular image S “take this direction sideways”.

  As described above, the “conversion magnification” is determined by the equation m = k / d using the distance d between the two points P and Q. Usually, when the planar regular image T is displayed on the screen of the display, the size of the display area is fixed. For this reason, the larger the conversion magnification m is set, the larger the planar regular image T is displayed on the display, but the area of the cutout region E corresponding to the displayed planar regular image T Becomes smaller. Therefore, if the value of the proportionality constant k is set to an appropriate value considering the size of the display screen of the display (in the example shown here, the number of pixels of the screen corresponding to the U-axis direction), the auxiliary point The position of Q can be used as a parameter that substantially indicates the boundary of the cutout region E.

If each parameter value is set by such a method, for example, the position of the cut-out center point P (x ₀ , y ₀ ) of the distorted circular image S shown in FIG. 13 is the position of the planar regular image T shown in FIG. The direction of the reference straight line J shown in FIG. 13 corresponding to the origin T (0, 0) corresponds to the direction of the U axis shown in FIG. 9, and the position of the auxiliary point Q (x ₁ , y ₁ ) shown in FIG. Corresponds to the boundary point B shown in FIG. In other words, the user assumes a desired planar regular image T as shown in FIG. 9 in the head, and displays a portion to be displayed at the position of the origin T (0, 0) on the distorted circular image S. Is designated as a cut-out center point P (x ₀ , y ₀ ), and a portion to be displayed at the position of the boundary point B is designated as an auxiliary point Q (x ₁ , y ₁ ) on the distorted circular image S. Therefore, it is possible to perform an extremely intuitive instruction input operation with the desired planar regular image T in mind.

FIG. 14 is a plane showing an example in which three parameters of a cut center point P, an angle θ indicating the cut direction, and a conversion magnification m are set by designating two points P and Q on a specific distorted circular image. FIG. Here, as shown in the figure, a distorted face image is displayed in a part of the distorted circular image, and the user desires to obtain a planar regular image T for the portion of the cutout area E surrounding the face. Let's consider. In this case, the user first designates the cut-out center point P (x ₀ , y ₀ ) near the center of the face, and then designates the auxiliary point Q (x ₁ , y ₁ ) near the top of the head. You just have to do the operation. Then, a reference straight line J as shown in the figure is defined, an angle θ formed with the X axis is recognized, a plane inclination angle φ is determined based on this θ, and the position and orientation of the UV coordinate system (orientation of the U axis) Is determined.

FIG. 15 is a plan view showing a planar regular image T obtained based on such parameter settings. Points P and Q shown in FIG. 15 correspond to the cut center point P (x ₀ , y ₀ ) and auxiliary point Q (x ₁ , y ₁ ) shown in FIG. The reason why the face displayed on the planar regular image T is sideways is that the position of the auxiliary point Q (x ₁ , y ₁ ) is designated near the top of the face in FIG. That is, the image conversion is performed so that the direction of the reference straight line J connecting the two points P and Q shown in FIG. 14 is the U-axis direction in the planar regular image T.

In order to obtain an upright face on the planar regular image T, the user first designates a cut-out center point P (x ₀ , y ₀ ) near the center of the face in FIG. An input operation for designating the auxiliary point Q ′ (x ₂ , y ₂ ) may be performed at the right side separation position. Then, a reference straight line J ′ as shown is defined, an angle θ ′ formed with the X axis is recognized, a plane inclination angle φ is determined based on this θ ′, and the position and orientation of the UV coordinate system (U axis) Direction) is determined.

FIG. 16 is a plan view showing a planar regular image T obtained based on such parameter settings. Points P and Q ′ shown in FIG. 16 correspond to the cut center point P (x ₀ , y ₀ ) and auxiliary point Q ′ (x ₂ , y ₂ ) shown in FIG. The reason why the face displayed on the planar regular image T is an erect image is that the position of the auxiliary point Q ′ (x ₂ , y ₂ ) is designated as the right side separation position of the face in FIG. That is, the image conversion is performed such that the direction of the reference straight line J ′ connecting the two points P and Q ′ shown in FIG. 14 is the U-axis direction in the planar regular image T.

Both the image shown in FIG. 15 and the image shown in FIG. 16 are images displayed on the same display, and both have a horizontal dimension of a (the number of pixels in the horizontal direction is a) and a vertical dimension of b ( The number of pixels in the vertical direction is b). However, the values of the conversion magnification m are different between the two images, and the image shown in FIG. 15 is enlarged compared to the image shown in FIG. This is because the position of the auxiliary point is designated so that the distance between the two points P and Q ′ is larger than the distance between the two points P and Q in FIG. In FIG. 14, the cutout area E when the auxiliary point Q (x ₁ , y ₁ ) is designated is as illustrated, but the cutout area when the auxiliary point Q ′ (x ₂ , y ₂ ) is designated. E ′ (not shown) is a wider area than the cutout area E. Therefore, in the example shown in FIG. 16 in which the regular image for the cutout area E ′ is displayed, the image enlargement magnification is lower than in the example shown in FIG. 15 in which the regular image for the cutout area E is displayed. .

Thus, if the conversion magnification m is determined by the equation m = k / d using the distance d between the two points P and Q, and the proportionality constant k is set to an appropriate value, the position of the auxiliary point Q is determined as a plane. Since it can be used as a parameter for defining the boundary of the regular image T, it is convenient for providing an intuitive input operation to the user. As a point corresponding to the position of the origin T (0, 0) and the boundary point B in the desired planar regular image T as shown in FIG. 9, the user can extract the cut center point P (x ₀ , By performing an input operation for designating the positions of y ₀ ) and auxiliary points Q (x ₁ , y ₁ ), it is possible to obtain a planar regular image T corresponding to a desired cutout region E.

  Finally, the plane inclination angle φ is determined based on the angle θ indicating the cut-out direction (the angle formed by the reference straight line J and the X axis shown in FIG. 13 and θ = 0 ° when both are parallel). A specific method will be described. FIG. 17 is a perspective view showing the relationship between the angle θ and the plane inclination angle φ. The angle θ is an angle formed between the reference straight line J located on the XY plane and the X axis, whereas the plane inclination angle φ is a vector U facing the U axis direction on the UV plane and on the XY plane. It is an angle defined as an angle formed with the vector X facing the X-axis direction. FIG. 17 shows a U ′ axis (vector U ′) obtained by translating the U axis on the tangent plane S2 to the inclined plane S1, and shows the plane inclination angle φ as an angle formed by the U ′ axis and the X axis. ing.

Thus, since the reference straight line J is a straight line on the XY plane, the U ′ axis is an axis on the inclined plane S1, so that θ = φ holds when the zenith angle β = 0. If β> 0, they are not equal. In order to determine an accurate value of the plane inclination angle φ based on the angle θ, as shown in FIG. 18, an arithmetic expression using the azimuth angle α and the zenith angle β as parameters.
φ = f (θ) Equation (21)
(The function f is uniquely determined based on the geometrical arrangement shown in FIGS. 6 and 17, but is represented by a complicated expression using a trigonometric function. .)

  However, for practical use, even if the angle θ indicating the cut-out direction is approximately used as the plane inclination angle φ, no major trouble occurs. As shown in FIG. 4, the plane inclination angle φ used in the present invention is an angle defined as an angle formed by the U axis and the X ′ axis, and is an angle defined with reference to the X axis. Because it is. Of course, when the value of the angle θ is approximately used as the value of the plane inclination angle φ, the direction of the U axis intended by the user (the cut-out direction indicated by the reference straight line J) and the U coordinate system actually defined U There will be a slight deviation from the axial direction. For this reason, the planar regular image T actually obtained is slightly different from the image in the cutout area E intended by the user. However, such a difference is merely a difference in the trimming frame of the image. In addition, the direction of the reference straight line J specified by the user (that is, the position of the auxiliary point Q) is usually only an index indicating the rough cutting direction desired by the user, so that the actual cutting direction is slightly shifted. However, practically, the user does not feel uncomfortable. For this reason, in the general embodiment of the present invention, there is no problem if the angle θ indicating the cut-out direction is approximated as the value of the plane inclination angle φ as it is.

<<< §4. Basic configuration of image conversion device >>
Here, the basic configuration of the image conversion apparatus according to the basic embodiment of the present invention will be described with reference to the block diagram of FIG. In FIG. 19, the portion surrounded by the alternate long and short dash line is the image conversion apparatus 100 according to the present invention. This apparatus has a function of cutting out a part of a distorted circular image S obtained by photographing using a fisheye lens and converting it into a planar regular image T. FIG. 19 shows a camera using a fisheye lens. The distorted circular image S photographed by 10 is captured in the image conversion apparatus 100 as digital data.

  The distorted circular image storage unit 110 is a component for storing such a distorted circular image S, and includes a storage device such as a memory or a hard disk device. Here, the distorted circular image S is constituted by an aggregate of a large number of pixels arranged at positions indicated by coordinates (x, y) on a two-dimensional XY orthogonal coordinate system, for example, as illustrated in FIG. A circular image having a radius R centered on the origin O of the XY coordinate system is obtained.

  On the other hand, the planar regular image storage unit 120 is a component for storing the planar regular image T after the conversion processing as digital data, and is also configured by a storage device such as a memory or a hard disk device. Here, the planar regular image T is, for example, an aggregate of a large number of pixels arranged at positions indicated by coordinates (u, v) on a two-dimensional UV orthogonal coordinate system as illustrated in FIGS. 15 and 16. It becomes the image comprised by. If the planar regular image T in the planar regular image storage unit 120 is output as converted image digital data, the planar regular image T illustrated in FIGS. 15 and 16 can be displayed on the display screen.

  The conversion calculation unit 130 associates the coordinates (u, v) and the coordinates (x, y) using the conversion calculation formula of the present invention shown in FIG. 12, and puts them at the position indicated by the coordinates (u, v). The pixel value of the pixel on the arranged planar regular image T has a function of determining based on the pixel value of the reference pixel on the distorted circular image S arranged at the position indicated by the corresponding coordinates (x, y). The plane regular image T is calculated based on the distorted circular image S stored in the distorted circular image storage unit 110, and the generated plane regular image T is stored in the plane regular image storage unit 120. . As described in §2, it is necessary to set several parameters in order to perform an operation based on the conversion operation expression of the present invention. The following constituent elements are for performing this parameter setting.

  The distorted circular image display unit 140 is a component that displays the distorted circular image S stored in the distorted circular image storage unit 110 on the display, and an image as shown in FIG. 2 is displayed on the screen of the display.

  On the distorted circular image S displayed on the display, the instruction input unit 150 inputs an instruction for designating the positions of two points of the cut center point P and the auxiliary point Q as shown in FIG. . Specifically, the instruction input unit 150 includes a pointing device such as a mouse and a control device for controlling the pointing device, and allows the user to move the pointer on the display screen using the mouse or the like. It has a function of reading the position coordinates of the pointer when an operation such as a mouse click is performed.

  For example, the position of the pointer when the user clicks for the first time is taken in as the position of the cutting center point P, and the position of the pointer when the user clicks for the second time is taken as the position of the auxiliary point Q. The user can perform an input for instructing the positions of the cutting center point P and the auxiliary point Q by performing two click operations in succession. Alternatively, both points can be designated by a “drag and drop” operation. In this case, the position of the pointer when the user presses the mouse button is taken as the position of the cut-out center point P, the pointer is moved while the user is pressing the mouse button, and the mouse button is released. The position of the pointer may be taken in as the position of the auxiliary point Q.

In short, the instruction input unit 150 is based on some instruction input from the user who specifies the positions of the cut center point P and the auxiliary point Q on the distorted circular image S displayed by the distorted circular image display unit 140. As long as it has a function of taking in the coordinate values P (x ₀ , y ₀ ) and Q (x ₁ , y ₁ ), any device may be used.

As already described in §3, the positions of the two points P and Q input by the instruction input unit 150 are set for three parameters, that is, the position of the cutting center point P, the angle θ indicating the cutting direction, and the conversion magnification m. Used for First, the coordinate value (x ₀ , y ₀ ) of the point P itself is used as the position of the cut-out center point P. The value of the angle θ is determined as an angle formed by the reference straight line J connecting the two points P and Q and the X axis. Actually, the value of the angle θ can be calculated by a geometric calculation using the coordinate values P (x ₀ , y ₀ ) and Q (x ₁ , y ₁ ). The conversion magnification m is set in advance after the distance d between the two points P and Q is obtained by the calculation d = √ ((x ₀ −x ₁ ) ² + (y ₀ −y ₁ ) ² ). It can be calculated by performing the calculation of m = k / d using the proportional constant.

  When the conversion magnification m is set to a fixed value, it is only necessary to set the fixed value in the conversion calculation unit 130 in advance, so that it is not necessary to determine the value of m in the instruction input unit 150. In this case, since the conversion magnification m is not a parameter that can be specified by the user, the position specifying operation of the auxiliary point Q performed by the user means that the angle θ indicating the cutout direction (that is, the direction of the reference line J) is specified. The distance d between the two points P and Q has no meaning. In other words, in this case, the instruction input unit 150 has only a function of inputting the position of the cutting center point P and the cutting direction based on a user instruction, and does not have a function of inputting the conversion magnification m. It will be.

  However, in practical use, it is more convenient to obtain a planar regular image T scaled based on the conversion magnification m and to be variable according to the user's request rather than fixing the conversion magnification m. Therefore, in practice, as described above, the instruction input unit 150 has a function of inputting the conversion magnification m based on the user's instruction, and the conversion operation unit 130 receives the conversion magnification m input by the instruction input unit 150. It is preferable to perform an operation using.

The intersection calculation unit 160 determines the position coordinates (x ₀ , y ₀ , z ₀ ) of the point G based on the position (x ₀ , y ₀ ) of the cut center point P input by the instruction input unit 150. Perform the requested process. Here, as shown in FIG. 4, when a virtual spherical surface H having a radius R around the origin O is defined in the three-dimensional XYZ Cartesian coordinate system, the point G passes through the cut-out center point P to the Z axis. This is a point given as an intersection of a parallel straight line and the virtual spherical surface H. In fact, since the coordinate values x _0, y ₀ of the coordinate values x _0, y ₀ and the point G of the point P is the same value, the arithmetic processing to be performed by the intersection point calculation unit 160, the coordinate value of the point G z ₀ is only geometric operation for obtaining the.

On the other hand, as shown in FIG. 4, the angle determination unit 170 determines the U-axis direction of the two-dimensional UV orthogonal coordinate system to be defined on the tangent plane that is in contact with the virtual spherical surface H at the intersection G (x ₀ , y ₀ , z ₀ ). The cut-out direction (that is, the angle θ) input by the instruction input unit 150 with the plane inclination angle φ given as the angle formed by the oriented vector U and the vector X oriented in the X-axis direction of the two-dimensional XY orthogonal coordinate system. The process of determining based on is performed. Since the relationship between the angle θ and the angle φ is defined by the equation shown in FIG. 18, the angle determination unit 170 may calculate the plane inclination angle φ based on this equation. However, practically, as described in §3, there is no problem even if the angle θ indicating the cut-out direction is approximated as it is as the value of the plane inclination angle φ. In that case, the angle determination unit 170 may perform the process of giving the angle θ formed by the reference straight line J and the X axis to the conversion calculation unit 130 as it is as the plane inclination angle φ.

Now, as described in §2, the conversion calculation formula of the present invention shown in FIG. 12 is an orthogonal projection conversion calculation formula including the position coordinates (x ₀ , y ₀ , z ₀ ) and the plane inclination angle φ as parameters. In order for the conversion calculation unit 130 to obtain the coordinates (x, y) corresponding to the coordinates (u, v) using this conversion calculation formula, six parameters φ, x ₀ , y ₀ , z It is necessary to determine ₀ , R, and m. Here, the parameter φ is given from the angle determination unit 170, the parameters x ₀ , y ₀ , z ₀ are given from the intersection calculation unit 160, and the parameter m is given from the instruction input unit 150. The parameter R is known as the radius value of the distorted circular image S stored in the distorted circular image storage unit 110.

Eventually, all six parameters φ, x ₀ , y ₀ , z ₀ , R, m are prepared in the conversion operation unit 130. In this way, the conversion calculation unit 130 arranges the pixel value of the pixel on the planar regular image T arranged at the position indicated by the coordinates (u, v) at the position indicated by the corresponding coordinates (x, y). By determining based on the pixel value of the reference pixel on the distorted circular image S, the plane regularity for the partial image extracted from the distorted circular image S in the direction indicated by the plane inclination angle φ with the cut center point P as the center. Arithmetic processing for generating the image T can be executed.

  Note that the distorted circular image S is an image formed by an aggregate of a large number of pixels arranged at the position indicated by the coordinates (x, y) on the two-dimensional XY orthogonal coordinate system. In this example, the position of a large number of grid points arranged vertically and horizontally is constituted by digital data defining unique pixel values. For this reason, the position of the corresponding coordinate (x, y) obtained as a calculation result by the conversion calculation unit 130 is usually a position between a plurality of grid points. For example, when the distorted circular image S is composed of digital data defining pixel values at a large number of grid point positions arranged vertically and horizontally at a pitch 1, the coordinate value of any grid point is an integer value. . Therefore, if the values of the corresponding coordinates x and y obtained as the calculation result by the conversion calculation unit 130 are values including decimal numbers (in many cases, it will be the case), the position of the corresponding coordinates (x, y) is The position is between the plurality of grid points, and the corresponding pixel value cannot be determined as one.

  Therefore, actually, when the transformation calculation unit 130 determines the pixel value of the pixel on the planar regular image T arranged at the position indicated by the coordinates (u, v), the corresponding coordinates (x, y It is necessary to perform an interpolation operation on the pixel values of a plurality of reference pixels on the distorted circular image S arranged in the vicinity of the position indicated by (). Various methods such as bilinear interpolation and bicubic / spline interpolation are known as methods for performing such interpolation calculations, and thus detailed description thereof is omitted here. In addition, when a camera incorporating a single-plate image sensor is used as the camera 10 using a fisheye lens, image data called a so-called RAW mode is used as the distorted circular image S in order to perform highly accurate interpolation. Highly preferred. Some fisheye lenses have conspicuous chromatic aberrations. Therefore, if necessary, the chromatic aberration of the fisheye lens used can be measured and a calculation for correcting the chromatic aberration can be performed during the calculation by the conversion calculation unit 130. .

  The image conversion apparatus according to the present invention is particularly suitable for use in a system in which a fisheye lens is incorporated in a surveillance camera, a conference room camera, an in-vehicle camera, an in-tube probe camera, or the like. In such a system, it is necessary to handle a moving image in real time, and it is necessary to immediately convert an image in a cutout area arbitrarily designated by a user into a planar regular image and display it. In the image conversion apparatus according to the present invention, since the calculation burden is greatly reduced, it is possible to perform real-time conversion processing to a planar regular image.

  The basic configuration of the image conversion apparatus according to the basic embodiment of the present invention has been described above with reference to the block diagram of FIG. 19. This image conversion apparatus incorporates a dedicated program into a general-purpose computer. Can be configured. In this case, the distorted circular image storage unit 110 and the planar regular image storage unit 120 are configured by the storage device for the computer, and the distorted circular image display unit 140 and the instruction input unit 150 include the computer display, the mouse, and the like. What is necessary is just to comprise by the hardware and software which control this. Moreover, the conversion calculation part 130, the intersection calculation part 160, and the angle determination part 170 are implement | achieved by the calculation function of the computer based on a dedicated program.

  Of course, this image conversion apparatus can also be configured by a dedicated hardware logic circuit in which various arithmetic units and registers are combined. Specifically, at least a portion that functions as the conversion operation unit 130 may be configured by an electronic circuit, and a semiconductor integrated circuit incorporating the electronic circuit may be designed. Various ideas effective in configuring the conversion operation unit 130 by such hardware logic circuits will be described in §6.

  Further, the image conversion apparatus 100 shown in FIG. 19, the camera 10 using a fisheye lens, and a monitor apparatus (not shown) are prepared, and the distorted circular image S obtained by photographing using the camera 10 is a distorted circular image. If the planar regular image T stored in the storage unit 110 and obtained in the planar regular image storage unit 120 is displayed by the monitor device, a fish-eye monitoring system can be realized. If a digital video camera is used as the camera 10, it is possible to obtain an arbitrary part of a planar regular image on the screen of the monitor device in real time based on a distorted circular image taken in real time.

<<< §5. Application to non-orthographic images >>>
The most important feature of the present invention is that, as described in §2, the conversion calculation formula shown in FIG. 12 is used for the calculation of coordinate conversion. This conversion calculation formula is an orthogonal projection type conversion calculation formula. It is a formula on the assumption that the projection method of the fisheye lens used for photographing is an orthographic projection method. However, a commercially available fisheye lens is not necessarily an orthographic lens. Actually, various methods such as equidistant projection method, stereoscopic projection method, equisolid angle projection method are known as fisheye lens projection methods, and fisheye lenses that use these various projection methods are used depending on the application. ing. Here, a method for applying the present invention to a non-orthographic image captured by such a non-orthogonal fish-eye lens will be described.

  The optical characteristics of the orthographic fisheye lens can be explained by a model as shown in FIG. That is, with respect to an arbitrary incident point H (x, y, z) on the phantom spherical surface H, an incident light beam L1 incident from the normal direction is an XY plane as an incident light beam L2 traveling in a direction parallel to the Z axis. The characteristic is that the upper point S (x, y) is reached. FIG. 20 is a front view showing a projection state of incident light in this orthographic fisheye lens. As shown in the figure, an incident ray L1 incident from the normal direction to an incident point H (x, y, z) on the virtual sphere H having a zenith angle β is incident in a direction parallel to the Z axis. As a light ray L2, it reaches a point S (x, y) on the XY plane.

  Here, the zenith angle β of the incident point H (x, y, z) and the arrival point S (x, y) where the incident light beam L2 passing through the incident point H (x, y, z) reaches on the XY plane. In the case of an orthographic image formed by photographing using an orthographic fisheye lens, the relationship between the distance r from the origin O is expressed by the equation r = f · sinβ. Here, f is a constant specific to the fisheye lens. On the other hand, for example, in the case of an equidistant projection image formed by photographing using a fisheye lens of an equidistant projection method, the relationship between the two is expressed by the equation r = f · β.

  FIG. 21 is a perspective view showing a relationship between an orthographic projection image formed using an orthographic fisheye lens and an equidistant projection image formed using an equidistant projection fisheye lens. Concentric lines in the figure show a set of arrival points on the XY plane for incident light rays that have passed through the incident point having the same zenith angle β. In the case of the orthographic projection image shown in FIG. 21 (a), the equation r = f · sinβ holds, so that the arrangement interval between adjacent concentric lines decreases from the center toward the periphery. On the other hand, in the case of the equidistant projection image shown in FIG. 21 (b), the equation r = f · β holds, so that concentric lines are arranged at equal intervals from the center toward the periphery.

  Thus, although the orthographic projection image and the equidistant projection image are both common in terms of a distorted circular image, the distortion state is different between the two, so of course, to convert it into a planar regular image The conversion formula used is also different. Therefore, when converting an orthographic projection image into a planar regular image, the conversion formula shown in FIG. 12 can be used. However, when converting a non-orthographic projection image such as an equidistant projection image into a planar regular image. , It is necessary to use a dedicated conversion equation for each.

  However, since the optical characteristics of the orthographic fisheye lens can be handled as a model using a virtual spherical surface H as shown in FIG. 20, a conversion operation expression as shown in FIG. 12 can be defined. The optical characteristics of the non-orthographic fisheye lens cannot be handled as a model using such a virtual spherical surface H. For this reason, even if some conversion arithmetic expression can be defined, the form of the expression is expected to be extremely complicated.

  Therefore, the inventor of the present application converts the non-orthographic image into an orthographic image once instead of using a conversion operation expression that directly converts the non-orthogonal image into a planar regular image, and the orthographic projection method shown in FIG. The idea of the method of obtaining the coordinates (x, y) corresponding to the coordinates (u, v) using the conversion calculation formula is obtained. Hereinafter, this method will be described by taking an example of using an equidistant projection image as a non-orthographic projection image.

As shown in FIG. 21, an arbitrary one point on the orthographic projection image and a specific one point on the equidistant projection image can define a one-to-one correspondence relationship (the broken line in the figure is Shows this correspondence). Specifically, the coordinate of an arbitrary point on the equidistant projection image is (x ′, y ′) and the coordinate of a specific point on the orthographic image corresponding to this is (x, y). As shown in the upper part of FIG.
x = sinc (π / 2 · √ (x ′ ² + y ′ ² )) × π / 2 · x ′ Equation (31)
y = sinc (π / 2 · √ (x ′ ² + y ′ ² )) × π / 2 · y ′ Equation (32)
The following formula holds. Conversely, if the coordinate of an arbitrary point on the orthographic image is (x, y) and the coordinate of a specific point on the equidistant projection image corresponding to this is (x ′, y ′), As shown in the lower part of FIG.
x ′ = 2 / π · x / sinc (π / 2 · √ (x ² + y ² )) Equation (33)
y ′ = 2 / π · y / sinc (π / 2 · √ (x ² + y ² )) Equation (34)
Holds.

  In these equations (31) to (34), the function sinc (t) when the variable is t is a function called a sink function (cardinal sign), and is defined as sinc (t) = sin (t) / t. Is done. However, the function value when t = 0 is defined as sinc (0) = 1. After all, the above equations (31) and (32) are equations (hereinafter referred to as first coordinates) for converting coordinates (x ′, y ′) on the equidistant projection image into coordinates (x, y) on the orthographic image. The above formulas (33) and (34) are formulas for converting the coordinates (x, y) on the orthographic projection image to the coordinates (x ′, y ′) on the equidistant projection image ( Hereinafter, this is referred to as a second coordinate conversion formula.

Therefore, when the image stored in the distorted circular image storage unit 110 shown in FIG. 19 is an equidistant projection image photographed by a fisheye lens of the equidistant projection method, the distorted circular image display unit 140 displays the image on the display. The equidistant projection image is displayed as it is, and the designation input unit 150 inputs and designates two points P and Q on the equidistant projection image. Subsequently, in the intersection calculation unit 160, the coordinates (x ₀ ′, y ₀ ′) of the cut-out center point P are converted into coordinates (x ₀ , y ₀ ), and the position coordinates (x ₀ , y ₀ , z ₀ ) of the intersection point G are calculated using the converted coordinates. Then, in the conversion calculation unit 130, the coordinates (x, y) corresponding to the coordinates (u, v) are calculated using the orthogonal calculation type conversion calculation formula shown in FIG. Perform the requested process. However, instead of using the obtained coordinates (x, y) as they are to determine the position of the reference pixel, the coordinates (x, y) are used as the second coordinate conversion formulas (formulas (33) and (34)). Are converted into coordinates (x ′, y ′), and the position of the reference pixel on the equidistant projection image stored in the distorted circular image storage unit 110 is specified using the converted coordinates.

  By adopting such a method, even when the image stored in the distorted circular image storage unit 110 is an equidistant projection image captured by an equidistant projection fisheye lens, the orthographic projection method shown in FIG. It becomes possible to apply a conversion process using a conversion operation expression. Of course, this method is not limited to application to equidistant projection images, and can be widely applied to non-orthographic projection images in general.

That is, as a general theory, the image stored in the distorted circular image storage unit 110 is not an orthographic image captured by an orthographic fisheye lens, but an uncorrected image captured by a non-orthogonal fisheye lens. In the case of a projected image, the first coordinate conversion formula for converting the coordinates on the non-orthographic image to the coordinates on the orthographic image and the coordinates on the orthographic image are converted to the coordinates on the non-orthographic image. A process using the second coordinate conversion formula is performed. First, the intersection calculation unit 160, cutting center point P of coordinates _{(x 0 ', y 0'} ) is converted to coordinates _(x 0, _{y 0)} with the first coordinate conversion formula. Further, a process for obtaining the position coordinates (x ₀ , y ₀ , z ₀ ) of the intersection using the converted coordinates is executed. On the other hand, after obtaining the coordinates (x, y) corresponding to the coordinates (u, v) using the orthographic projection conversion expression shown in FIG. Is converted into coordinates (x ′, y ′) using the second coordinate conversion formula, and the position of the reference pixel on the non-orthographic image stored in the distorted circular image storage unit 110 using the converted coordinates. Can be specified.

<<< §6. Device to further reduce the computational burden >>>
As described in §2, if the conversion calculation formula of the present invention shown in FIG. 12 is used instead of the conventional conversion calculation formula shown in FIG. 11, the calculation load of the trigonometric function can be greatly reduced. Here are some ideas to further reduce this computational burden. The contrivance of the arithmetic technique described here is effective in simplifying the circuit configuration, particularly when the conversion arithmetic unit 130 is configured by a dedicated hardware logic circuit combining various arithmetic units and registers. .

<6-1: Device for Denominator Operation of Equations (11) and (12)>
The expressions (11) and (12) in FIG. 12 do not include the trigonometric function calculation, but the denominator includes three sets of square calculation terms and square root calculation terms. In order to reduce these calculation burdens as much as possible, the form of the denominator equation may be slightly changed.

That is, if only the denominator part of the equations (11) and (12) is extracted,
1 / √ ((u−x ₀ ) ² + (v−y ₀ ) ² + (w−z ₀ ) ² )
Where
a = u−x ₀ , b = v−y ₀ , c = w−z ₀
Then, the above formula takes the following form.
1 / √ (a ² + b ² + c ² )
If this equation is modified as shown in the upper part of FIG.
1 / c · 1 / √ ((a / c) ² + (b / c) ² +1) Formula (41)
Is obtained. The equation before the transformation includes three sets of square operation terms a ² , b ² , and c ² , but in the equation (41) after the transformation, the square operation term is (a / c) ^2. , (B / c) ² to 2 sets. here,
(A / c) ² + (b / c) ² + 1 = ξ
Then, the equation (42) shown in the lower part of FIG. 23 is obtained.
1 / √ (a ² + b ² + c ² ) = 1 / c · 1 / √ξ Equation (42)
In general, the reciprocal number calculation “1 / c” and the reverse square root number calculation “1 / √ξ” are relatively heavy processing operations. A method that uses a function table is a very effective method for reducing such a calculation burden. That is, the first function table T1 in which the value of “function f (c) = 1 / c” is associated with the value of various variables c in advance, and the value of “function f (ξ) = 1 / √ξ” By preparing a second function table T2 associated with the values of various variables ξ and referring to these function tables, the function value can be obtained without actually performing the calculation. Good.

  FIG. 24 is a diagram showing the configuration of such function tables T1 and T2. In the function table T1 shown in FIG. 24 (a), f (c) values (1 / c values) corresponding to individual c values with a predetermined accuracy in the range that c is expected to take. ) Is posted. Similarly, in the function table T2 shown in FIG. 24 (b), the value (1 /) of f (ξ) corresponding to each value of ξ with a predetermined accuracy in the range that ξ is expected to take. √ξ value).

If such function tables T1 and T2 are prepared in the conversion calculation unit 130, the value of 1 / c and 1 / √ξ can be obtained without performing actual calculation based on the values of c and ξ. Can be obtained. Eventually, the denominator part 1 / √ ((u−x ₀ ) ² + (v−y ₀ ) ² + (w−z ₀ ) ² ) of the equations (11) and (12) in FIG.
To perform the operation of
a = u−x ₀ , b = v−y ₀ , c = w−z ₀
After obtaining a, b, c by
ξ = (a / c) ² + (b / c) ² +1
The value of ξ is obtained by performing the following calculation, and the function f (c) corresponding to the obtained values of c and ξ by referring to the first function table T1 and the second function table T2 shown in FIG. = 1 / c, f (ξ) = 1 / √ξ
√ ((u−x ₀ ) ² + (v−y ₀ ) ² + (w−z ₀ ) ² )
= F (c) x f (ξ)
The following calculation may be performed.

  If the upper limit of the size of the planar regular image T is set, the values of the coordinate values u and v can be limited to the range of 0 ≦ u ≦ R and 0 ≦ v ≦ R, for example. The value of the magnification m can also be limited to about 0.1 ≦ m ≦ 10, for example. If such a restriction is performed, the value of c and the value of ξ can be suppressed within a predetermined range, so the values of variables c and ξ to be prepared in the function tables T1 and T2 shown in FIG. 24 are also within the predetermined range. Can be suppressed.

It should be noted that the actual calculation of the square calculation terms (a / c) ² and (b / c) ² can be omitted using a function table as needed. Since the hardware configuration of the arithmetic unit becomes simpler than that of normal multiplication, in practice, it is sufficient to actually perform the arithmetic operation using the square arithmetic unit for the square operation term.

<6-2: Device for interpolation calculation using a table>
As described above, the method of omitting actual calculation processing using the function table is a very effective method for reducing the calculation load. However, in order to improve the accuracy of the function values obtained by referring to the function table, it is necessary to secure a sufficient number of significant digits for the variable values and function values listed in the table, and the data capacity of the table increases accordingly. There is a harmful effect of doing. In order to avoid such an adverse effect, an interpolation calculation method is usually used.

  FIG. 25 is a diagram showing a linear interpolation method when referring to a function table as shown in FIG. Here, consider a case where the function value f (t) corresponding to the variable t is obtained. The illustrated t-axis is an axis indicating a variable t. Theoretically, the variable t takes an arbitrary value on the t-axis. However, the value of the variable t posted in the function table must be a discrete value of the interval W corresponding to a predetermined number of significant digits. In FIG. 25, values of A, B, and C are plotted on the t-axis as discrete values having an interval W, and function values f (A), f (B), and f (C) are defined for each. An example is shown. In this example, the value of the variable t listed in the function table is a discrete value such as A, B, and C, and the value of the function value f (t) is also f (A), f (B), f ( C).

In order to obtain a function value f (t) corresponding to an arbitrary variable t using a function table in which only the function value f (t) for the discrete variable t is defined as described above, Linear interpolation may be performed. For example, when the variable t is a value within the range of A <t <B, the function value f (t) is
f (t) = ((W−δ) / W) × f (A) + (δ / W) × f (B)
Formula (43)
Can be calculated based on the following formula. Here, δ = (t−A), which is a value corresponding to the fractional part for the discrete variable A.

  Here, an example of a hardware configuration suitable for causing the conversion calculation unit 130 to execute the interpolation calculation based on the equation (43) will be described with reference to the block diagram of FIG. As shown, the hardware shown here includes a T register 21, an even function table 22, an odd function table 23, an even reading unit 24, an odd reading unit 25, an even / odd selector 26, an even / odd selector 27, an A register 28, and a B register 29. , An interpolation calculation unit 30 and an interpolation value register 31.

  An important feature of this hardware configuration is that the data of the variable t and the function value f (t) that should be accommodated in one function table is divided into an even function table 22 and an odd function table 23. It is in. Here, the even function table 22 is a table in which the value of the predetermined function f (t) is associated with the even variable t among the variables t that take discrete values of the interval W according to the predetermined number of significant digits. The odd function table 23 is a table in which the value of a predetermined function f (t) is associated with the odd variable t among the variables t that take discrete values of the interval W according to the number of significant digits. Yes.

  Here, for convenience of explanation, a specific example in which a function table is prepared for a variable t having a range of “0.1000 to 0.1111” indicated by a 4-bit decimal part will be described. As shown in the figure, the even function table 22 contains only the variable t that takes an even number, that is, “0.1000”, “0.1010”, “0.1100”, “0.1110”, respectively. The function value f (t) corresponding to is stored (in the figure, the function value is omitted and indicated by data “********” for convenience). On the other hand, in the odd function table 23, only the variable t taking an odd number, that is, “0.1001”, “0.1011”, “0.1101”, and “0.1111” are accommodated. The function value f (t) to be stored is stored.

  As described above, the function tables 22 and 23 contain discrete values of the variable t having 4-bit significant digits, but here, as the arbitrary variable t to be calculated, 8-bit effective Consider the case where data with digits is given. The T register 21 is a register for storing the variable t having such an 8-bit effective digit number. That is, the T register 21 has a high-order bit consisting of the number of significant bits of 4 bits (the number of significant digits of the variable t in the function table), a low-order bit indicating 4 bits that are lower than the number of significant digits, Is stored. In the figure, as an example, a state where 4 bits of effective digits of “0.1010” are stored as upper bits of an arbitrary variable t and an arbitrary bit “***” is stored as lower bits. It is shown.

  The upper bits of the T register 21 are used to specify the variable t to be read from the function tables 22 and 23, and the value of the function value f (t) accommodated corresponding to the corresponding variable t is Reading is performed by the even number reading unit 24 and the odd number reading unit 25.

  Specifically, when the upper bit of the T register 21 is an even number, the even number reading unit 24 sets the value of the function f (t) associated with the even variable t indicated by the upper bit to the even function. When the upper bit of the T register 21 is an odd number read from the table 22, the value of the function f (t) associated with the smallest even variable t larger than the odd variable t indicated by the upper bit is obtained. Processing to read from the even function table 22 is performed. For example, in the illustrated example, since the upper bit “0.1010” of the T register 21 is an even number, the even number reading unit 24 is associated with the even variable t indicated by the upper bit “0.1010”. The value of the function f (t) is read from the even function table 22. If the upper bit of the T register 21 is “0.1011” (odd variable), the upper bit “0.1011” is not accommodated in the even function table 22. Therefore, the even number reading unit 24 reads the value of the function f (t) associated with “0.1100”, which is the smallest even variable larger than the odd variable “0.1011”, from the even function table 22. It will be.

On the other hand, when the upper bit of the T register 21 is an odd number, the odd reading unit 25 obtains the value of the function f (t) associated with the odd variable t indicated by the upper bit from the odd function table 23. When the upper bit of the T register 21 is an even number, the value of the function f (t) associated with the smallest odd variable t larger than the even variable t indicated by the upper bit is stored in the odd function table. The process of reading from 23 is performed. For example, in the example shown in the figure, since the upper bit “0.1010” of the T register 21 is an even number, the upper bit “0.1010” is not accommodated in the odd function table 23. Therefore, the odd number reading unit 25 reads from the odd number function table 23 the value of the function f (t) associated with “0.1011” which is the smallest odd variable larger than the even number variable “0.1010”. It will be. If the upper bit of the T register 21 is “0.1011” (odd variable), the odd reading unit 25 is a function associated with the odd variable t indicated by the upper bit “0.1011”. The value of f (t) is read from the odd function table 23.

When the function values f (t) are read from the function tables 22 and 23 by such a method, the two read function values become function values corresponding to discrete variables adjacent to each other. For example, function values f (A) and f (B) corresponding to two variables A and B adjacent on the t-axis shown in FIG. 25 are read out. In addition, the variable t having 8-bit precision accommodated in the T register is A ≦ t <B. Therefore, these two function values f (A) and f (B) are stored in the A register 28 and the B register 29, respectively, and in the interpolation calculation unit 30,
f (t) = ((W−δ) / W) × f (A) + (δ / W) × f (B)
Formula (43)
If the calculation based on is performed, the interpolation value f (t) can be obtained. Here, W is an interval of discrete values determined based on a predetermined number of significant digits constituting the upper bits of the variable t stored in the T register 21, and δ is a variable stored in the T register 21. This is the fractional part indicated by the lower bits of t.

  In short, the interpolation calculation unit 30 sets the value stored in the A register 28 to f (A), sets the value stored in the B register 29 to f (B), and indicates the value indicated by the lower bits of the T register 21. Is set to δ, the calculation based on the equation (43) is executed, and the interpolation value f (t) is calculated. The interpolation value f (t) calculated in this way is stored in the interpolation value register 31. The conversion calculation unit 130 performs a necessary calculation using the interpolation value f (t) stored in the interpolation value register 31.

  However, in order to perform an operation based on Expression (43), the function value f (A) when variable A <variable B is set is stored in the A register 28, and the function value f (B) is stored in the B register 29. There is a need to. However, the function value f (A) or the function value f (B), which is the function value read by the even number reading unit 24 from the even number function table 22 or the function value read by the odd number reading unit 25 from the odd number function table 23, is used. Whether the upper bit of the T register 21 is an even number or an odd number depends on the LSB of the upper bit (hereinafter referred to as an even / odd bit).

  Therefore, when the higher order bit is an even number (when the even / odd bit is 0), the value of the function f (t) read by the even number reading unit 24 is stored in the A register 28, and the odd number reading unit 25 The value of the read function f (t) may be stored in the B register 29. Conversely, when the higher order bit is an odd number (when the even / odd bit is 1), the value of the function f (t) read by the odd number reading unit 25 is stored in the A register 28 and the even number reading unit. The value of the function f (t) read by 24 may be stored in the B register.

  Therefore, the even / odd selector 26 selects the function f (t) read from the even number reading unit 24 and the odd number reading unit 25 according to the even / odd bit given from the T register 21 and is selected. The other is stored in the A register 28. That is, the even / odd selector 26 selects the function f (t) read from the even reading unit 24 when the even / odd bit is 0, and the function f read from the odd reading unit 25 when the even / odd bit is 1. (T) is selected, and the selected function f (t) is stored in the A register 28.

  Similarly, the even / odd selector 27 selects and selects the function f (t) read from the even number reading unit 24 and the odd number reading unit 25 according to the even / odd bit given from the T register 21. The other is stored in the B register 29. That is, the even / odd selector 27 selects the function f (t) read from the odd reading unit 25 when the even / odd bit is 0, and the function f read from the even reading unit 24 when the even / odd bit is 1. (T) is selected, and the selected function f (t) is stored in the B register 29.

<6-3: Device for Trigonometric Function Calculation of Expressions (13) to (18)>
Next, a device for reducing the load of trigonometric function calculation included in equations (13) to (18) in FIG. 12 will be described. These equations include trigonometric functions of cosφ and sinφ. Since the hardware of the arithmetic unit for obtaining the value of such a trigonometric function is considerably complicated, it is preferable in practice to avoid the trigonometric function calculation as much as possible.

  As described in §3, the angle θ indicating the cut-out direction (an angle determined based on the positions of the two points P and Q specified by the user) and the plane inclination angle φ used in the conversion formula are originally different. However, in practice, there is no problem even if the angle θ indicating the cut-out direction is approximated as the value of the plane inclination angle φ as it is. Thus, in the embodiment in which θ = φ, cosφ = cosθ, so that the value of cosφ can be obtained without performing the trigonometric function calculation according to the following principle.

As shown in FIG. 13, the angle θ indicating the cut-out direction is an angle determined by the user specifying two points P and Q on the XY plane. Therefore, as shown in FIG. 27, the user designates the point P (x ₀ , y ₀ ) and the point Q (x ₁ , y ₁ ), and the coordinate value (x ₀ , y ₀ ) is designated by the instruction input unit 150. , (X ₁ , y ₁ ).

As shown in FIG. 27, a reference straight line J is defined as a straight line connecting the two points P and Q, and an angle θ indicating the cut-out direction is defined as an angle formed by the reference straight line J and the X axis. Therefore, as illustrated, a vector J facing the direction of the reference line J and a vector X facing the X-axis direction are defined. In the illustrated example, the vector J is a vector having two points P and Q as both end points, but the size of the vector may be arbitrary. Similarly, the vector X is a vector having two points P and P ′ as both end points. Since the Y coordinate values of the point P and the point P ′ are the same, the coordinate value of the point P ′ is P (x ₂ , y ₀ ). Further, in the figure, for convenience, since the shown aligned to the position of both vectors J, point to the starting point of the X _{P (x} 0, _{y 0),} the angle θ is, the vertex point _{P (x} 0, _{y 0)} It has become a corner.

Now, the inner product of the vector J and the vector X is as shown in FIG.
(J →) ・ (X →) = cosθ ・ (Absolute value J →) ・ (Absolute value X →)
Formula (51)
Indicated by In the present specification, the vector J is represented by the symbol (J →) and the absolute value thereof is represented by (absolute value J →) due to restrictions of the electronic application. The formula expressed in the method of is published). If we take this equation for cosθ,
cosθ = (J →) ・ (X →) / ((absolute value J →) ・ (absolute value X →))
Formula (52)
Get.

Here, if vector X is a unit vector,
(Absolute value X →) = 1, x ₂ = x ₀ +1 (53)
It is. Also,
(Absolute value J →) = √ ((x ₁ −x ₀ ) ² + (y ₁ −y ₀ ) ² )
Formula (54)
Therefore, the denominator value of the equation (52) can be calculated based on the equations (53) and (54). On the other hand, the value of the numerator of equation (52) is the inner product of vector J and vector X.
(J →) · (X →) = x ₁ · x ₂ + y ₁ · y ₀
Formula (55)
It can be calculated by the following formula.

After all, the value of cosθ is the addition / subtraction / division division / square operation based on the coordinate values (x ₀ , y ₀ ) and (x ₁ , y ₁ ) of the two points P and Q (x ₁ , y ₁ ) specified by the user. Therefore, trigonometric function calculation can be avoided. In short, the conversion calculation unit 130 uses the vector J oriented in the direction of the reference straight line J, calculates the value of cosφ, the inner product of the vector J and the vector X, and the product of the magnitude of the vector J and the magnitude of the vector X. It can be obtained by an operation to be divided.

  The handling described above is handling when the angle θ indicating the cut-out direction is approximated as the value of the plane inclination angle φ, and instead of the vector U facing the direction of the U axis, the direction of the reference straight line J This means that the vector J facing the direction is approximately used. When such an approximation is not performed, the direction of the vector U is obtained by calculation, and the value of cosφ is calculated by calculating the inner product of the vector U and the vector X by the product of the size of the vector U and the size of the vector X. It can be obtained by dividing.

Thus, if the value of cosφ can be obtained, the value of sinφ is
sinφ = √ (1-cos ² φ) Equation (56)
Therefore, the values of the trigonometric functions cosφ and sinφ included in the equations (13) to (18) in FIG. 12 can be calculated without performing any trigonometric function at all. This is extremely effective in simplifying the hardware configuration of the arithmetic unit.
<6-4: Device relating to the function calculation of Expressions (31) to (34)>
In §5, the method of applying the present invention to a non-orthographic image captured using a non-orthogonal fish-eye lens has been described. In this method, as shown in FIG.
x = sinc (π / 2 · √ (x ′ ² + y ′ ² )) × π / 2 · x ′ Equation (31)
y = sinc (π / 2 · √ (x ′ ² + y ′ ² )) × π / 2 · y ′ Equation (32)
A first coordinate conversion formula
x ′ = 2 / π · x / sinc (π / 2 · √ (x ² + y ² )) Equation (33)
y ′ = 2 / π · y / sinc (π / 2 · √ (x ² + y ² )) Equation (34)
An operation using the second coordinate conversion formula is required.

  As described above, in each of the above equations, the function sinc (t) when the variable is t is a function called a sink function (cardinal sign) and is defined as sinc (t) = sin (t) / t. The However, the function value when t = 0 is defined as sinc (0) = 1. Here, a method for reducing the calculation burden of the calculation term relating to the function sinc (t) will be described.

Now, as shown in the upper part of FIG. 28, functions g (t) and h (t) are defined as follows.
g (t) = sinc (t) = sin (t) / t
However, g (0) = sinc (0) = 1 Formula (61)
h (t) = 1 / sinc (t) = t / sin (t)
However, h (0) = 1 Formula (62)
Here, when sin (t) is Taylor-expanded, as shown in the middle of FIG.
^{sin (t) = t-t} 3/3! + ^T 5/5! -T ^7/7! + ^T 9/9! − ....
Formula (63)
So, after all,
^{g (t) = 1-t} 2/3! + ^T 4/5! -T ^6/7! + ^T 8/9! − ....
Formula (64)
An expression of the form

Further, as shown in the lower part of FIG. 28, h (t) is expressed using coefficients a ₂ , a ₄ , a ₆ , a ₈ ,.
h (t) = 1 + a ₂ t ² + a ₄ t ⁴ + a ₆ t ⁶ + a ₈ t ⁸ +.
Formula (65)
Since it is h (t) = 1 / g (t) considering that it is expressed by an expression of the form: From the above expression (64),
^{h (t) = 1 / (} !! 1-t 2/3 + t 4/5 -t 6/7 + t 8/9 -!! ....)
Formula (66)
Is obtained. From Equation (65) and Equation (66),
1 = (1 + a ₂ t ² + a ₄ t ⁴ + a ₆ t ⁶ + a ₈ t ⁸ +...) ×
^{(!! 1-t 2/} 3 + t 4/5 -t 6/7 + t 8/9 -!! ....)
Formula (67)
So when you solve this,
h (t) = 1 + (1/6) t ² + (7/360) t ⁴ + (31/15120) t ⁶ + (127/604800) t ⁸ +...
Formula (68)
The following formula is obtained.
Therefore, when the intersection calculation unit 160 calculates the first coordinate conversion equations (31) and (32), the function sinc (t) is
^{sinc (t) = 1-t} 2/3! + ^T 4/5! -T ^6/7! + ^T 8/9! − ....
If the calculation is performed based on the expression in the Taylor expanded form, trigonometric function calculation can be avoided. In addition, when t = 0, the value on the right side of the above equation is 1, so that a function value that matches the mathematical definition of sinc (0) = 1 can be obtained. Note that the right side of the above equation theoretically includes an infinite number of terms, but of course it is practically sufficient to calculate up to a term that provides the required number of significant digits.

Similarly, when the conversion calculation unit 130 calculates the second coordinate conversion formulas (33) and (34), the function 1 / sinc (t) is converted into predetermined coefficient values a ₂ , a ₄ , a ₆ , a _8, 1 / sinc with ^{_{.... (t) = 1 + a}} 2 t 2 + a 4 t 4 + a 6 t 6 + a 8 t 8 + ....
If the calculation is performed based on an expression of the form, trigonometric function calculation can be avoided. Here, the values of the coefficients a ₂ , a ₄ , a ₆ , a ₈ ,... Can be obtained as specific numerical values as in the equation (68). It is only necessary to store typical numerical values. Also in this equation, when t = 0, the value on the right side is 1, so that a function value that matches the mathematical definition of 1 / sinc (0) = 1 can be obtained. Again, the right side of the above equation theoretically includes an infinite number of terms, but in practice it is sufficient to calculate up to the term that provides the required number of significant digits.

<<< §7. Mathematical process for deriving transformation formulas >>>
As described above, in §1 to §6, the best mode for carrying out the image conversion apparatus according to the present invention has been described. Here, a mathematical process for deriving the conversion arithmetic expression of the present invention shown in FIG. 12 will be briefly described with reference to the mathematical expressions shown in FIGS. The conversion equation shown in FIG. 12 is geometrically equivalent to the conversion equation shown in FIG. 11. The difference between the two is the parameter setting including the difference in definition of the plane inclination angles ψ and φ. It is only how. In any of the conversion arithmetic expressions, an arbitrary point S (x, y) on the orthographic image S formed using an orthographic fisheye lens having optical characteristics as shown in FIG. This is an expression showing the correspondence with a specific point T (u, v) on T, and can be derived by performing a geometric analysis. However, since complicated expressions including trigonometric functions need to be handled, analysis using determinants is usually performed. In the following, although the explanation will require some mathematical knowledge, a process for deriving a conversion operation expression using this determinant will be described.

In FIG. 6, the UV coordinate system is defined on the tangent plane S2, and the direction of the U axis is determined by the plane inclination angle φ. The origin of this UV coordinate system is given as a point G (x ₀ , y ₀ , z ₀ ). Therefore, if a rotation vector N (x ₀ , y ₀ , z ₀ ) starting from the origin O of the XYZ coordinate system and ending at the origin G of the UV coordinate system is defined, it is arranged in the space of the XYZ coordinate system. A specific UV coordinate system is defined by the rotation vector N (x ₀ , y ₀ , z ₀ ) and the plane tilt angle φ (rotation factor around the rotation vector N). As described above, the specific UV coordinate system is defined by the rotation vector N and the rotation factor φ (plane inclination angle) around the rotation vector, and therefore can be expressed as a rotation matrix.

In the present invention, it is sufficient if a two-dimensional UV coordinate system can be defined. Here, a three-dimensional UVW coordinate system in which a W axis perpendicular to the UV plane is further added will be considered. For example, in the example shown in FIG. 6, the W axis is defined as an axis in a direction from the point G (x ₀ , y ₀ , z ₀ ) toward the origin O. Then, the transformation calculation formula to be obtained here is a formula for performing coordinate transformation between the three-dimensional XYZ coordinate system and the three-dimensional UVW coordinate system (w shown in the formula (19) in FIG. It is a value corresponding to a coordinate value along the W axis).

  Here, it is assumed that the radius R of the phantom spherical surface H is 1, and the rotation vector N is a unit vector. Further, in order to be able to derive a conversion formula taking into account only the rotation of the coordinate system, the three-dimensional UVW coordinate system Let us consider the mutual relationship between the two coordinate systems while the three-dimensional UVW coordinate system is translated so that the position of the origin G overlaps the position of the origin O of the three-dimensional XYZ coordinate system. That is, in FIG. 6, the UV coordinate system located on the tangent plane S2 is translated on the inclined plane S1.

Now, a three-dimensional rotation matrix defined by the rotation vector N = (x, y, z) and the rotation angle φ is R (φ, N). Then, as shown in the equation (71) in FIG. 29, a matrix S called an alternating matrix is considered for the rotation vector N = (x, y, z). Here, the code T on the right side of the matrix means a transposed matrix. Then, the matrix S ² is expressed by the equation (72), and the matrix S ³ is expressed by the equation (73).

  On the other hand, the exponential matrix expS for the matrix S is defined in an expanded form as shown in Expression (74). Here, E is a unit matrix. Since equation (74) can be transformed as in equation (75), the exponential matrix exp (φS) multiplied by the rotation angle φ can be transformed into equation (76) in FIG. 30 to obtain equation (77). Can do. Since this matrix exp (φS) becomes the three-dimensional rotation matrix expression R (φN) = R (φ, N), when calculating by substituting the equations (71) and (72) into the equation (77), the equation (78) is obtained. Each element of the determinant of 3 rows and 3 columns shown at the end of the equation (78) is a coefficient value when coordinate conversion between the three-dimensional XYZ coordinate system and the three-dimensional UVW coordinate system is performed. If the nine elements of this determinant are replaced with symbols A to I, then equation (79) is obtained.

  Terms A to F shown in the equations (13) to (18) in FIG. 12 correspond to A to F shown in the equation (79). Since the image to be converted in the present invention is a distorted circular image arranged on the XY plane, the x coordinate value is obtained from equation (11) in FIG. 12, and the y coordinate value is obtained from equation (12) in FIG. Since it is sufficient if the z coordinate value is always 0, the elements G, H, and I in the equation (79) are not used.

Note that “u−x ₀ ”, “v−y ₀ ”, and “w−z ₀ ” are used instead of u, v, and w in the equations (11) and (12) in FIG. 12. This is because, as described above, this derivation process is performed in a state where the UV coordinate system located on the tangent plane S2 is translated on the inclined plane S1 in FIG. Actually, the origin of the UV coordinate system is not located at the origin O of the XYZ coordinate system, but is located at the point G (x ₀ , y ₀ , z ₀ ). The subtraction of the coordinate values x ₀ , y ₀ , and z ₀ in the terms “u−x ₀ ”, “v−y ₀ ”, and “w−z ₀ ” of the expressions (11) and (12) This is for correcting the position of.

<<< §8. Modification of parameter setting method >>
Finally, a modification of the parameter setting method in the present invention will be described. In §3, a basic example has been described in which the user can set parameters φ, x ₀ , y ₀ , m efficiently. As shown in FIG. 13, the outline of the method described in §3 presents the user with a distorted circular image S defined on the XY coordinate system and designates two points P and Q on this image, The coordinate value (x ₀ , y ₀ ) of the point P (x ₀ , y ₀ ) is used as it is as the value of the parameter x ₀ , y ₀ , and the distance d between the two points P and Q is used, and m = k / The conversion magnification m is determined by the formula d, and a straight line connecting the two points P and Q is set as a reference straight line J, and an angle θ (cutting direction) formed by the reference straight line J and the X axis is obtained. Based on this, the plane inclination angle φ is determined (practically, φ = θ is approximated).

On the other hand, in the modification described here, the conversion magnification m and the angle θ indicating the cut-out direction are set separately in advance, and only one point P (x ₀ , y ₀ ) on the distorted circular image S. It is a method to specify. In this modification, first, the instruction input unit 150 shown in FIG. 19 displays a “direction / magnification setting screen 151” as shown in FIG. 31 on the display screen. In the case of the illustrated example, the user performs an operation of inputting a value of θ in the angle θ setting input field 152 and inputting a value of m in the magnification m setting input field 153 using an input device such as a keyboard. Become. Specifically, for example, the value of θ may be input as “30 °” in the input field 152, and the value of m may be input as “2” in the input field 153. Θ thus input becomes a value that defines the direction of the reference straight line J to be drawn on the distorted circular image S. Of course, various other methods can be used for inputting θ and m. For example, an arrow mark for increasing / decreasing a numerical value may be displayed on the screen, and a specific numerical value may be designated by a user's operation of clicking the arrow mark.

In this way, if the conversion magnification m and the angle θ indicating the cut-out direction are set by some method, only the cut-out center point P (x ₀ , y ₀ ) can be designated on the distorted circular image S. That's fine. FIG. 32 is a plan view showing an operation for setting such a cutting center point P. FIG. Although not shown in FIG. 32, actually, a distorted circular image S is displayed in the circle in the figure, and the user can select any arbitrary image in the distorted circular image S by, for example, clicking the mouse. An operation of designating one point as the cut center point P (x ₀ , y ₀ ) is performed. When the user designates the position of the cutting center point P (x ₀ , y ₀ ), as a straight line that passes through the point P and forms the angle θ (angle value input to the input field 152 by the above operation), A reference straight line J as shown by a one-dot chain line in the figure can be defined, and an image is cut out based on the direction of the reference straight line J.

  The modification of the parameter setting method described in §8 is optimal for applications in which plane regular images at various positions are obtained with the cutout direction (direction of the reference straight line J) always fixed. For example, as shown in the side view of FIG. 33, consider an embodiment in which the present invention is applied to a surveillance camera attached to the wall of a building. As shown in the figure, a surveillance camera 230 is attached to the wall surface of a building 220 built on the road surface 210, and it is assumed that the task of monitoring a visual field as shown by a one-dot chain line in the figure is performed. Here, if the monitoring camera 230 is a camera provided with a normal lens, an image taken from the position of the monitoring camera 230 becomes a normal flat regular image as shown in FIG. 34, for example. However, when a camera using a fish-eye lens is used as the monitoring camera 230 in order to expand the field of view, a distorted circular image S as shown in FIG. 35 is obtained on the XY plane.

  Here, comparing the distorted circular image S shown in FIG. 2 with the distorted circular image S shown in FIG. 35, both are images taken by a camera equipped with a fisheye lens. Is the image obtained by setting the camera so that the vertical direction is vertical and photographing the landscape of the ground over 360 degrees around the latter, the latter is set the camera so that the optical axis is horizontal, It can be seen that this is an image obtained by photographing the scenery on one side of the building 220. Therefore, considering that the planar regular image obtained on the UV coordinate system after the conversion is directed to be in the correct direction according to the natural world top and bottom, in the former case, the cut-out direction (reference straight line defined by the angle θ) J direction) must be specified in various directions depending on the position of the cut-out center point. In the latter case, it is understood that the direction should always be set parallel to the X axis.

  Therefore, when the present invention is used for converting the distorted circular image S obtained by using the fisheye lens monitoring camera 230 in the environment shown in FIG. 33, for example, the method shown in FIG. The angle θ indicating the cutout direction is set to 0 ° in advance, and a predetermined numerical value is also set in advance for the conversion magnification m, and only the position of the cutout center point P is designated on the distorted circular image S shown in FIG. It is preferable to perform this. In the present invention, θ is an angle defined as an angle formed by the reference straight line J and the X axis. As described above, when θ = 0 °, the reference straight line J and the X axis are parallel. Indicates. Therefore, when the setting of θ = 0 ° is performed, the reference straight line J is a straight line that passes through the cutting center point P specified by the user and is parallel to the X axis.

  FIG. 36 is a plan view showing a cutting center point designation operation on the distorted circular image S shown in FIG. 35 when θ = 0 ° is set in advance. For example, when the user wants to visually recognize the vicinity of one point P1 on the road surface as a planar regular image while viewing the distorted circular image S as shown in the figure, the user can click the point P1 by clicking the point P1 or the like. What is necessary is just to perform operation designated as a cutting center point. Then, the reference straight line J1 passing through the point P1 and parallel to the X axis is a straight line indicating the cut-out direction. Similarly, when it is desired to visually recognize the vicinity of one point P2 of a distant building as a planar regular image, an operation of specifying the point P2 as a cut-out center point by a click operation or the like may be performed. Then, the reference straight line J2 that passes through the point P2 and is parallel to the X axis is a straight line indicating the cut-out direction.

  As shown in FIG. 36, the reference straight lines J1 and J2 are both in the direction along the horizontal plane of the natural world, and the direction of the U axis is substantially defined in this direction. Accordingly, the finally obtained planar regular image is an image oriented in the correct direction according to the top of the natural world with the cut center points P1 and P2 as the centers. Further, since the user only needs to perform a simple operation of designating “the center of the region desired to be visually recognized as a planar regular image” as the cut-out center point P while viewing the distorted circular image S, good operability can be obtained. It is done.

It is a perspective view which shows the basic model which forms the distortion circular image S by imaging | photography using the fisheye lens of an orthogonal projection system. It is a top view which shows an example of the distortion circular image S obtained by imaging | photography using a fisheye lens (It shows the general image of the distortion circular image S, and does not show an exact image.). 6 is a plan view showing an example in which a cutout region E is defined in a part of a distorted circular image S. FIG. 4 is a perspective view showing a relationship between an XY coordinate system including a distorted circular image S and a UV coordinate system including a planar regular image T. FIG. It is a top view which shows the relationship between the plane regular image T defined on the UV coordinate system, and plane inclination-angle (phi). It is a perspective view which shows the principle of the coordinate transformation from XY coordinate system to UV coordinate system. It is the figure which looked at each component shown by the perspective view of FIG. 6 from the horizontal direction. It is the figure which looked at each component shown by the perspective view of FIG. 6 from upper direction. It is a top view which shows the plane regular image T defined on UV coordinate system. It is a top view which shows the distortion circular image S defined on the XY coordinate system. It is a figure which shows the conventional general conversion formula which used Euler angles (alpha), (beta), and ( psi) as a parameter. It is a figure which shows the conversion formula based on this invention using the coordinate value (x0, y0, z0) which shows a rotating shaft, and plane inclination angle (phi) (rotating angle around a rotating shaft) as a parameter. By specifying two points P and Q on the XY plane, three parameters are set: the cut center point, the angle θ indicating the cut direction (the angle used to determine the plane tilt angle φ), and the conversion magnification m. It is a top view which shows the principle to do. An example of setting three parameters, ie, a cutting center point, an angle θ indicating the cutting direction, and a conversion magnification m, by specifying two points P and Q or two points P and Q ′ on a specific distorted circular image. FIG. In the example shown in FIG. 14, it is a top view which shows the planar regular image T obtained when two points P and Q are designated. FIG. 15 is a plan view showing a planar regular image T obtained when two points P and Q ′ are designated in the example shown in FIG. 14. It is a perspective view which shows the relationship between angle (theta) which shows the cutting direction shown in FIG. 13, and plane inclination angle (phi). It is a figure which shows the computing equation which calculates planar inclination | tilt angle (phi) based on angle (theta) which shows a cutting direction. 1 is a block diagram showing a basic configuration of an image conversion apparatus according to the present invention. It is a front view which shows the projection state of the incident light in the fisheye lens of an orthogonal projection system. It is a perspective view which shows the relationship between the orthographic projection image formed using the orthographic projection fisheye lens, and the equidistant projection image formed using the equidistant projection fisheye lens. It is a figure which shows the conversion type for performing coordinate conversion between the coordinate on an equidistance projection image, and the coordinate on an orthographic projection image. It is a figure which shows the formula used in order to reduce the calculation burden of the conversion formula based on this invention shown in FIG. It is a figure which shows the function table used in order to reduce the calculation burden of the conversion formula based on this invention shown in FIG. It is a figure which shows the method of the linear interpolation in the case of referring the function table as shown in FIG. FIG. 26 is a block diagram showing hardware components for performing the linear interpolation shown in FIG. 25. It is a figure which shows another type | formula used in order to reduce the calculation burden of the conversion formula based on this invention shown in FIG. It is a figure which shows the formula used in order to reduce the calculation burden of the coordinate transformation shown in FIG. It is a figure which shows the mathematical process for deriving | leading-out the calculation formula which concerns on this invention shown in FIG. It is another figure which shows the mathematical process for deriving | leading-out the calculation formula which concerns on this invention shown in FIG. FIG. 10 is a plan view showing an “orientation / magnification setting screen” displayed on a display screen in a parameter setting method according to a modification of the present invention. It is a top view which shows operation which sets the cutting center point by designating 1 point P on XY plane. It is a side view which shows embodiment which utilized this invention for the surveillance camera attached to the wall surface of a building. It is a top view which shows an example of the picked-up image obtained when the camera provided with the normal lens is used as the monitoring camera 230 shown in FIG. It is a top view which shows an example of the picked-up image (distortion circular image S) obtained when the camera provided with the fisheye lens is used as the monitoring camera 230 shown in FIG. FIG. 36 is a plan view showing a cutting center point designation operation on the distorted circular image S shown in FIG. 35.

Explanation of symbols

10: Camera using a fisheye lens 21: T register 22: Even function table 23: Odd function table 24: Even reading unit 25: Odd reading unit 26: Even / odd selector 27: Even / odd selector 28: A register 29: B register 30: Interpolation Calculation unit 31: Interpolation value register 100: Image conversion device 110: Distorted circular image storage unit 120: Planar regular image storage unit 130: Conversion calculation unit 140: Distorted circular image display unit 150: Instruction input unit 151: Direction / magnification setting Screen 152: Angle θ setting input field 153: Magnification m setting input field 160: Intersection calculation unit 170: Angle determination unit 210: Road surface 220: Building 230: Surveillance camera a: Horizontal dimension of display (number of pixels in the horizontal direction)
b: Vertical dimension of display (number of pixels in vertical direction)
B: boundary point d: distance between two points PQ: cut-out area f: constant specific to fisheye lens f (t): function value G for variable t (x ₀ , y ₀ , z ₀ ): passing through point P Intersection H of a straight line parallel to the Z-axis and the virtual spherical surface H: virtual spherical surface H (x, y, z): incident points J, J ′, J1, J2 on the virtual spherical surface H: reference straight lines L1, L2: incident light rays m: conversion magnification n: normal vector N: rotation vector O: origin P (x ₀ , y ₀ ), P 1, P _{2 of} the three-dimensional XYZ orthogonal coordinate system: cutting center point P ′ (x ₂ , y ₀ ): Auxiliary point Q (x ₁ , y ₁ ), Q ′ (x ₂ , y ₂ ): auxiliary point R: radius of distorted circular image S (radius of phantom spherical surface H)
r: distance from the center point of the distorted circular image S: distorted circular image S1: inclined surface S2: tangent plane S (x, y): point on the distorted circular image S: variable T: planar regular image T (u , V): points T1, T2 on the plane regular image T: function table U: coordinate axis U ′ of the two-dimensional UV orthogonal coordinate system: axis parallel to the U axis V: coordinate axis W of the two-dimensional UV orthogonal coordinate system W: variable t Discrete value interval X: coordinate axis X ′ of three-dimensional XYZ orthogonal coordinate system X: axis parallel to X axis Y: coordinate axis Z of three-dimensional XYZ orthogonal coordinate system Z: coordinate axis α of three-dimensional XYZ orthogonal coordinate system α: azimuth β: zenith Angle δ: fraction θ indicated by lower bits, θ ′: angle φ indicating cutting direction, ψ: plane inclination angle ξ: variable

Claims (26)

An image conversion apparatus that performs a process of cutting out a part of a distorted circular image obtained by shooting using a fisheye lens and converting it into a planar regular image,
A distorted circular image having a radius R centered at the origin O of the XY coordinate system, which is composed of a collection of a large number of pixels arranged at positions indicated by coordinates (x, y) on a two-dimensional XY orthogonal coordinate system. A distorted circular image storage unit for storing
A planar regular image storage unit for storing a planar regular image configured by an aggregate of a large number of pixels arranged at a position indicated by coordinates (u, v) on a two-dimensional UV orthogonal coordinate system;
A distorted circular image display unit for displaying the distorted circular image stored in the distorted circular image storage unit on a display;
An instruction input unit for inputting the position and cutting direction of the cut-out center point P on the distorted circular image displayed on the display based on a user's instruction;
In a three-dimensional XYZ Cartesian coordinate system including the two-dimensional XY Cartesian coordinate system, when a virtual spherical surface having a radius R with the origin O as the center is defined, a straight line passing through the cut-out center point P and parallel to the Z-axis An intersection calculation unit for obtaining position coordinates (x ₀ , y ₀ , z ₀ ) of the intersection G with the virtual sphere;
A vector U oriented in the U-axis direction of the two-dimensional UV orthogonal coordinate system and a vector X oriented in the X-axis direction of the two-dimensional XY orthogonal coordinate system to be defined on the tangent plane that is in contact with the virtual sphere at the intersection G And an angle determination unit that determines a plane inclination angle φ given as an angle formed by the cut-out direction,
The coordinate (u, v) and the coordinate (x, y) are obtained using an orthographic projection conversion equation including the position coordinates (x ₀ , y ₀ , z ₀ ) and the plane inclination angle φ as parameters. The pixel values of the pixels on the planar regular image arranged at the position indicated by the coordinates (u, v) are referred to on the distorted circular image arranged at the position indicated by the corresponding coordinates (x, y). An operation for generating a planar regular image for a partial image cut out from the distorted circular image in the direction indicated by the plane inclination angle φ from the distorted circular image by determining based on the pixel value of the pixel A conversion operation unit that stores the generated planar regular image in the planar regular image storage unit,
An image conversion apparatus comprising: The image conversion apparatus according to claim 1,
The conversion calculation unit is an orthographic conversion calculation expression.
x = R [(u−x ₀ ) A + (v−y ₀ ) B + (w−z ₀ ) E] /
√ ((u−x ₀ ) ² + (v−y ₀ ) ² + (w−z ₀ ) ² )
y = R [(u−x ₀ ) C + (v−y ₀ ) D + (w−z ₀ ) F] /
√ ((u−x ₀ ) ² + (v−y ₀ ) ² + (w−z ₀ ) ² )
here,
A = 1- (1-cosφ) (y ₀ ² + z ₀ ² )
B = −z ₀ sinφ + x ₀ y ₀ (1-cosφ)
C = z ₀ sinφ + x ₀ y ₀ (1-cosφ)
D = 1- (1-cosφ) (z ₀ ² + x ₀ ² )
E = y ₀ sinφ + z ₀ x ₀ (1-cosφ)
F = −x ₀ sinφ + y ₀ x ₀ (1-cosφ)
w = mR (where m is a predetermined conversion magnification)
An image conversion apparatus using the following formula. The image conversion apparatus according to claim 1 or 2,
The instruction input unit has a function of inputting an instruction for defining the reference straight line J drawn on the distorted circular image,
An image conversion apparatus characterized in that an angle determination unit determines a plane inclination angle φ based on an angle θ formed by the reference straight line J and the X axis (provided that θ = 0 ° when both are parallel). . The image conversion apparatus according to claim 3.
The instruction input unit has a function of inputting an instruction for designating the positions of two points of the cut-out center point P and the auxiliary point Q on the distorted circular image, and the cut-out center point P and the auxiliary point An image conversion apparatus characterized in that a straight line connecting Q is a reference straight line J. The image conversion apparatus according to claim 3.
An instruction input unit inputs a numerical value indicating an angle θ formed by the reference straight line J and the X axis on a predetermined input screen, and an instruction for specifying the position of the cutting center point P on the distorted circular image. And an input function. In the image conversion device according to any one of claims 3 to 5,
An image conversion apparatus characterized in that the angle determination unit approximately sets the angle θ formed by the reference straight line J and the X axis to a plane inclination angle φ. The image conversion apparatus according to claim 2,
The instruction input unit has a function of inputting the conversion magnification m based on a user instruction,
An image conversion apparatus, wherein the conversion operation unit performs an operation using the conversion magnification m input by the instruction input unit. The image conversion device according to claim 7,
The instruction input unit has a function of inputting an instruction for designating the positions of two points of the cut-out center point P and the auxiliary point Q on the distorted circular image, and the cut-out center point P and the auxiliary point An image conversion apparatus characterized in that, based on a distance d to Q, a numerical value given as m = k / d (k is a predetermined proportionality constant) is set as a conversion magnification m. In the image conversion device according to any one of claims 1 to 8,
When the transformation calculation unit determines the pixel value of the pixel on the planar regular image arranged at the position indicated by the coordinates (u, v), it is arranged near the position indicated by the corresponding coordinates (x, y). An image conversion apparatus that performs an interpolation operation on pixel values of a plurality of reference pixels on a distorted circular image. In the image conversion device according to any one of claims 1 to 9,
If the image stored in the distorted circular image storage unit is not an orthographic image captured by an orthographic fisheye lens but a non-orthographic image captured by a non-orthogonal fisheye lens, A first coordinate conversion formula that converts coordinates on the projected image into coordinates on the orthographic image, and a second coordinate conversion formula that converts coordinates on the orthographic image into coordinates on the non-orthographic image; , Using
The intersection calculation unit converts the coordinates of the cut-out center point P using the first coordinate conversion formula and obtains the position coordinates (x ₀ , y ₀ , z ₀ ) of the intersection using the converted coordinates. And
After the conversion calculation unit obtains the coordinates (x, y) corresponding to the coordinates (u, v) using the orthogonal calculation type conversion calculation expression, the coordinates (x, y) are converted into the second coordinate conversion expression. An image conversion apparatus characterized in that the position of a reference pixel on a distorted circular image is specified using the converted coordinates using the converted coordinates. The image conversion apparatus according to claim 10, wherein
When the image stored in the distorted circular image storage unit is an equidistant projection image taken by an equidistant projection fisheye lens,
As the first coordinate conversion formula, the intersection calculation unit converts the coordinates (x ′, y ′) on the equidistant projection image into the coordinates (x, y) on the orthographic projection image.
x = sinc (π / 2 · √ (x ′ ² + y ′ ² )) × π / 2 · x ′
y = sinc (π / 2 · √ (x ′ ² + y ′ ² )) × π / 2 · y ′
Using the formula
As a second coordinate conversion formula for converting the coordinate (x, y) on the orthographic projection image into the coordinate (x ′, y ′) on the equidistant projection image,
x ′ = 2 / π · x / sinc (π / 2 · √ (x ² + y ² ))
y ′ = 2 / π · y / sinc (π / 2 · √ (x ² + y ² ))
An image conversion apparatus using the following formula.   The program for functioning a computer as an image conversion apparatus in any one of Claims 1-11.   A semiconductor integrated circuit in which an electronic circuit functioning as a conversion calculation unit that is a constituent element of the image conversion apparatus according to claim 1 is incorporated. An image conversion device according to any one of claims 1 to 11, a camera using a fisheye lens, and a monitor device that displays a planar regular image on a screen,
A distorted circular image obtained by photographing using the camera is stored in a distorted circular image storage unit, and a planar regular image obtained in the planar regular image storage unit is displayed on the monitor device. This is a fish-eye monitoring system. The image conversion apparatus according to claim 2,
The conversion operation part
A first function table in which the value of “function f (c) = 1 / c” is associated with the value of various variables c, and the value of “function f (ξ) = 1 / √ξ” A second function table associated with the values of
a = u−x ₀ , b = v−y ₀ , c = w−z ₀ ,
ξ = (a / c) ² + (b / c) ² +1
To obtain the values of c and ξ,
By referring to the first function table and the second function table, values of the functions f (c) and f (ξ) corresponding to the obtained values of c and ξ are obtained.
In the transformation formula of orthographic projection
√ ((u−x ₀ ) ² + (v−y ₀ ) ² + (w−z ₀ ) ² )
An image conversion apparatus characterized in that the following value is obtained by a calculation of f (c) × f (ξ). The image conversion device according to any one of claims 1 to 11,
The conversion operation part
An even function table in which a value of a predetermined function f (t) is associated with an even variable t among variables t taking discrete values of the interval W according to a predetermined number of significant digits;
An odd function table in which a value of a predetermined function f (t) is associated with an odd variable t among variables t taking discrete values of the interval W according to the number of significant digits;
A T register for storing a variable t composed of an upper bit composed of the significant digits and a lower bit indicating a digit lower than the significant digits;
When the upper bit is an even number, the value of the function f (t) associated with the even variable t indicated by the upper bit is read from the even function table, and the upper bit is an odd number. An even number reading unit that reads out the value of the function f (t) associated with the smallest even number variable t larger than the odd number variable t indicated by the upper bits from the even number function table;
When the upper bit is an odd number, the value of the function f (t) associated with the odd variable t indicated by the upper bit is read from the odd function table, and the upper bit is an even number. Is an odd reading unit that reads the value of the function f (t) associated with the smallest odd variable t larger than the even variable t indicated by the upper bits from the odd function table;
An A register that stores the value of the function f (t) read from the even function table or the odd function table;
A B register for storing the value of the function f (t) read from the even function table or the odd function table;
When the upper bits are even, the value of the function f (t) read by the even reading unit is stored in the A register, and the value of the function f (t) read by the odd reading unit is stored in the A register. When stored in the B register and the upper bit is an odd number, the value of the function f (t) read by the odd reading unit is stored in the A register, and the function f (t) read by the even reading unit is stored. an even / odd selector for storing the value of t) in the B register;
The value stored in the A register is f (A), the value stored in the B register is f (B), the value indicated by the lower bits is δ, and the function f (t after interpolation) ) Is obtained by an operation of f (t) = ((W−δ) / W) × f (A) + (δ / W) × f (B),
An image conversion apparatus that performs calculation using the value of the interpolated function f (t) obtained by the interpolation calculation unit. The image conversion apparatus according to claim 11,
The intersection calculation unit calculates the function sinc (t) in the first coordinate conversion formula,
^{sinc (t) = 1-t} 2/3! + ^T 4/5! -T ^6/7! + ^T 8/9! − ....
Calculate based on the expression of the Taylor expansion form
The transformation calculation unit uses the function 1 / sinc (t) in the second coordinate transformation formula as 1 / sinc (t) using predetermined coefficient values a ₂ , a ₄ , a ₆ , a ₈ ,. = 1 + a ₂ t ² + a ₄ t ⁴ + a ₆ t ⁶ + a ₈ t ⁸ +.
An image conversion apparatus that performs an operation based on an expression of the form: The image conversion apparatus according to claim 2,
An image conversion apparatus characterized in that the conversion operation unit obtains the value of cosφ by dividing the inner product of the vector U and the vector X by the product of the magnitude of the vector U and the magnitude of the vector X. The image conversion apparatus according to claim 2,
The instruction input unit has a function of inputting an instruction for defining the reference straight line J drawn on the distorted circular image,
The transformation operation unit defines a vector J facing the direction of the reference line J, and divides the value of cosφ by the inner product of the vector J and the vector X by the product of the vector J and the vector X. An image conversion apparatus characterized by being obtained by: The image conversion apparatus according to claim 2,
Image converting apparatus converting calculation unit, the value of sin [phi, and obtaining by sin [phi = √ becomes (1-cos ² φ) operation. An image conversion apparatus that performs a process of cutting out a part of a distorted circular image obtained by shooting using a fisheye lens and converting it into a planar regular image,
A distorted circular image storage unit for storing a distorted circular image configured by an aggregate of a large number of pixels arranged at a position indicated by coordinates (x, y) on a two-dimensional XY orthogonal coordinate system;
A planar regular image storage unit for storing a planar regular image configured by an aggregate of a large number of pixels arranged at a position indicated by coordinates (u, v) on a two-dimensional UV orthogonal coordinate system;
A distorted circular image display unit for displaying the distorted circular image stored in the distorted circular image storage unit on a display;
On the distorted circular image displayed on the display, the positions of two points of the cut center point P and the auxiliary point Q are input based on a user instruction, and the cut center point P and the auxiliary point Q Is determined as a reference straight line J, and a numerical value given as m = k / d (k is a predetermined proportional constant) based on the distance d between the cut-out center point P and the auxiliary point Q is converted into a conversion magnification. an instruction input unit for recognizing m;
The pixel value of the pixel on the planar regular image arranged at the position indicated by the coordinates (u, v) is calculated using a conversion operation expression that associates the coordinates (u, v) with the coordinates (x, y). The reference is made around the cut-out center point P from the distorted circular image by determining based on the pixel value of the reference pixel on the distorted circular image arranged at the position indicated by the corresponding coordinates (x, y). For the partial image cut out in a direction corresponding to the straight line J, an operation for generating a planar regular image scaled based on the conversion magnification m is performed, and the generated planar regular image is stored in the planar regular image storage unit. A conversion operation unit;
An image conversion apparatus comprising: An image conversion apparatus that performs a process of cutting out a part of a distorted circular image obtained by shooting using a fisheye lens and converting it into a planar regular image,
A distorted circular image storage unit for storing a distorted circular image configured by an aggregate of a large number of pixels arranged at a position indicated by coordinates (x, y) on a two-dimensional XY orthogonal coordinate system;
A planar regular image storage unit for storing a planar regular image configured by an aggregate of a large number of pixels arranged at a position indicated by coordinates (u, v) on a two-dimensional UV orthogonal coordinate system;
A distorted circular image display unit for displaying the distorted circular image stored in the distorted circular image storage unit on a display;
On a predetermined input screen, an angle θ defined as an angle formed by the reference straight line J and the X axis and a conversion magnification m are input based on a user instruction, and the distorted circle displayed on the display On the image, an instruction input unit for inputting the position of the cutting center point P based on a user instruction;
The pixel value of the pixel on the planar regular image arranged at the position indicated by the coordinates (u, v) is calculated using a conversion operation expression that associates the coordinates (u, v) with the coordinates (x, y). The reference is made around the cut-out center point P from the distorted circular image by determining based on the pixel value of the reference pixel on the distorted circular image arranged at the position indicated by the corresponding coordinates (x, y). For the partial image cut out in a direction corresponding to the straight line J, an operation for generating a planar regular image scaled based on the conversion magnification m is performed, and the generated planar regular image is stored in the planar regular image storage unit. A conversion operation unit;
An image conversion apparatus comprising: An image conversion method in which a part of a distorted circular image obtained by shooting using a fisheye lens is cut out and converted into a planar regular image,
A distorted circular image having a radius R centered at the origin O of the XY coordinate system, which is composed of a collection of a large number of pixels arranged at positions indicated by coordinates (x, y) on a two-dimensional XY orthogonal coordinate system. Storing in the distorted circular image storage unit;
Displaying the distorted circular image stored in the distorted circular image storage unit on a display;
Inputting the position and orientation of the cutout center point P on the distorted circular image displayed on the display based on a user instruction;
In a three-dimensional XYZ Cartesian coordinate system including the two-dimensional XY Cartesian coordinate system, when a virtual spherical surface having a radius R with the origin O as the center is defined, a straight line passing through the cut-out center point P and parallel to the Z-axis Obtaining position coordinates (x ₀ , y ₀ , z ₀ ) of the intersection point G with the virtual sphere;
A vector U oriented in the U-axis direction of the two-dimensional UV orthogonal coordinate system to be defined on a tangent plane contacting the virtual sphere at the intersection G, and a vector X oriented in the X-axis direction of the two-dimensional XY orthogonal coordinate system Determining a plane inclination angle φ given as an angle formed by, based on the cutout direction;
The coordinate (u, v) and the coordinate (x, y) are obtained using an orthographic projection conversion equation including the position coordinates (x ₀ , y ₀ , z ₀ ) and the plane inclination angle φ as parameters. The pixel value of each pixel on the planar regular image configured by the association and a large number of pixels arranged at the position indicated by the coordinates (u, v) is indicated by the corresponding coordinates (x, y). By deciding on the basis of the pixel value of the reference pixel on the distorted circular image arranged at the position, it was cut out from the distorted circular image in the direction indicated by the plane inclination angle φ with the cut center point P as the center Performing an operation to generate a planar regular image for the partial image;
Is executed by a computer or an electronic circuit. The image conversion method according to claim 23, wherein
As a transformation formula of orthographic projection method,
x = R [(u−x ₀ ) A + (v−y ₀ ) B + (w−z ₀ ) E] /
√ ((u−x ₀ ) ² + (v−y ₀ ) ² + (w−z ₀ ) ² )
y = R [(u−x ₀ ) C + (v−y ₀ ) D + (w−z ₀ ) F] /
√ ((u−x ₀ ) ² + (v−y ₀ ) ² + (w−z ₀ ) ² )
here,
A = 1- (1-cosφ) (y ₀ ² + z ₀ ² )
B = −z ₀ sinφ + x ₀ y ₀ (1-cosφ)
C = z ₀ sinφ + x ₀ y ₀ (1-cosφ)
D = 1- (1-cosφ) (z ₀ ² + x ₀ ² )
E = y ₀ sinφ + z ₀ x ₀ (1-cosφ)
F = −x ₀ sinφ + y ₀ x ₀ (1-cosφ)
w = mR (where m is a predetermined conversion magnification)
An image conversion method characterized by using the following formula. The image conversion method according to claim 23 or 24, wherein:
On the distorted circular image displayed on the display, an instruction for designating the positions of two points of the cut center point P and the auxiliary point Q is input, and the cut center point P and the auxiliary point Q are input. The connecting straight line is a reference straight line J, and the plane inclination angle φ is determined on the basis of an angle θ formed by the reference straight line J and the X axis (however, θ = 0 ° when both are parallel). Image conversion method. The image conversion method according to claim 23 or 24, wherein:
On the distorted circular image displayed on the display, an instruction for designating the position of the cutting center point P is input, and on the predetermined input screen, the reference straight line J and the X axis on the distorted circular image are An image conversion method characterized in that the plane inclination angle φ is determined based on the angle θ.

Images