JP2005244861A - Imaging apparatus and imaging system parameter correction method - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To provide an imaging apparatus and method for correcting imaging system parameters in the same, in which at the same time unobstructed sky spherical surface images can be obtained. <P>SOLUTION: An imaging apparatus 3 comprises a pair of fish-eye lenses 10l, 10r of which surfaces 11l, 11r, to which external lights are incident are disposed so as to be positioned opposite to each other and in which the angle of view w is 180° or larger; an optical path change element 20 which is disposed between the pair of fish-eye lenses and changes optical paths of the external lights, passed through the fish-eye lenses to propagate the external lights to the same side; and an imaging means 30 capable of capturing object images 50l, 50r formed, based on the external lights of which the optical paths are changed by the optical path change element. In this imaging apparatus 3, a first imaging section 60l is constituted by including one of the pair of fish-eye lenses, the optical path change element and the imaging means, and a second imaging section 60r is constituted by including the other of the pair of fish-eye lenses, the optical path change element and the imaging means. <P>COPYRIGHT: (C)2005,JPO&NCIPI

Description

  The present invention relates to an imaging apparatus for acquiring an all-round spherical image and a method for calibrating imaging system parameters thereof.

As a method of capturing a 360-degree all-sky image, for example, there is a method using an imaging device described in Patent Document 1. The imaging device described in Patent Document 1 includes a fisheye lens having an angle of view of at least 90 degrees in each direction with respect to the optical axis in the entire visual field direction centering on the optical axis of the fisheye lens. In the method described in Patent Document 1, when the space is considered as one sphere, the imaging device is arranged at the center, and first, an image of a hemisphere in one direction among the spheres is acquired. Next, the imaging apparatus is rotated 180 degrees to acquire a hemispherical image in the opposite direction. Then, the images of the entire sky are formed by combining the images.
JP 2000-221391 A

  However, in the technique using the imaging device described in Patent Document 1, images taken at different times for each hemisphere are synthesized by rotating the imaging device. Therefore, it is not possible to obtain an image of the entire sky at the same time.

  SUMMARY OF THE INVENTION An object of the present invention is to provide an imaging apparatus that can obtain a spherical image of the entire sky at the same time and a calibration method for imaging system parameters in the imaging apparatus.

  In order to solve the above problems, an imaging apparatus according to the present invention is arranged such that surfaces on which external light is incident are located on opposite sides, a pair of fisheye lenses having an angle of view of 180 degrees or more, and a pair of An optical path changing element that is arranged between the fisheye lenses, changes the optical path of the external light passing through each fisheye lens, and propagates the external light to the same side, and the external light whose optical path is changed by the optical path changing element An imaging unit that can capture each of the formed subject images, and includes a first imaging unit including one of the pair of fisheye lenses, the optical path changing element, and the imaging unit, and the pair of fisheye lenses. The second image pickup unit is configured to include the other, the optical path changing element, and the image pickup means.

  In the imaging apparatus, external light from a subject located around the imaging apparatus is taken into the apparatus by each fisheye lens included in the first and second imaging units. The optical path of the external light taken into the imaging device through each fisheye lens is changed by the optical path changing element so as to propagate to the imaging means side. Then, both subject images formed by the external light are imaged by the imaging means.

  The imaging apparatus uses a pair of fisheye lenses having an angle of view of 180 degrees or more, and they are arranged as described above. Therefore, one fisheye lens captures external light from a subject located in approximately half of the 360 degrees around the imaging device, and external light from the subject located in the other half of the region from the other fisheye lens. It is captured. Accordingly, each of the subject images captured together by the first and second imaging units has image information covering 360 degrees around the imaging device. Therefore, it is possible to form a spherical image of the whole sky by using those subject images. In the imaging apparatus, since the subject images are captured by the above-described imaging means without moving the fisheye lens, a spherical image of the entire sky at the same time can be obtained.

  Moreover, it is preferable that the said imaging means in the imaging device which concerns on this invention is comprised from one imaging element. In this case, it is possible to simultaneously capture a subject image using a fisheye lens included in the first and second imaging units.

  Furthermore, the optical path changing element in the imaging apparatus according to the present invention is preferably a reflecting mirror having a reflecting surface on each fisheye lens side. In this case, the external light from each fisheye lens can be reflected by the reflecting surface on the fisheye lens side of the reflector.

  Furthermore, the imaging apparatus according to the present invention includes image forming means for forming a spherical image having image information on the entire sphere outside the system from each subject image captured by the imaging means. Is preferred. The spherical image can be obtained by this image forming means.

  In the imaging apparatus according to the present invention, the image forming unit converts each subject image captured by the imaging unit into the image center of the image plane of the fisheye lens included in the first and second imaging units and the focal length of the fisheye lens. Based on the image conversion unit that converts the image information of the subject image into a hemispherical image on the surface of the hemisphere, and the respective hemispherical images obtained in the image conversion unit are converted into relative orientations of the first and second imaging units. It is preferable to include an image forming unit that forms the spherical image by combining information and the image center of the image plane of each fisheye lens.

  As described above, the image information of each subject image acquired together by the first and second imaging units corresponds to image information for almost half of the entire sky outside the system (outside the imaging device). Therefore, each hemispherical image obtained by the image conversion unit has image information about every half of the whole sky. Therefore, it is possible to obtain the above-described spherical image by combining the hemispherical images in the image forming unit.

  Further, the image forming means in the image pickup apparatus according to the present invention includes (1) two sets of vanishing point pairs in a subject image of two sets of parallel lines extending in different directions for each subject image picked up by the image pickup means. And (2) two sets of vanishing point pairs calculated by the vanishing point calculating unit with respect to the image center of the image plane of the fisheye lens with respect to the fisheye lens of the first and second imaging units. And (3) two sets of vanishing point pairs calculated by the vanishing point calculating unit for the focal length of the fisheye lens with respect to the fisheye lens included in the first and second imaging units. A focal length calculation unit that calculates based on the distance between at least one of the vanishing point pairs, and (4) an object image that is converted based on the image center and focal length calculated by the image center calculation unit and the focal length calculation unit. Hemispherical image Relative posture information of the first and second imaging units is obtained based on the first vanishing point of the straight line group that is rare and parallel to the horizontal line in the object space, and the second vanishing point of the straight line group that is substantially orthogonal to the straight line group. And (5) focal length calculation based on the correlation of images in the area corresponding to the overlapping area of the field of view of each fisheye lens included in each hemispherical image for acquiring relative attitude information. A calibration unit that calibrates the relative orientation information acquired by the focal length and orientation information acquisition unit calculated by the unit, and (6) the image conversion unit is a subject image captured by the first and second imaging units. On the other hand, the subject image is converted into a hemispherical image based on the image center calculated by the image center calculation unit and the focal length calibrated by the calibration unit, and (7) the image forming unit is obtained by the image conversion unit. Each hemispherical image It is preferable to form the spherical image combined with on the basis of the calibrated relative orientation information in the image center and the calibration unit calculated by the section.

  In this case, two pairs of vanishing points in the subject image of the two sets of parallel lines in the object space are calculated by the vanishing point calculation unit, and based on the two pairs of vanishing points, the image plane of each fisheye lens is calculated. The image center is obtained by the image center calculation unit. Thereby, the image center of the image plane of each fisheye lens is calibrated.

  Further, the focal length calculation unit determines the focal length of each fisheye lens based on the distance between at least one of the two vanishing point pairs. Further, the first and second imaging are performed using the first vanishing point and the second vanishing point in the hemispherical image converted from the subject image using the image center and the focal length calculated as described above. The relative posture information of the part is acquired by the posture information acquisition unit. The relative attitude information and focal length are further calibrated by the calibration unit. For this reason, the focal length and the relative posture information are more accurate values.

  Then, the image conversion unit converts the subject image into a hemispherical image using the image center calculated by the image center calculation unit and the focal length formed by the calibration unit. Further, the hemispherical image is combined with the spherical image using the relative posture information calibrated by the calibrating unit and the image center. As described above, since the two hemispherical images obtained based on the calibrated image center and focal length are combined based on the image center and relative posture information, the image shift in the combined region Is reduced.

  The imaging system parameter calibration method according to the present invention includes (1) an imaging apparatus including the above-described pair of fish-eye lenses, an optical path changing element, and imaging means, and one of the pair of fish-eye lenses and the optical path changing. Images are taken by a first imaging unit including an element and the imaging unit, and a second imaging unit including the other of the pair of fisheye lenses, the optical path changing element, and the imaging unit. A vanishing point calculating step of calculating two vanishing point pairs in the subject image of two sets of parallel lines extending in different directions with respect to the subject image; and (2) first and second imaging units. An image center calculation step of calculating the image center of the image plane of the fisheye lens based on the two pairs of vanishing points calculated in the vanishing point calculation step, and (3) first and second Fisheye lens in the imaging unit On the other hand, a focal length calculation step for calculating a focal length of the fisheye lens based on a distance between at least one vanishing point pair of the two vanishing point pairs calculated in the vanishing point calculation step; An image for converting each subject image captured by the image information into a hemispherical image having the image information of the subject image on the surface of the hemisphere based on the image center and the focal length calculated in the image center calculating step and the focal length calculating step. A conversion step; and (5) a first vanishing point of a straight line group that is included in each hemispherical image obtained in the image conversion step and is substantially parallel to a horizontal line in the object space, and a first straight line group that is substantially orthogonal to the straight line group. (2) a posture information acquisition step for acquiring relative posture information of the first and second imaging units based on two vanishing points; and (6) each fisheye lens included in each hemispherical image for acquiring the relative posture information. Overlapping fields of view Characterized in that it comprises a calibration step of calibrating the relative position information acquired by the focal length and orientation obtaining step calculated by the focal length calculating step based on a correlation of the image in the corresponding region.

  In this imaging system parameter calibration method, the image center of the image plane of each fisheye lens is calculated based on the two pairs of vanishing points calculated in the vanishing point calculation step. Further, the focal length of each fisheye lens is calculated using at least one of the two sets of vanishing points.

  Next, based on the image center of the image plane of each fisheye lens calculated in this way and the focal length of each fisheye lens, each of the subject images imaged together by the first and second imaging units is converted into a hemispherical image. Convert. Since the fisheye lens has an angle of view of 180 degrees or more, the first vanishing point and the second vanishing point described above appear in each hemispherical image. In the posture information acquisition step, relative posture information of the first imaging unit and the second imaging unit is acquired using the first vanishing point and the second vanishing point.

  Further, since the angle of view of the fisheye lens is 180 degrees or more, there is an overlapping region in the field of view of each fisheye lens. Since the image characteristics of the regions in the respective hemispherical images corresponding to the overlapping regions substantially match, the focal length calculated in the focal length calculation step and the posture information acquisition step are obtained by taking the correlation of the images in that region. The relative posture information acquired in step 1 is further calibrated. The correlation between the images may be such that, for example, the luminance information matches.

  In the above calibration method, the image center is calibrated by calculating based on two pairs of vanishing points in the subject image of the two sets of parallel lines. Then, the focal length and the relative posture information obtained in the focal length calculation step and the posture information acquisition step are further calibrated in the calibration step. Therefore, the above-described calibration method makes the image center, focal length, and relative posture information, which are imaging system parameters of the imaging apparatus, more accurate.

  Therefore, as described above, when the spherical image is formed by combining each hemispherical image having image information of each subject image captured by the first and second imaging units, Image shift can be reduced.

  Further, the posture information acquisition step in the imaging system parameter calibration method according to the present invention includes the first erasure in the coordinate system of the first imaging unit from the position vector of the first vanishing point and the position vector of the second vanishing point in the reference coordinate system. A first matrix calculation step of calculating a first transformation matrix to be converted into a point position vector and a second vanishing point position vector, and a first vanishing point position vector and a second vanishing point position vector in the reference coordinate system; A second matrix calculating step of calculating a second transformation matrix to be converted into a position vector of the first vanishing point and a position vector of the second vanishing point in the coordinate system of the second imaging unit, and the first and second It is preferable to include a third matrix calculating step of calculating a third conversion matrix for converting from the coordinate system of the first imaging unit to the coordinate system of the second imaging unit based on the conversion matrix.

  Thereby, based on the first vanishing point and the second vanishing point of each hemispherical image, calculating a third conversion matrix for converting from the coordinate system of the first imaging unit to the coordinate system of the second imaging unit. Can do. This third transformation matrix corresponds to relative posture information between the first imaging unit and the second imaging unit.

  ADVANTAGE OF THE INVENTION According to this invention, the imaging device which can obtain the spherical image of the whole sky at the same time, and the calibration method of the imaging system parameter in the imaging device can be provided.

  Hereinafter, preferred embodiments of the present invention will be described with reference to the drawings. In the following description, the same reference numerals are used for the same elements, and redundant descriptions are omitted.

  First, a spherical image to be obtained with the imaging apparatus of the present embodiment will be described. In the present embodiment, a spherical image is a projection of image information of each subject located on the entire sky outside the system (around 360 degrees) on the surface of a sphere. A more specific description will be given with reference to FIG.

この点ｐ_ｉは点Ｐ_ｉが球体の表面へ投影された点に相当する。すなわち、点Ｐ_ｉからの光が中心Ｏに向かって進んでいるときに球体１の表面２と交わる点がｐ_ｉであり、点ｐ_ｉは点Ｐ_ｉからの光が担う画像を形成するために必要な情報（画像情報）を有している。したがって、球体１の表面２には、中心Ｏから見ることができる、系外の周囲３６０度に位置する被写体を投影することができる。このように画像情報を有している球体１の表面２が球面画像に相当する。 FIG. 1 is a diagram for explaining a spherical image. As shown in FIG. 1, a sphere 1 having a unit radius is considered in space, and an X axis, a Y axis, and a Z axis that are orthogonal to each other with the center O as an origin are set. A point on the surface 2 of the sphere connecting an arbitrary point P _i (i is an arbitrary integer) in space and the center O of the sphere is defined as _pi . When the elevation angle from the Z axis of the straight line connecting the point p _i and the origin O is θ _i and the azimuth angle from the X axis is φ _i , the position vector of the point p _i is expressed as follows.

This point p _i corresponds to a point where the point P _i is projected onto the surface of the sphere. That is, when the light from the point P _i is traveling toward the center O, the point that intersects the surface 2 of the sphere 1 is p _i , and the point p _i forms an image carried by the light from the point P _i. Necessary information (image information). Accordingly, a subject located at 360 degrees outside the system that can be seen from the center O can be projected onto the surface 2 of the sphere 1. Thus, the surface 2 of the sphere 1 having image information corresponds to a spherical image.

  The imaging apparatus of the present embodiment is characterized in that such a spherical image can be obtained by one shooting.

  The imaging device 3 of the present embodiment will be described with reference to FIGS. FIG. 2 is a schematic diagram illustrating a configuration of the imaging device 3 of the present embodiment. The imaging device 3 includes a pair of fish-eye lenses 10l and 10r, a right angle mirror (optical path changing element) 20, an imaging element (imaging means) 30, and an image forming means 40.

  The fish-eye lenses 10l and 10r are arranged so that the angle of view w is 185 degrees and the surfaces 11l and 11r on which external light is incident are located on opposite sides. In the present embodiment, the fish-eye lenses 10l and 10r do not necessarily mean one lens, but also include a lens system designed to have an angle of view w of 180 degrees or more and having fish-eye lens characteristics. . The fish-eye lenses 10l and 10r in FIG. 2 project a subject located in an almost half area of the whole sky to the image plane. In other words, the image information on the surface of the hemisphere of the sphere 1 in FIG. 1 is projected onto the image plane. The projection method of the fisheye lenses 10l and 10r is an equidistant projection method.

  The right angle mirror 20 is disposed between the pair of fisheye lenses 10l and 10r and on the optical axis of each fisheye lens 10l and 10r. The right angle mirror 20 has reflecting surfaces 21l and 21r on the fisheye lens 10l and 10r sides, respectively, and the reflecting surfaces 21l and 21r are orthogonal to each other. The right angle mirror 20 is disposed at an approximately equal distance from each of the fisheye lenses 10l and 10r. The external light from the subject incident from the fish-eye lenses 10l and 10r is propagated to the image sensor 30 side with its optical path bent at substantially right angles by the reflecting surfaces 21l and 21r. In other words, the right-angle mirror 20 is a reflector that is disposed between the pair of fisheye lenses 10l and 10r, changes the optical path of the external light that has passed through the fisheye lenses 10l and 10r, and propagates the external light to the same side. Functions as an optical path changing element.

  The imaging element 30 has an imaging plane (hereinafter, also simply referred to as “image plane”) in which the imaging element 30 has a light receiving surface 31 composed of a plurality of pixels via the right angle mirror 20 of each fisheye lens 10l, 10r. It is arranged to be located in. The imaging element 30 is, for example, a CCD (Charge Coupled Device) in which a plurality of pixels are arranged on a plane.

  FIG. 2B is a plan view of the light receiving surface 31 of the image sensor 30. The light receiving surface 31 has first and second light receiving regions 31l and 31r adjacent to each other. In FIG. 2B, the two-dot chain line is an imaginary line and has one light receiving surface 31 (that is, one image sensor 30). A subject image 50l by the fisheye lens 10l is formed in the first light receiving region 31l, and a subject image 50r by the fisheye lens 10r is formed in the second light receiving region 31r. The image sensor 30 simultaneously acquires subject images 50l and 50r formed by external light from the subject incident on the fisheye lenses 10l and 10r.

  In the above configuration, the imaging device 3 includes the first imaging unit 60l configured to include the fisheye lens 101, the right-angle mirror 20, and the imaging element 30. In addition, the imaging device 3 includes a second imaging unit 60 r configured to include the fish-eye lens 10 r, the right angle mirror 20, and the imaging element 30.

  Referring to FIG. 2A, the subject images 50 l and 50 r formed and acquired on the image sensor 30 are converted into electric signals for each pixel constituting the light receiving surface 31 and input to the image forming unit 40. The

FIG. 3 is a block diagram of the image forming unit 40. The image forming means 40 is a so-called computer having a CPU and the like, and forms a spherical image from a pair of subject images 50l and 50r acquired by the image sensor 30. The image forming unit 40 includes a parameter storage unit 41, an image conversion unit 42, and an image forming unit 43. The parameter storage unit 41 is the imaging system parameters of the imaging device 3, the position coordinates of the image centers O _Il and O _{Ir on} the image planes of the fish-eye lenses 10 l and 10 r, the focal lengths f _l and f _{r of} the fish-eye lenses 10 l and 10 _r , and the first The relative posture information of the first and second imaging units 60l and 60r is stored.

  The image conversion unit 42 converts the subject image 50l that is a planar image into a hemispherical image having image information of the subject image 50l on the surface of the hemisphere. The image conversion unit 42 converts the subject image 50r, which is a planar image, into a hemispherical image having the image information of the subject image 50r on the surface of the hemisphere.

The principle for converting a subject image into a hemispherical image will be described with reference to FIG. FIG. 4A is a diagram for explaining the imaging characteristics of the fisheye lens. FIG. 4A shows a state where the fisheye lens is located at the center O, and external light from a point P _i (corresponding to P _i in FIG. 1) in the field of view of the fisheye lens is incident on the fisheye lens. . Here, in order to simplify the explanation, the angle of view of the fisheye lens shown in FIG. 4 is 180 degrees.

ここで、ｆは、魚眼レンズの焦点距離である。（式２）より、仰角θ_ｉは、以下の式から算出される。

（式３）を（式１）に代入すると、以下の式が得られる。

図４に示す点ｑ_ｉは、図１の点Ｐ_ｉの平面上への投影点であって、図１において点Ｐ_ｉを球体１の表面２上に投影した点ｐ_ｉに対応している。ところで、点ｐ_iと点ｑ_iとで方位角φ_iは一致している。したがって、（式４）は、点ｑ_ｉの位置座標から点ｐ_iの位置座標を得ることができることを示している。 FIG. 4B is a diagram illustrating the image plane 50 of the fisheye lens. In the equidistant projection type fisheye lens, the point P _i shown in FIG. 4A is projected onto the point q _i represented by the position coordinates (r _i , φ _i ) on the image plane 50 in FIG. 4B. The That is, the point P _i does not change the azimuth angle φ _i and the distance r _i from the image center O _I is projected to the point q _i represented by the following expression.

Here, f is the focal length of the fisheye lens. From (Expression 2), the elevation angle θ _i is calculated from the following expression.

Substituting (Equation 3) into (Equation 1) yields the following equation:

A point q _i shown in FIG. 4 is a projection point on the plane of the point P _i in FIG. 1, and corresponds to a point p _i which is obtained by projecting the point P _i on the surface 2 of the sphere 1 in FIG. . By the way, the azimuth angle φ _i coincides with the point p _i and the point q _i . Therefore, (Expression 4) indicates that the position coordinates of the point p _i can be obtained from the position coordinates of the point q _i .

Here, the subject image obtained with the fisheye lens is a collection of points q _i . Therefore, the subject image can be converted into a hemispherical image by specifying the position coordinates of each pixel (that is, the point q _i ) constituting the subject image.

  Since the subject images 50l and 50r acquired by the image sensor 30 are converted into electric signals for each pixel as described above, the position of each pixel constituting the subject image is the pixel of the image sensor 30. Can be obtained in correspondence with the position of.

In the following description, the hemispherical images obtained from the subject images acquired by the first and second imaging units 60l and 60r are referred to as first and second hemispherical images, respectively. The first and second hemispherical images are represented by polar coordinate systems of the first and second imaging units 60l and 60r. At this time, the image centers O _Il and O _Ir correspond to the optical centers of the fish-eye lenses 10 l and 10 r.

Referring to FIG. 3, the image forming unit 43 uses the first and second hemispherical images obtained by the image converting unit 42 as the relative posture information of the first and second imaging units 60l and 60r, and the image center. A spherical image is formed by combining on the basis of O _Il and O _Ir . More specifically, the image centers O _Il and O _Ir of the first and second hemispherical images are matched, and the first and second hemispherical images are combined so as to be opposite to each other, thereby combining the spherical images. Form.

  As described above, the spherical image is formed using the imaging system parameters (image center, focal length, and relative posture information). Therefore, the imaging system parameters need to be calibrated in the imaging device 3.

  Referring to FIG. 3 again, the image forming unit 40 includes a vanishing point calculation unit 44, an image center calculation unit 45, a focal length calculation unit 46, and an attitude information acquisition unit 47 for calibrating the imaging system parameters. And a calibration unit 48.

  The vanishing point calculation unit 44 calculates two sets of vanishing point pairs in the subject image of two sets of parallel lines extending in different directions in the object space, based on the subject images 50l and 50r captured by the image sensor 30. . The two sets of parallel line groups mean straight line groups substantially parallel to two straight lines that are oblique to each other. The vanishing point is a point where a plurality of parallel straight lines included in the parallel line group intersect at infinity. Since the fisheye lenses 10l and 10r have an angle of view w of 185 degrees, vanishing points appear in the subject images 50l and 50r. Since the straight line extends on both sides with respect to a certain point, a pair of vanishing points (that is, a vanishing point pair) appears in each of the subject images 50l and 50r.

  The vanishing point calculation unit 44 may calculate the position coordinates of the two pairs of vanishing points from the subject images of the two sets of parallel lines that are oblique to each other. Further, the vanishing point calculation unit 44 calculates one vanishing point pair from the subject image of one parallel line group (first parallel line group), and extends in a direction different from the straight line constituting the first parallel line group. Another vanishing point pair may be calculated from a subject image of a parallel line group (second parallel line group) formed of straight lines.

The principle of vanishing point calculation will be described. The straight line in the object space becomes a conic curve in the image plane of the fisheye lens (in the subject image). Therefore, first, in a subject image, a plurality of conic curve functions f _j (x, y) are calculated by fitting a curve corresponding to a straight line included in the parallel line group with a conic curve. In the case of fitting, a least square method is used to reduce errors due to fitting.

Since the vanishing point appears in the subject image (located on the plane), f _j (x _v , y _v ) = 0 is obtained when the position coordinates of the vanishing point are (x _v , y _v ). To establish. Further, theoretically, since the conic curves intersect each other at the vanishing point, the position coordinates (x _v , y _v ) of the vanishing point can be calculated by applying the least square method again.

  The vanishing point calculation unit 44 calculates two sets of vanishing point pairs in the subject image 50l and calculates two sets of vanishing point pairs in the subject image 50r.

The image center calculating unit 45 acquires the position coordinates of the two vanishing point pairs calculated by the vanishing point calculating unit 44 from the vanishing point calculating unit 44, and uses the fish-eye lens as an intersection of two straight lines connecting each vanishing point pair. The image center O _I in the image plane is calculated.

The image center calculation unit 45 calculates the image centers O _Il and O _Ir for the image planes of the fisheye lenses 10 l and 10 r included in the first and second imaging units 60 l and 60 r. The image center calculation unit 45 stores the position coordinates of the image centers O _Il and O _{Ir in} the parameter storage unit 41.

ここで、ｒ_πは、１組の消失点間の距離である。 The focal length calculation unit 46 acquires the position coordinates of the two pairs of vanishing points calculated by the vanishing point calculation unit 44 from the vanishing point calculation unit 44, and calculates the focal length f of the fisheye lens from the following equation.

Here, r _π is the distance between a set of vanishing points.

Focal length calculating unit 46, the fish-eye lens 10l, using at least one of the corresponding two sets of vanishing point pairs to calculate the focal length f _l. Focal length calculating unit 46, for the fish-eye lens 10r, it calculates a focal length f _r by using at least one of the corresponding two sets of vanishing point pairs. The focal length calculation unit 46 stores these focal lengths f _l and _fr in the parameter storage unit 41.

  The posture information acquisition unit 47 acquires the relative posture information of the first and second imaging units 60l and 60r. In the present embodiment, the relative posture information means a rotation matrix (third conversion matrix) from the polar coordinate system of the first imaging unit 60l to the polar coordinate system of the second imaging unit 60r.

  The principle of acquiring a rotation matrix (relative attitude information) from the polar coordinate system of the first imaging unit 60l to the polar coordinate system of the second imaging unit 60r will be described. The posture information acquisition unit 47 acquires the relative posture information using the vanishing points appearing in the first and second hemispherical images because the angle of view w of the fisheye lenses 10l and 10r is 185 degrees. . More specific description will be given below.

  The first and second hemispherical images converted from the subject images 50l and 50r obtained by the first and second imaging units 60l and 60r are the polar coordinate systems of the first and second imaging units 60l and 60r. It is expressed using. Usually, each of the first and second hemispherical images includes an image of a straight line group substantially parallel to the horizontal line in the object space, its vanishing point (hereinafter referred to as “first vanishing point”), and a direction orthogonal to the horizontal line. An image of a straight line group substantially parallel to a straight line extending in the (vertical direction) and its vanishing point (hereinafter referred to as “second vanishing point”) appear.

Ｒ１_ｃｗは上述したように回転行列であって、３つの変数が含まれているが、（式６）及び（式７）を用いることで、６個の方程式が成り立つため、Ｒ１_ｃｗを算出可能である。 Here, the position vector of the first vanishing point in the polar coordinate system of the first imaging unit 60l is p1 _hc, and the position vector of the second vanishing point is p1 _vc . Then, the position vector of the first vanishing point in the world coordinate is a three-dimensional coordinate system (reference coordinate) and p _hw, the position vector of the second vanishing point and p _vw. In this case, if the rotation matrix (first transformation matrix) from the world coordinate system to the polar coordinate system of the first imaging unit 60l is R1 _cw , the following equation is established.

R1 _cw is a rotation matrix as described above, and includes three variables. By using (Equation 6) and (Equation 7), six equations can be established, so that R1 _cw can be calculated. It is.

（式８）及び（式９）よりＲ２_ｃｗが算出できる。 By the way, when one spherical image is formed by combining the first and second hemispherical images, the first vanishing point of the first hemispherical image and the first vanishing point of the second hemispherical image are. The second vanishing point of the first hemispherical image coincides with the second vanishing point of the second hemispherical image. Thus, _{p hw} and _{p vw} in Equation (6) and (7) can also be converted into a first vanishing point _{p2 hc} and second vanishing _{p2 vc} of the second hemisphere image. That is, when a transformation matrix (second transformation matrix) from the world coordinate system to the polar coordinate system of the second imaging unit 60r is R2 _cw , the following equation is established.

R2 _cw can be calculated from (Equation 8) and (Equation 9).

Then, from Expression (6) and Expression (8), or (Expression 7) and (Expression 9), based on R1 _cw and R2 _cw , the polar coordinate system of the first imaging unit 60l is used to change the second imaging unit 60r. A rotation matrix R to the polar coordinate system can be calculated.

The posture information acquisition unit 47 calculates R1 _cw from (Equation 6) and (Equation 7) described above, calculates R2 _cw from (Equation 8) and (Equation 9), and further calculates a rotation matrix R, Store in the parameter storage unit 41.

The first and second hemispherical images used to acquire the transformation matrix R are the image centers O _Il and O _Ir calculated by the image center calculation unit 45 and the focal length f calculated by the focal length calculation unit 46. _l, a first and second hemisphere image converted object image 50 l, from 50r based on _{f r.} The first and second hemispherical images may be formed by the image conversion unit 42 according to instructions from the posture information acquisition unit 47. When the calibration is performed, if the image converting unit 42 forms the first and second hemispherical images, the image forming unit 43 does not acquire the image data, and the posture information acquiring unit 47 You may make it acquire.

As the parallel line group for obtaining the first vanishing points p1 _hc and p2 _hc and the second vanishing points p1 _vc and p2 _vc , for example, artifacts (buildings, roads, etc.) included in the subject images 50l and 50r are used. ) Can be used.

The calibration unit 48 calibrates the focal lengths f ₁ and f _{r of} the fisheye lenses 10 l and 10 _r and the relative posture information (rotation matrix R). The principle by which the calibration unit 48 calibrates the focal lengths f ₁ and f _{r of} the fisheye lenses 10l and 10r and the relative posture information will be described.

FIG. 5 is a diagram illustrating a region where the imaging apparatus 3 can capture images. The shaded area in FIG. 5 indicates the field of view of the fisheye lenses 10l and 10r. Since the angle of view w of the fish-eye lenses 10l and 10r is 185 degrees, there is an overlapping region α in their field of view. Calibration unit 48, a point p _l in the region corresponding to the overlapping area α included in the first and second hemispherical surface _image, extracts a p _r. In other words, extraction first and point p _l in hemisphere image corresponding to the object point in the overlapping area α in the object space, and a point p _r in the second hemisphere image corresponding to the same object point To do. The first and second hemispherical images used here are images used to acquire the rotation matrix R.

実際には、撮像系パラメータの誤差により（式１０）は近似的に成立する。したがって、次の目的関数Ｆ（the objective function）が最小になるような焦点距離ｆ_ｌ，ｆ_ｒ及び回転行列Ｒを求めることで焦点距離ｆ_ｌ，ｆ_ｒ及び相対姿勢情報を校正することが可能である。言い換えれば、重複領域αに対応する第１及び第２の半球面画像の領域の画像の相関に基づいて焦点距離算出部４６で算出された焦点距離ｆ_ｌ，ｆ_ｒ及び回転行列Ｒを校正することが可能である。なお、ここでは、画像中心Ｏ_Ｉｌ，Ｏ_Ｉｒは、上述したように２本の直線の交点として求められているため、より正確な値になっているとしている。

Ｎは、物体空間における重複領域αに対応する第１及び第２の半球面画像内の領域に含まれる画素の数である。（式１１）において、右辺の和（式中のΣ）は、重複領域αに対応する第１の半球面画像（又は第２の半球面画像）内の領域に含まれる複数（Ｎ個）のｐ_ｌ（又はｐ_ｒ）に対する和である。またＬ_ｌａｖ及びＬ_ｒａｖは、第１及び第２の半球面画像において、重複領域αに対応する領域に含まれる複数の画素の輝度の平均値である。（式１１）を適用するため、校正部４８は、Ｌ_ｌａｖ及びＬ_ｒａｖを算出する機能も有している。 Since the points p ₁ and p _r are image points corresponding to object points included in the overlapping region α and included in the first and second hemispherical images, the image information (for example, luminance) Match. Therefore, if the luminances of the images at the points p ₁ and p _r are L ₁ (p ₁ ) and L _r (p _r ), the following equation holds theoretically.

Actually, (Equation 10) is approximately established by the error of the imaging system parameter. Therefore, it is possible to calibrate the focal lengths f _l and _fr and the relative posture information by obtaining the focal lengths f _l and _fr and the rotation matrix R that minimize the next objective function F (the objective function). It is. In other words, the focal lengths f _l and _fr and the rotation matrix R calculated by the focal length calculation unit 46 are calibrated based on the correlation between the images of the first and second hemispherical image regions corresponding to the overlapping region α. It is possible. Here, the image centers O _Il and O _Ir are obtained as intersections of two straight lines as described above, and are therefore more accurate values.

N is the number of pixels included in the regions in the first and second hemispherical images corresponding to the overlapping region α in the object space. In (Expression 11), the sum (Σ in the expression) of the right side is a plurality of (N) included in the region in the first hemispherical image (or the second hemispherical image) corresponding to the overlapping region α. It is the sum for p _l (or p _r ). L _lav and L _rav are average values of luminance of a plurality of pixels included in the region corresponding to the overlapping region α in the first and second hemispherical images. In order to apply (Formula 11), the calibration unit 48 also has a function of calculating L _lav and L _rav .

  Next, a method for forming an all-round spherical image by the imaging device 3 will be described. In order to form an all-round spherical image, first, a method for calibrating the imaging system parameters of the imaging device 3 will be described.

  FIG. 6 is a flowchart of the imaging system parameter calibration method. Note that it is assumed that the imaging system parameters are not stored in the parameter storage unit 41 before the vanishing point calculation step S10 illustrated in FIG. 6 is performed.

(Vanishing point calculation step S10)
7-9 is a figure which shows the calculation process of a pair of vanishing point. First, the grid pattern 70 in which the black areas and the white areas shown in FIG. In the lattice pattern 70, since the boundary between the black region and the white region is a straight line parallel to each other, a plurality of boundary lines 71 _{1 to} 71 ₆ are parallel line groups (first parallel line group) 71 extending in a certain direction. It corresponds to. FIG. 8 is a schematic diagram of the subject image 72 (50 l) of the lattice pattern 70. As shown in FIG. 8, the subject image acquired by the fisheye lens 101 is circular (spherical image).

When the subject image 72 is acquired by the image sensor 30 (see FIG. 2), the vanishing point calculation unit 44 (see FIG. 3) fits a conic curve to the curve of the subject image 72 shown in FIG. Accordingly, the vanishing point calculation unit 44 calculates a plurality of conic curve functions f _j (x, y) (for example, f ₁ , f _2, etc.) shown in FIG. Subsequently, vanishing point pairs 73l and 74l are calculated based on the principle described in the description of the vanishing point calculation unit 44 using the conic curve function f _j (x, y).

Next, the same lattice pattern 70 is rotated by, for example, approximately 90 degrees and imaged again by the first imaging unit 60l. The vanishing point calculation unit 44 calculates the vanishing point pair 75l and 76l shown in FIG. 9B in the same manner as when the vanishing point pair 73l and 74l are calculated. In this case, since the lattice pattern 70 is rotating, the parallel line group (second parallel line group) different from the direction in which the straight lines (boundary lines 71 _{1 to} 71 ₆ in FIG. 7) extending in the _first parallel line group extend. The vanishing point pair 75l and 76l corresponding to () is calculated. The above process is similarly performed on the second imaging unit 60r to calculate the vanishing point pair 73r and 74r corresponding to the vanishing point pair 73l and 74l, and the vanishing point pair corresponding to the vanishing point pair 75l and 76l. 75r and 76r are calculated.

(Image Center Calculation Step S11)
10 and 11 are diagrams illustrating a process of calculating the image center. The image center calculation unit 45 acquires the position coordinates of the vanishing points 73l, 74l, 75l, and 76l from the vanishing point calculation unit 44.

As shown in FIGS. 10A and 10B, the image center calculation unit 45 calculates a straight line 77 connecting the vanishing point pairs 73l and 74l and a straight line 78 connecting the vanishing point pairs 75l and 76l. Then, the intersection of the straight line 77 and the straight line 78 is specified as the image center O _Il of the image plane of the fisheye lens 10l. FIG. 11 is a diagram showing the calculated image center O _Il superimposed on the subject image 72 of FIG.

Although the first imaging unit 60l has been described above, the image center calculation unit 45 calculates the position coordinates of the image center O _{Ir with} respect to the image plane of the fisheye lens 10r. The image center calculation unit 45 stores the acquired position coordinates of the image centers O _Il and O _{Ir in} the parameter storage unit 41.

(Focal distance calculation step S12)
Further, the focal length calculation unit 46 obtains the position coordinates of the vanishing point pair 73l, 74l (or vanishing point pair 75l, 76l) and the vanishing point pair 73r, 74r (or the vanishing point pair 75r, 76r) from the vanishing point calculation unit 44. get. Focal length calculating unit 46, vanishing point pairs 73l, 74l (or vanishing point pairs 75l, 76l) to calculate the focal length _{f l} fisheye lens and applied to the distance between the (Equation 5) 10l.

In the above has described the first imaging unit 60l, a focal length calculating unit 46, a fisheye lens focal length _{f r} also vanishing point pairs 10r 73r, 74r (or, vanishing point pairs 75 r, 76r) distance between To calculate from (Equation 5). The focal length calculation unit 46 stores the calculated focal lengths f ₁ and f _r in the parameter storage unit 41.

  In the present embodiment, as shown in FIG. 6, the focal length calculation step S12 is performed after the image center calculation step S11. However, the order is not particularly limited, and the focal length calculation step S12 is performed first. May be implemented.

(Image conversion step S13)
Next, the scenery (subject) around the imaging device 3 is imaged except for the lattice pattern 70. In this case, the image conversion unit 42 converts the subject images 50l and 50r captured by the image sensor 30 into the image centers O _Il and O _Ir and the focal length f calculated in the image center calculation step S11 and the focal length calculation step S12. _l, based on the f _r, is converted into the first and second hemisphere image. Note that this image conversion step S13 may be performed based on an instruction from the posture information acquisition unit 47 to the image conversion unit 42.

(Attitude information acquisition process S14)
Next, the posture information acquisition unit 47 extracts the first vanishing point p1 _hc and the second vanishing point p1 _vc described above from the first hemispherical image obtained in the image conversion step S13. In addition, the posture information acquisition unit 47 extracts the first vanishing point p2 _hc and the second vanishing point p2 _vc from the obtained second hemispherical image. As described above, the first vanishing points p1 _hc and p2 _hc are vanishing points in the first and second hemispherical images of a group of straight lines substantially parallel to the horizontal line in the object space. The second vanishing points p1 _vc and p2 _vc are vanishing points in the first and second hemispherical images of a straight line group substantially parallel to a straight line extending in a direction (vertical direction) perpendicular to the horizontal line in the object space. It is.

Next, the posture information acquisition unit 47 calculates the rotation matrix R1 _cw from the world coordinate system to the polar coordinate system of the first imaging unit 60l using (Expression 6) and (Expression 7) (first matrix calculation). Process). Further, the posture information acquisition unit 47 calculates the rotation matrix R2 _cw from the world coordinate system to the polar coordinate system of the second imaging unit 60r using (Expression 8) and (Expression 9) (second matrix calculation step) ). Further, the posture information acquisition unit 47 calculates a transformation matrix R from the polar coordinate system of the first imaging unit 60l to the polar coordinate system of the second imaging unit 60r using the rotation matrices R1 _cw and R2 _cw (third Matrix calculation step). The posture information acquisition unit 47 stores the transformation matrix R in the parameter storage unit 41.

(Calibration step S15)
When the relative posture information is acquired in the posture information acquisition step S14, the calibration unit 48 uses the overlapping region α of the visual field of the fisheye lenses 10l and 10r in the first and second hemispherical images converted in the image conversion step S11. point p _l in the corresponding region (see FIG. 5), extracts a _{p r.} Then, the point p _l, based on the correlation of the image at p _r, more specifically on the basis of (Equation 11), the focal length calculated second process S12 focal length f _l calculated _in, f _r and orientation obtaining step The relative posture information acquired in S14 (that is, the focal lengths f ₁ and _fr and the relative posture information stored in the parameter storage unit 41) is calibrated.

  Through the above steps, the imaging system parameters of the imaging device 3 are calibrated. Once this imaging system parameter is calibrated once by the above method, it can be used to form a spherical image.

  Next, a spherical image forming method will be described. FIG. 12 is a flowchart for forming a spherical image.

(Imaging process S20)
First, the image capturing apparatus 3 is installed at a position where a landscape to be imaged can be captured, and the first and second image capturing units 60l and 60r capture the scenery around the image capturing apparatus 3.

(Image conversion step S21)
When the image sensor 30 acquires a pair of subject images 50l and 50r, the image sensor 30 converts them into electric signals and inputs them to the image converter 42. The image conversion unit 42 converts the pair of subject images 50l and 50r into first and second hemispherical images. At this time, the image converter 42, image center O _Il stored calibrated in the parameter storage unit 41 by the above calibration _method, O _Ir and focal length f _l, the first and second hemisphere on the basis of f _r Form an image.

(Image forming step S22)
Next, the image forming unit 43 converts the first and second hemispherical images formed by the image converting unit 42 into the image centers O _Il and O _Ir stored in the parameter storage unit 41 and relative posture information (that is, , Rotation matrix R) to form a spherical image of the whole sky.

  In the description of the calibration method, as described above, the imaging system parameters are not stored in the parameter storage unit 41 before the vanishing point calculation step S10 (see FIG. 6) is performed. A design value obtained when designing may be stored. In this case, the parameters stored for each process are calibrated.

  Next, the effects of the imaging system parameter calibration method of the present embodiment described above with reference to the examples will be described more specifically. FIG. 13 is a diagram showing a camera system to which the calibration method is actually applied. The camera system of FIG. 13 is a model of the imaging device 3 shown in FIG. In the camera system of FIG. 13, cameras having fisheye lenses with an angle of view w of 185 degrees are arranged in opposite directions. As the fisheye lens, FCON-02 (manufactured by Olympus Corporation) was used.

FIG. 14 is a schematic diagram showing a shooting direction by the camera system shown in FIG. The camera system is arranged at the center O _c of the circle shown in FIG. 14. In this case, the pair of cameras captures the opposite directions (A direction and B direction in the drawing).

  FIG. 15 is a diagram of a subject image photographed by the camera system. Of the pair of images in the figure, for example, the left figure is an image in the A direction in FIG. 14, and the right figure is an image in the B direction. The pair of images shown in FIG. 15 are obtained by one shooting.

  FIG. 16 is a diagram of a part of a spherical image formed without configuring imaging system parameters for comparison. FIG. 16A is an image in the C direction in FIG. 14 among the formed spherical images, and FIG. 16B is an image in the D direction in FIG. Further, the enlarged view on the right side in the figure is an enlarged view of the combined region of the first and second hemispherical images constituting the spherical image. As shown in FIG. 16, it can be seen that there is a shift in the image in the combined area.

  FIG. 17 is a partial view of a spherical image formed after performing the above-described calibration method. FIGS. 17A and 17B are images in the C direction and D direction of FIG. 14 in the spherical image, similarly to FIGS. 16A and 16B. When the calibration method is performed, the image shift in the combined area is reduced as shown in FIG.

  In the imaging apparatus 3 according to the present embodiment described above, the fisheye lenses 10l and 10r having an angle of view w of 185 degrees are arranged so as to photograph subjects on opposite sides. Therefore, the fish-eye lens 10l captures external light from the subject located in approximately half of the 360 degrees around the imaging device 3, and captures external light from the fish-eye lens 10r from the subject located in the remaining almost half region. It is. Therefore, the subject images 50l and 50r imaged together by the first and second imaging units 60l and 60r have image information covering 360 degrees around the imaging device 3 together. Therefore, a spherical image of the whole sky at the same time can be obtained by converting each of the subject images 50l and 50r into a hemispherical image and combining them. In the imaging device 3, this spherical image is formed by the image forming means 40.

  In the process of forming a spherical image from the subject images 50l and 50r, the imaging system parameters of the imaging device 3 are used. However, since the imaging system parameters can be calibrated by the above-described imaging system parameter calibration method of the present embodiment, When a spherical image is formed, image shift is reduced in the spherical image.

In particular, in the calibration method of the present embodiment, since the fish-eye lenses 10l and 10r having an angle of view w of 185 degrees are used, vanishing points appearing in the subject images 50l and 50r (in the first and second hemispherical images) are detected. It is used and proofread. More specifically, since the image center O _Il (or image center O _Ir ) is calculated using two sets of vanishing point pairs, the image centers O _Il and O _Ir are more accurate. Further, after calculating the focal length f ₁ (or the focal length f _r ) using a pair of vanishing point pairs, the relative posture information (rotation matrix R) is calculated. Then, the focal lengths f _l and _fr and the rotation matrix R are further calibrated using the overlapping region α of the visual field of the fish-eye lenses 10 _l and 10 _r . Therefore, the imaging system parameter has a more accurate value, and as shown in FIG. 17, it is possible to form a spherical image with little image shift in the combined region.

  As mentioned above, although preferred embodiment of this invention was described, it cannot be overemphasized that this invention is not limited to the said embodiment. For example, the imaging device 3 may not include the image forming unit 40. In this case, a spherical image may be formed by transferring the electrical signal from the imaging device to a computer or the like located outside the imaging device 3 and performing the processing performed by the image forming means 40.

  Further, although the optical path changing element is a right-angle mirror, it is sufficient that the light from the fisheye lenses 10l and 10r can be reflected to the image sensor side. For example, a reflecting mirror 90 as shown in FIG. 18 may be used as the optical path changing element. In the reflecting mirror 90, the angle Q formed by the two reflecting surfaces 91l and 91r is 60 degrees. In this case, the light receiving surface 31 of the image sensor 30 can be made small.

Furthermore, in the vanishing point calculating step S10, the lattice pattern 70 as shown in FIG. 7 is used as the calibration pattern, but for example, a calibration pattern 80 as shown in FIG. 19 may be used. The calibration pattern 80 includes a plurality of straight lines 81 _{1 to} 81 ₆ that are parallel to each other and a plurality of straight lines 82 _{1 to} 82 ₆ that are orthogonal to each other. Incidentally, the straight line ₈₁ 1 to 81 ₆ and the straight line ₈₂ 1 to 82 ₆ need only obliquely intersects also not necessarily be orthogonal. Further, a parallel line group constituting the first vanishing point p1 _hc (p2 _hc ) and the second vanishing point p1 _vc (p2 _vc ) may be used without using a calibration pattern such as the lattice pattern 70.

  Furthermore, the angle of view w of the fisheye lenses 10l and 10r is 185 degrees, but may be 180 degrees or more. Furthermore, although the projection system of the fish-eye lenses 10l and 10r is an equidistant projection system, it is not limited to this. Further, the image forming unit 40 includes the imaging system parameter storage unit 41. However, the image forming unit 40 may not include the imaging system parameter storage unit 41, and may use a memory included in a so-called computer. Furthermore, although the imaging means is one imaging element 30, for example, two imaging elements may be used. In this case, the fish-eye lenses 10l and 10r may be arranged so that the light-receiving surfaces of the imaging elements are positioned on the image formation planes through the optical path changing elements.

  The imaging apparatus and imaging system parameter calibration method according to the present invention can be used for monitoring sensors, visual sensors of self-propelled robots, and the like.

It is a figure explaining a spherical image It is a schematic diagram which shows the structure of the imaging device 3 of this embodiment. 2 is a block diagram of an image forming unit 40. FIG. It is a figure for demonstrating the principle which converts a to-be-photographed image into a hemispherical image. FIG. 3 is a schematic diagram illustrating a region where an image can be captured by the imaging apparatus 3; It is a flowchart of the calibration method of an imaging system parameter. It is a figure which shows the calculation process of a pair of vanishing point. It is a figure which shows the calculation process of a pair of vanishing point. It is a figure which shows the calculation process of a pair of vanishing point. It is a figure which shows the process of calculating an image center. It is a figure which shows the process of calculating an image center. It is a flowchart for forming a spherical image. It is a figure of the photograph of the camera system to which the imaging system parameter calibration method of this embodiment is applied. It is a figure which shows the imaging | photography direction of the camera system of FIG. It is the figure of the to-be-photographed object image imaged at once with the camera system shown in FIG. It is a figure of a part of spherical image for a comparison. It is a figure of a part of spherical image formed by applying the imaging system parameter calibration method concerning this embodiment. It is a figure which shows the other form of an optical path changing element. It is a figure which shows the other example of the test pattern used at a vanishing point calculation process.

Explanation of symbols

  DESCRIPTION OF SYMBOLS 3 ... Imaging device, 10l, 10r ... Fisheye lens, 11l, 11r ... Fisheye lens surface, 20 ... Right angle mirror (optical path changing element), 30 ... Imaging element (imaging means) 40 ... Image forming means, 41 ... Parameter storage part, 42 Image conversion unit 43 Image formation unit 44 Vanishing point calculation unit 45 Image center calculation unit 46 Focal length calculation unit 47 Posture information acquisition unit 48 Calibration unit 60l First imaging Part, 60r ... 2nd imaging part.

Claims (8)

A pair of fish-eye lenses that are arranged such that surfaces on which external light is incident are located on opposite sides of each other, and an angle of view of 180 degrees or more;
An optical path changing element that is disposed between the pair of fisheye lenses, changes the optical path of the external light passing through each fisheye lens, and propagates the external light to the same side;
An imaging means capable of capturing both subject images formed based on external light whose optical path has been changed by the optical path changing element;
The first imaging unit is configured to include one of the pair of fisheye lenses, the optical path changing element, and the imaging means, and includes the other of the pair of fisheye lenses, the optical path changing element, and the imaging means. An image pickup apparatus comprising two image pickup units.   The imaging apparatus according to claim 1, wherein the imaging unit includes a single imaging device.   The imaging apparatus according to claim 1, wherein the optical path changing element is a reflecting mirror having a reflecting surface on each fisheye lens side.   The image forming means for forming a spherical image having image information on the entire surface of the sphere outside the system from each subject image picked up by the image pickup means. The imaging device according to any one of the above. The image forming unit includes:
Based on the image center of the image plane of the fisheye lens included in the first and second imaging units and the focal length of the fisheye lens, each subject image captured by the imaging unit is converted into hemispherical image information. An image conversion unit for converting to a hemispherical image on the surface;
The spherical images are formed by combining the hemispherical images obtained in the image converting unit based on the relative posture information of the first and second imaging units and the image center of the image plane of each fisheye lens. The imaging apparatus according to claim 4, further comprising an image forming unit. The image forming unit includes:
For each subject image captured by the imaging means, a vanishing point calculation unit that calculates two pairs of vanishing points in the subject image of two sets of parallel lines extending in different directions;
An image center calculation unit that calculates an image center of the image plane of the fisheye lens based on the two pairs of vanishing points calculated by the vanishing point calculation unit with respect to the fisheye lens included in the first and second imaging units; ,
Based on the distance between at least one vanishing point pair of the two vanishing point pairs calculated by the vanishing point calculation unit, the focal length of the fisheye lens with respect to the fisheye lens of the first and second imaging units is determined. A focal length calculation unit for calculating
A first group of straight lines included in the hemispherical image converted from the subject image based on the image center and focal length calculated by the image center calculating unit and the focal length calculating unit and substantially parallel to a horizontal line in the object space. A posture information acquisition unit that acquires relative posture information of the first and second imaging units based on a vanishing point and a second vanishing point of a straight line group that is substantially orthogonal to the straight line group;
The focal length calculated by the focal length calculation unit based on the correlation of the images in the region corresponding to the overlapping region of the visual field of each fisheye lens included in each hemispherical image for acquiring the relative posture information A calibration unit that calibrates the relative posture information acquired by the posture information acquisition unit,
The image conversion unit is configured based on the image center calculated by the image center calculation unit and the focal length calibrated by the calibration unit with respect to the subject image captured by the first and second imaging units. Convert the subject image into a hemispherical image,
The image forming unit combines the hemispherical images obtained by the image conversion unit based on the image center calculated by the image center calculation unit and the relative posture information calibrated by the calibration unit, and The imaging apparatus according to claim 5, wherein an image is formed. The surfaces on which the external light is incident are disposed on opposite sides of each other, and are disposed between a pair of fisheye lenses having an angle of view of 180 degrees or more and the pair of fisheye lenses, and the external light passing through each fisheye lens An optical path changing element that changes the optical path and propagates the external light to the same side, and an imaging unit that can capture both subject images formed based on the external light whose optical path is changed by the optical path changing element A first imaging unit configured to include one of the pair of fisheye lenses, the optical path changing element, and the imaging means, the other of the pair of fisheye lenses, the optical path changing element, and the Two sets of vanishing point pairs in the subject image of two sets of parallel lines extending in different directions with respect to the subject image captured by each of the second imaging units including the imaging means, Calculate And the vanishing point calculation step,
An image center calculation step of calculating an image center of the image plane of the fisheye lens based on the two pairs of vanishing points calculated in the vanishing point calculation step with respect to the fisheye lens included in the first and second imaging units; ,
Based on the distance between at least one vanishing point pair of the two vanishing point pairs calculated in the vanishing point calculation step, the focal length of the fisheye lens with respect to the fisheye lens included in the first and second imaging units is determined. A focal length calculation step to calculate,
A hemispherical surface having image information of the subject image on the surface of the hemisphere based on the image center and focal length calculated in the image center calculating step and the focal length calculating step. An image conversion process for converting to an image;
Based on a first vanishing point of a straight line group that is included in each hemispherical image obtained in the image conversion step and is substantially parallel to a horizontal line in the object space, and a second vanishing point of a straight line group that is substantially orthogonal to the straight line group. A posture information acquisition step of acquiring relative posture information of the first and second imaging units;
The focal length calculated in the focal length calculation step based on the correlation of the images in the area corresponding to the overlapping area of the visual field of each fisheye lens, which is included in each hemispherical image for acquiring the relative posture information, and the An imaging system parameter calibration method comprising: a calibration step of calibrating the relative posture information acquired in the posture information acquisition step. The posture information acquisition step includes
The position vector of the first vanishing point and the position vector of the second vanishing point in the reference coordinate system are converted into the position vector of the first vanishing point and the position vector of the second vanishing point in the coordinate system of the first imaging unit. A first matrix calculating step of calculating a first transformation matrix;
From the position vector of the first vanishing point and the position vector of the second vanishing point in the reference coordinate system, the position vector of the first vanishing point and the position vector of the second vanishing point in the coordinate system of the second imaging unit A second matrix calculating step of calculating a second conversion matrix to be converted into
A third matrix calculating step of calculating a third conversion matrix for converting from the coordinate system of the first imaging unit to the coordinate system of the second imaging unit based on the first and second conversion matrices; The imaging system parameter calibration method according to claim 7, comprising: an imaging system parameter calibration method according to claim 7.

Images