JP2002535700A - Resolution-invariant panoramic imaging - Google Patents

Abstract

A panoramic imaging system includes an imaging device having an image plane and a first field of view, at least one radially convex at the first field of view for providing an enlarged panoramic second field of view. Including a first reflective surface having a circularly symmetric portion. The contour of the or each convex portion provides a varying gain between fields in the radial direction to limit solid angle variation across the image plane of the imaging device.

Description

DETAILED DESCRIPTION OF THE INVENTION

FIELD OF THE INVENTION The present invention relates generally to the formation of wide-angle images of a space referred to as panoramic imaging.

BACKGROUND ART Panoramic imaging is becoming an important tool in the field of mobile robots and machine vision. There are many described methods for recording a panoramic view of a landscape. One simple method involves placing a series of cameras on a ring to give a view around a full 360 ° horizontal. If each had a 90 ° field of view and some integration of the image, this would include, for example, four cameras. There are also a number of single camera methods for panoramic imaging, including rotating the camera about its vertical axis and taking continuous pictures to get a full panoramic view. Another approach uses wide-angle lenses to achieve a large field of view, but these lenses are heavy, expensive, and distort the image. An attractive approach to panoramic imaging is to place a single fixed camera below a curved reflective surface that covers a hemisphere, such as a cone, sphere, hyperboloid, or other shape. The optical axis of the camera is aligned with the central axis of the mirror. The known family of constant gain reflecting surfaces can form a large field of view, such as for a hemispherical or hyperbolic mirror, but has the advantage of still maintaining a linear relationship between changes in the angle of incidence and reflection of the ray seen by the camera. Having. This linear relationship simplifies the processing of the image and ensures a constant elevation angle resolution of the image. The shape of the surface is determined by the gain of the linear relationship. At unity gain, the surface is conical; at higher gains, the surface is defined by a series of polynomial functions. For ease of description herein, as applicable to a robot moving in a horizontal plane, it is assumed that the panoramic plane is horizontal and the field of view is vertical. It will be clear that in the general case, the orientation of the faces is arbitrary.

[0003] All of the mirror shapes described above have common disadvantages. That is, the CCD camera used in imaging always has a uniform Cartesian array of pixels to capture a polar image of the scenery, and thus the pixel density per solid angle increases with the diameter of the polar image. The distortion correction process converts the image from polar coordinates to Cartesian coordinates, so that the angular coordinates of the original polar image are mapped to x-coordinates in the distortion-corrected image, while the radial coordinates are mapped to y-coordinates.
Thus, the pixel density in the distortion corrected image varies from "low" for small x values corresponding to the center of the original image to "high" for large x values corresponding to the outer edges of the polar image. This is illustrated in FIG. 1, which shows the distortion correction of an image captured with a bi-faceted mirror. Variations in image quality are evident in the distortion corrected version. One way to avoid this problem is to use a specially designed CCD camera with a pole array of pixels with a decreasing pixel density with diameter. There is an alignment problem with such an approach.

SUMMARY OF THE INVENTION In a first aspect, the invention is an imaging device having an image plane and a first field of view, disposed in the first field of view. Includes a first reflective surface having at least one radially convex circularly symmetric portion to provide an enlarged panoramic second field of view, wherein the contour of each or each convex portion varies radially between the fields of view. A panoramic imaging system is provided that provides gain to limit solid angle variation across an image plane of an imaging device. Preferably, the contour of the convex portion provides a substantially uniform solid angle across the image plane.
That is, the shape ensures that the resolving power in the image is invariant with changes in elevation. Thus, if the imaging system includes a device having an array of pixels evenly spaced in the image plane, the shape of the reflective surface will result in a solid angle pixel density invariant. The contour of the reflective surface in polar coordinates is preferably given by the equation:

[0005]

(Equation 5) (Where r is the radial distance from the reflecting surface to the imaging device, θ is the angle from the optical axis of the imaging device, and α (θ) is

[0006]

(Equation 6) Where φ and φ are the maximum and minimum elevation angles seen from bars θ and θ , respectively,
And θ are the maximum and minimum radial angles imaged).

In one approach, r is plotted against θ at selected intervals and contoured by solving the equation for selected values of θ.
For example, determining the value of r for increments of θ of about 1/5 ° has been found to produce sufficiently accurate contours in practical applications. There are a number of methods for panoramic distance measurement. One method uses a conical mirror above the camera. The camera mirror assembly can be moved to another location during image acquisition, or
Two camera mirror assemblies are used to obtain the two views required for distance measurement. Although this method gives distance information in the horizontal plane at video speed, the disadvantage is that there is no distance information in the vertical (elevation) direction, and the object must be larger than the minimum distance from the camera. , There may be blind spots due to the second camera system. Discontinuous axisymmetric mirrors, essentially a pair of coaxial mirrors, are known above the camera to obtain two views of the entire scene for stereo mismatch distance measurement. However, there is no proposal for a specific mirror shape to achieve certain desirable properties.
In addition, known constant gain mirror profiles have been generalized to guide such a series of coaxial mirror pair profiles for panoramic stereoscopic imaging and processing based on discrepancies in the vertical plane.

[0008] In another aspect, the invention provides a distance measurement using a panoramic imaging system that includes two resolution invariants. Preferably, the mirror or reflecting mirror surface has at least two said convex portions arranged to provide at least partially overlapping panoramic second views for distance measurement. The second field of view is preferably substantially co-incident. In a preferred form of the invention, the two convex parts form a continuous mirror or reflecting mirror.

In a further aspect, the present invention provides a design for a back-to-back stereoscopic mirror system with the desirable properties of two cameras, and thus equal pixel sharing between the two stereoscopic images. The solid cone in this case is preferably symmetric in a direction perpendicular to the camera axis, which is a desirable property for some applications. In this aspect of the invention, the imaging system preferably comprises two first reflective surfaces each having an image plane associated with a corresponding first field of view, and each first reflective surface providing a respective panoramic second field of view. And the first reflective surface is arranged back-to-back such that the reflective second views at least partially overlap.

In some applications, the second reflective surface can be located between the image plane and the second reflective surface. This allows, for example, positioning of the imaging device behind the first reflective surface. In one variation, an aperture can be provided in the first reflective surface to provide a first field of view from the imaging device.

In another aspect, the invention provides a reflective surface for use in a panoramic imaging system, including an imaging device having an imaging surface and a first field of view, and an enlarged panorama provided by the reflective surface. Limiting solid angle variation across the image plane of the imaging device with at least one radially convex circularly symmetric portion having a contour that provides a radially varying gain between the second field of view and the first field of view. I do.

In yet a further aspect, the present invention provides a minimally penetrating design that minimally penetrates the observation “hemisphere”. These are also called forward facing designs. They include the relocation of additional plane mirrors and cameras within the main reflective surface. The appeal of this arrangement is that the first reflecting mirror profile is of the same design as in the more usual arrangement. The invention will now be further described by way of example with reference to the accompanying drawings.

DETAILED DESCRIPTION OF THE INVENTION For clarity, various aspects of the invention are set forth under separate subheadings. 1. Resolution Invariant Mirror Family This section describes a series of mirror designs that achieve the goal of resolution invariant or equivalent solid angle pixel density invariant. 1.1 Constant Image Pixel Density-Variable Gain (α) Mirror According to one aspect of the invention, the mirror contour is adjusted so that relatively few scenes are imaged at the center of the image and relatively many images are formed at the periphery. By doing so, resolution invariance is achieved. That is, the mirror contour is selected to maintain a constant relationship between pixel density and elevation in the scene, or more precisely, solid angle. The mirror gain α is the relationship between the change in elevation of the ray incident on the mirror and the change in the angle of the ray reflected on the mirror, as follows:

(Equation 7) (Where δφ is the change in vertical elevation angle and δθ is the change in angle of the reflected ray received by the camera). With respect to the resolving power, the invariant α is a function of the image angle θ related to the radial coordinate ρ in the image as shown in FIG.

FIG. 3 schematically shows an imaging system including an imaging device in the form of a camera having an image plane and a first field of view. The reflecting surface or mirror provides an enlarged panoramic second field of view in the first field of view. The surface is circularly symmetric and radially convex. As shown in FIG. 3, consider a mirror profile (r, θ) expressed in polar coordinates, where r is the radial distance to the camera and θ is the angle from the optical axis of the camera to a point on the mirror surface.
The angle of incidence of the ray on the mirror is γ and the angle of the ray on normal is φ. Therefore,

[0015]

(Equation 8) Which is (from the law of reflection) a geometric constraint:

[0016]

(Equation 9) Obey. Differentiating (2) and (3) with respect to θ,

[0017]

(Equation 10) And substituting α from (1),

[0018]

[Equation 11] Is obtained. Now, for a variable gain mirror, α is a function of the image angle θ (related to the radial coordinates ρ in the image), so (4)

[0019]

(Equation 12) Become or organize

[0020]

(Equation 13) It becomes. The equation for mirror gain α (θ) to achieve pixel density invariance can be found using the following theory.

1.1.1 Pixel Density Invariant Contour There are p (ρ) pixels in the area of diameter ρ in the image. More formally, in the area of diameter ρ,

[0022]

[Equation 14] There are pixels, where κ is the number of pixels per unit area and is a constant. Differentiating with ρ,

[0023]

[Equation 15] It becomes. Now, the diameter ρ in the image is related to the radial angle θ of the light beam reflected from the mirror by the focal length f (constant) of the camera,

[0024]

(Equation 16) Then, p (ρ) is differentiated by θ, and (7) and (8) are substituted.

[0025]

[Equation 17] Is obtained. Now, the image pixel density is required to be invariant to the elevation angle in the scene, which leads to more scene being imaged towards the periphery,

[0026]

(Equation 18) Where β and C (φ) are constants. Differentiate both sides of (10) with φ,
Substituting (1) and (9),

[0027]

[Equation 19] Is obtained. Arranging (11),

[0028]

(Equation 20) Where B _α is a constant. This expression for α (θ) is integrated with θ to obtain the expression for φ (see (1)), and the elevation angle of the imaged material at angle θ is obtained. That is,

[0029]

(Equation 21) Where φ (θ = 0) is an integration constant. The constants B _α and φ (θ = 0) can be determined from the maximum and minimum values of θ and φ known in the desired mirror configuration using (13).

[0030]

(Equation 22) If α is a function of θ, it seems impossible to find an analytical solution to (6), so there is no clear equation for the mirror shape. Instead, the person solving differential equation 5 needs to find a solution to (6) over the range of θ (mirror surface).

FIGS. 4A and 4B show, for comparison, a constant gain mirror and a variable gain mirror at the same camera field of view and imaged elevation range. The rays shown are regularly spaced in θ, with approximately 2 ° between each ray. From FIG. 4A, for a constant gain, these rays are regularly spaced only at φ, with about 8 seconds between each ray.
. It is clear that it is 5 °, and from FIG. 4 it is clear that for variable gain, the spacing between the rays increases as φ increases. Thus, in the case of variable gain, a greater proportion of the scene is imaged toward the outer edge of the polar mirror.
This is also shown in FIGS. 5 and 5B, where the rays tracked the image reflected by the constant gain and variable gain mirrors at the same range of elevation angles visible.

1.2 Panoramic Stereo with Variable Gain Mirror A mirror with two convex parts or a double mirror is required. Diagram of radial profile of double mirror
FIG. The mirror arrangement for a panoramic stereo with a variable gain mirror,
It will differ from that for a constant gain mirror due to variations in the mirror gain α. The gain is
Constant over full duplex mirror so that constant pixel density theorem applies over the entire image
Must change. If the observed minimum and maximum elevation angles (bar φ and φ ) Is equal for both mirrors in a double mirror system, the range of reflected angles (
Bar θ−θ) Cannot be equivalent for the two mirrors. By camera over all mirror surfaces
Minimum and maximum angles of captured reflected rays are known from camera geometry
. Therefore, the lower mirror (θ ₁ ) And the upper mirror (bar θ)₂) Or
The maximum ray reflected off is known. Therefore, (12) is held over the entire mirror.
And from (14), B_αIs constant.

[0033]

(Equation 23) It is desirable to minimize the radial gap between the images from the two mirrors to maximize usage of the camera's field of view. For the minimum gap (bar θ ₁ = θ ₂ ),

[0034]

[Equation 24] It is. FIGS. 6A, 6B, 7A, and 7B show graphs and ray tracing comparisons of constant and variable gain double mirror systems viewing the same scene.

2.4 Distance Calculation for Variable Gain Panoramic Stereo System The information available in the distance calculation is the image angles, θ ₁ and θ ₂ , for a single object reflected on both mirrors, as shown in FIG. . The two mirrors θ ₁ and θ ₂ form a reflecting surface. Different equations (6) for the surface are known. In the following calculations, only the lower mirror is examined, since the results are identical for the upper mirror. To find the position of the object P, the equations of the rays incident from P on each of the mirror reflection points (r ₁ , θ ₁ ) and (r ₂ , θ ₂ ) must be found. Then, it is possible to obtain the position of the object P by solving these equations simultaneously (x _{p, y p).}

[0036]

(Equation 25) ( _Where m _I1 is the slope of the incident ray to the lower mirror and C _I1 is a constant in the equation). The constant of the equation is

[0037]

(Equation 26) Given here,

[0038]

[Equation 27] Is the Cartesian coordinates of the reflection point (r ₁ θ ₁ ). The slope of the incident ray is the law of reflection:

[0039]

[Equation 28] _Where m _R1 is the gradient of the ray reflected from the lower mirror to the camera and dy ₁ / dx ₁ is the gradient of the lower mirror profile at the point of reflection. The gradient of the reflected ray is

[0040]

(Equation 29) It is. The slope of the mirror profile for the lower variable gain mirror, as in the case of constant gain,

[0041]

[Equation 30] (Where dr / dθ for any mirror in the variable gain mirror configuration is found by integrating (5) and substituting (12)).

[0042]

[Equation 31] (Where D is the integration constant). (23)

[0043]

(Equation 32) Is obtained. From (23) and (2),

[0044]

[Equation 33] Therefore, for the lower variable gain mirror profile,

[0045]

(Equation 34) , And the similar, in D _2, is for the upper variable gain contour. Therefore, by substituting (24) and (22), the gradient of the variable gain mirror profile at any point is obtained. As in the case of constant gain, the slope is θ
Note that it only depends on you.

The equation constants for the incident ray equation from (18) require the polar coordinates (r ₁ , θ ₁ ) and (r ₂ , θ ₂ ) of the point of reflection from each mirror. Since the variable gain mirror equations are not exactly known, r ₁ and r ₂ must be found using different equation solutions to find a solution to (26) at θ ₁ and θ ₂ .

[0047] 2. Back-to-back stereo mirror family An important disadvantage of single-camera stereoscopic panoramic imaging is that since there are two images of "identical" scenery, the pixels assigned to each image are half that for non-stereoscopic panoramic imaging, The two images do not have an equal number of pixels in a fixed gain scheme. In fact, the panoramic stereo double mirror method typically involves one
One radial view is compressed to about 1/4 of the camera's field of view. A method of achieving panoramic stereo with smaller image compression is to use two cameras and two single curved mirrors back to back as shown in FIG. This method compresses the imaged scene to one-half of the camera's field of view, in effect each image has an equal share of the total number of available pixels. However, there are alignment problems possible with this system for any stereo system that uses two cameras to capture two views of the scene. An advantage of the scheme proposed in FIG. 9 is that the solid cone can be symmetric about the horizontal, using two cameras with equal fields of view, and the maximum and minimum elevation angles reflected by the two mirrors are equal. The angle covered by the solid cone is in this case 2 bars φ-π. FIG. 9 shows the general case where the maximum and minimum elevation angles of the radii observed by each camera need not be equal. The elevation range is
Still, it must be equivalent for the fields to be aligned. The number of free parameters to be specified here is reduced. This is because the minimum elevation angle ( φ ) from one mirror must be parallel to the maximum elevation angle (bar φ) from the other mirror. This ensures that the fields of view are parallel. So, referring to FIG.

[0048]

(Equation 35) It is. In the scheme of FIG. 9, the mirror family can be either constant gain or resolution invariant.

2.1 Use of Double Mirror in Back-to-Back Design FIG. 10 shows a back-to-back design incorporating a double mirror. Although the figures show a constant gain mirror, double mirrors can also have variable gain. An advantage to this system is that the solid cone from the back-to-back configuration combines with the solid cone from the double mirror configuration to increase the total area imaged stereoscopically. In this configuration, the field of view of each double mirror pair need not be aligned as in the previous example.
Regarding the symmetry around the horizontal, φ ₃ = φ ₁ , φ ₄ = φ ₂ , bar φ ₃ = bar φ ₁ , bar φ ₄ = bar φ ₂ . Restraint:

[0050]

[Equation 36] Aligns three solid cones. It is also possible to further increase the total solid cone by making the mirror pairs have different gains.

[0051] 3. Front view mirror design An example of a front view mirror design is shown in FIG. In many applications, it is desirable to view the panoramic camera from, for example, a hemisphere, somewhat like a bird's eye, or perhaps two on either side of a "nose cone". There are aerodynamic considerations that motivate such a "front view" system. This configuration is called a front view because the camera points in the direction of the scene. Either constant or variable gain mirrors (double or single) can be used with curved mirrors in the system. The plane mirror is placed in an annular or circular shape such that all light rays reflected from the curved mirror are reflected to the camera o located behind the curved mirror. The dashed line in FIG. 11 indicates where the reflected light rays would converge if the plane mirror were removed, and the dashed camera indicates camera o 'for an equivalent system without the plane mirror. In order for the light rays reflected by the plane mirror to converge at the new camera position, the plane mirror must be the perpendicular bisector of the line connecting the old and new camera positions. Thus, the distance between camera positions is 2D, where D is defined as the distance from any camera to the plane mirror in FIG. The introduction of a plane mirror into the system increases the possibility of alignment difficulties. Because the plane mirror must be perpendicular to the camera axis and must be positioned to reflect all rays from the curved mirror to the camera without interrupting the field of view of the curved mirror. . The maximum value for D and bar D is when the maximum ray reflected from the mirror system to the camera (the bar θ ray reflected at point b on the plane mirror) grazes through the curved contour at c. in this case,

[0052]

(37) It is. Bar D defines the minimum height H for the mirror system. In reality, the value for D needs to be slightly smaller to avoid blockage, leading to a larger mirror system height. The general equation for the height of the mirror system is

[0053]

[Equation 38] It is. Also note that θ must be greater than zero for camera o to be located behind the curved mirror. Also, if the minimum elevation ray φ is not blocked by a plane mirror, then φ > bar θ.

FIG. 12 shows a design that captures the ideas of sections 2 and 3. It consists of two back-to-back front view systems, giving an eye design reminder on a pattern such as a crab eye. The "handle" for this system will be hidden from view by the lower plane mirror. In this arrangement, portions are provided on the curved mirror to provide reflection of the light rays from the curved surface to the camera by the plane mirror. Thus far, only certain aspects of the invention have been described, however, modifications may be made without departing from the scope of the invention.

[Brief description of the drawings]

FIG. 1 illustrates a distortion correction process for a prior art panoramic imaging system.

FIG. 2 schematically shows the relationship between a camera image and a horizontal viewing direction in a panoramic imaging system.

FIG. 3 shows the geometric relationship between the reflective surface and the camera used to guide the mirror contour in accordance with the present invention.

4A and 4B are graphs showing a comparison between a constant gain mirror and a variable gain mirror used in an imaging system according to the present invention.

FIGS. 5A and 5B show ray tracing scenes, respectively, reflected by constant and variable gain mirrors.

FIGS. 6A and 6 graphically illustrate a comparison of panoramic imaging systems, respectively, utilizing a dual constant and variable gain mirror configuration.

FIGS. 7A and 7B correspond to panoramic imaging systems utilizing dual constant and variable gain mirror configurations, respectively, for scenery lacing (ra);
ced) shows a tracking image.

FIG. 8 schematically shows the relationship between a mirror and a reflective surface used in distance calculation using a resolution invariant double mirror according to the present invention.

FIG. 9 schematically shows a back-to-back mirror configuration according to the present invention.

FIG. 10 schematically shows a double back-to-back mirror configuration according to the present invention.

FIG. 11 schematically illustrates a front view panoramic imaging system according to the present invention.

FIG. 12 shows a system that utilizes a combination of the arrangements in FIGS. 10 and 11.

──────────────────────────────────────────────────続 き Continued on the front page (72) Tanya Louise Conroy Australia 2617 Australian Capital Territory, Canberra, Bel Connen, Fenwick Place No. 8 F term (reference) 2F065 AA51 BB05 FF04 FF42 GG10 JJ03 JJ05 JJ26 QQ17 QQ31 QQ42 2H059 BA03 BA11 CA00 5C022 AA00 AB68 AC51 5C054 AA01 AA05 CG02 EA01 FD02 HA00

Claims (21)

[Claims] 1. An imaging device having an image plane and a first field of view, comprising an enlarged panoramic second field of view having at least one radially convex circularly symmetric portion disposed in the first field of view. A panorama characterized in that the contour of the or each convex portion provides a varying gain between the fields of view in a radial direction to limit solid angle variations across the image plane of the imaging device. Imaging system. 2. The panoramic imaging system of claim 1, wherein the contour of the or each convex portion provides a substantially uniform solid angle across the image plane. 3. The contour of the or each convex portion is represented by the equation: (Where r is the radial distance from the reflecting surface to the imaging device, θ is the angle from the optical axis of the imaging device, and α (θ) is: Where φ and φ are the maximum and minimum elevation angles seen from bars θ and θ , respectively,
And θ is claim 1 or claim 2 panoramic imaging system according to at least approximate the contour defined by the polar coordinates by the maximum and minimum diameter direction angle obtained by the image). 4. The panoramic imaging system according to claim 3, wherein the contour of the or each convex portion is included in a series of spaced points defined by determining a distance r for a selected value of angle θ. . 5. The panoramic imaging system according to claim 4, wherein the selected value of θ is divided by about 1/5 °. 6. An at least partially overlapping panorama second for distance measurement.
A panoramic imaging system according to any one of the preceding claims, comprising a first mirror surface having at least two of said convex portions arranged to provide a field of view, respectively. 7. The panoramic imaging system according to claim 6, wherein said second panoramic field of view is substantially co-incident. 8. The panoramic imaging system according to claim 7, wherein said at least two convex portions form a continuous reflecting surface. 9. The method according to claim 1, further comprising the step of defining two first reflecting surfaces each having an image plane associated with a corresponding first field of view, and at least one convex portion of each first reflecting surface providing a respective panoramic second field of view. 9. A panoramic imaging system according to any one of the preceding claims, wherein the first reflecting surface is arranged back-to-back so that the reflected second fields of view at least partially overlap. 10. The panoramic imaging system according to claim 1, further comprising a second reflecting surface located between an image plane and said second reflecting surface. 11. The panoramic imaging system according to claim 10, wherein said imaging device is located behind a second reflective surface. 12. The panoramic imaging system according to claim 11, wherein an aperture is provided in the first reflective surface to provide the first field of view from the imaging device. 13. The method of claim 10, wherein said second reflecting surface is substantially planar.
2. The panoramic imaging system according to any one of 2. 14. A reflective surface for use in a panoramic imaging system including an imaging device having an image plane and a first field of view, said reflective surface comprising:
Imaging having at least one radially convex circularly symmetric portion with a profile providing a radially varying gain between the enlarged panoramic second and first fields of view provided by the reflective surface; The reflective surface characterized by limiting solid angle variations across the image plane of the device. 15. The reflective surface of claim 14, wherein the contour of the or each convex portion provides a substantially uniform solid angle across the image plane. 16. The contour of the or each convex portion is defined by the equation: (Where r is the radial distance from the reflecting surface to the imaging device, θ is the angle from the optical axis of the imaging device, and α (θ) is: Where φ and φ are the maximum and minimum elevation angles seen from bars θ and θ , respectively,
And θ are the maximum and minimum radial angles imaged) at least approximating the contour defined in polar coordinates.
8. The reflecting surface according to 8. 17. The reflective surface of claim 16, wherein the contour of the or each convex portion is included in a series of spaced points defined by determining a distance r for a selected value of angle θ. 18. The method according to claim 1, wherein the selected value of θ is divided by about 1/5 °.
7. The panoramic imaging system according to 7. 19. A method according to claim 14, including a first mirror surface having at least two said convex portions arranged to provide at least partially overlapping panoramic second fields of view for distance measurement. The panoramic imaging system according to claim 1, wherein 20. The reflecting surface of claim 19, wherein the panoramic second field of view is substantially co-incident. 21. A reflective surface according to claim 20, wherein said at least two convex portions form a continuous reflective surface.