JP2021182019A - Imaging apparatus and imaging system - Google Patents

Abstract

To provide an imaging apparatus and imaging system which can easily capture a panoramic image.SOLUTION: An imaging apparatus (11) comprises: an imaging element (12); and an optical system (IL). The imaging element constitutes an imaging surface (12a) which has first and second directions (X, Y) crossing each other. The optical system is arranged so as to form an image on the imaging surface. The optical system includes an asymmetric free-form surface lens (Lf) between the first and second directions in accordance with the projection relation with respect to a cylindrical surface (20) being a virtual curved surface that is curved so as to cover the opposite side to the imaging surface. The projection relation of the optical system makes the straight line direction (d2) crossing the arc direction on the cylindrical surface correspond to the second direction of the imaging surface while making the arc direction (d1) in which the cylindrical surface is curved correspond to the first direction of the imaging surface.SELECTED DRAWING: Figure 1

Description

The present disclosure relates to an image pickup apparatus provided with an optical system and an image pickup system.

Patent Document 1 discloses a method of capturing a digital panoramic image with a rectangular image sensor. In Patent Document 1, a ring lens is used as a fisheye objective lens, and a deformed panoramic image is projected onto an image sector so as to occupy a larger number of pixels than the disk-shaped image occupies. This improves the definition of the zone of the image used and enhances the quality of the digital zoom.

International Publication No. 2003/010599

The present disclosure provides an imaging device and an imaging system capable of easily capturing a panoramic image.

The image pickup apparatus according to the present disclosure includes an image pickup device and an optical system. The image pickup device constitutes an image pickup surface having first and second directions intersecting with each other. The optical system is arranged so as to form an image on the imaging surface. The optical system comprises a free-form surface lens that is asymmetric between the first and second directions depending on the projection relationship with respect to the cylindrical surface, which is a virtual curved surface that is curved so as to cover the side opposite to the imaging surface. Regarding the projection relationship of the optical system, the arc direction in which the cylindrical surface is curved corresponds to the first direction of the imaging surface, and the linear direction intersecting the arc direction in the cylindrical surface corresponds to the second direction of the imaging surface.

The image pickup system according to the present disclosure includes the above image pickup device and an image processing unit. The image processing unit generates a panoramic image based on the captured image captured by the imaging device.

According to the image pickup apparatus and the image pickup system according to the present disclosure, it is possible to easily capture a panoramic image based on the projection relationship with respect to the cylindrical surface.

The figure for demonstrating the outline of the image pickup system which concerns on Embodiment 1 of this disclosure. The figure which shows the structure of the image pickup system which concerns on Embodiment 1. The figure for demonstrating the image pickup element in an image pickup system. The figure which illustrates the structure of the optical system in an image pickup system. A diagram for explaining a cylindrical projection method in an imaging system. A diagram illustrating the correspondence between a horizontal plane and a vertical plane in a cylindrical projection method. A diagram showing another example of the correspondence between the horizontal plane and the vertical plane in the cylindrical projection method. The figure for demonstrating the optical property of the optical system in Embodiment 1. Scatter plot showing the relationship between the angle of view and the image point in the optical system of the first embodiment according to the first embodiment. Scatter plot exemplifying the relationship between the angle of view and the image point in the central projection method Scatter plot showing the relationship between the angle of view and the image point in the optical system of Example 2. Scatter plot showing the relationship between the angle of view and the image point in the optical system of Example 3. A flowchart illustrating a panoramic image composition process in an imaging system. The figure which illustrates the operation of the image pickup apparatus in an image pickup system The figure for demonstrating the composition process of a panoramic image in an imaging system. The figure for demonstrating the modification of the image pickup system.

Hereinafter, embodiments will be described in detail with reference to the drawings as appropriate. However, more detailed explanation than necessary may be omitted. For example, detailed explanations of already well-known matters or duplicate explanations for substantially the same configuration may be omitted. This is to avoid unnecessary redundancy of the following description and to facilitate the understanding of those skilled in the art.

It should be noted that the applicant is not intended to limit the subject matter described in the claims by those skilled in the art by providing the accompanying drawings and the following description in order to fully understand the present disclosure. No.

(Embodiment 1)
Hereinafter, Embodiment 1 of the present disclosure will be described with reference to the drawings. In the first embodiment, an image pickup apparatus and an image pickup system that employ a novel cylindrical projection method for the optical system will be described.

1. 1. Configuration The imaging system according to the first embodiment will be described with reference to FIG. FIG. 1 is a diagram for explaining an outline of an imaging system according to the present embodiment.

FIG. 1A shows an example of a cylindrical surface 20 with respect to the image pickup apparatus 11 in the image pickup system of the present embodiment. FIG. 1B shows an image captured by the image pickup apparatus 11 in the example of FIG. 1A.

The imaging system of the present embodiment employs a cylindrical projection method for optical imaging when imaging is performed by the imaging device 11. As illustrated in FIG. 1A, the cylindrical projection method is a projection method based on a cylindrical surface 20 which is a virtual curved surface having a cylindrical shape. The cylindrical surface 20 is set to cover the incident side of the light of the image pickup device 11 by, for example, an arc direction d1 curved along a horizontal plane with respect to the optical axis of the image pickup device 11 and a linear direction d2 orthogonal to the horizontal plane. NS. On the cylindrical surface 20, for example, an image of a scene around the image pickup apparatus 11 is developed in a panoramic manner.

As shown in FIG. 1A, for example, the image pickup apparatus 11 of the present embodiment captures a scene in a range defined by the horizontal angle of view θ1 and the vertical angle of view θ2 on the cylindrical surface 20. FIG. 1B shows an captured image generated per such imaging. The horizontal angle of view θ1 indicates a range of angles that can be imaged in the horizontal direction orthogonal to the optical axis of the image pickup device 11 on a horizontal plane. The vertical angle of view θ2 indicates, for example, an angle range that can be imaged in a vertical direction (that is, a linear direction d2) orthogonal to a horizontal plane.

According to the imaging system of the present embodiment, based on the cylindrical projection method, an image of a scene developed on the cylindrical surface 20 is imaged on the imaging surface inside the imaging device 11 so as not to be distorted, and an captured image is generated. NS. As a result, for example, even in one imaging, a panoramic image can be obtained with uniform image quality over the horizontal angle of view θ1 corresponding to the arc direction d1 of the cylindrical surface 20. This imaging system is applicable to various applications such as robot vision, and is applicable to, for example, mobile robots, Visual-SLAM (simultaneous positioning map creation), 360 ° cameras, conference systems, in-vehicle sensors, and the like.

1-1. System Configuration The configuration of the imaging system according to this embodiment will be described with reference to FIG. FIG. 2 is a diagram showing a configuration of an imaging system.

As shown in FIG. 2, for example, the image pickup system according to the present embodiment includes an image pickup device 11 and an image processing unit 13. The image pickup device 11 includes an optical system IL and an image pickup element 12. The image pickup device 11 is a device that captures images of various objects as subjects, and constitutes, for example, various cameras. The image processing unit 13 may be configured by an information processing device such as a PC, or may be incorporated in a camera or the like.

Hereinafter, the direction of the optical axis d0 of the optical system IL in the image pickup apparatus 11 is defined as the Z direction, the horizontal direction orthogonal to the Z direction is defined as the X direction, and the vertical direction orthogonal to the Z and X directions is defined as the Y direction. Further, the + Z side in the optical system IL may be referred to as an image plane side, and the −Z side may be referred to as an object side.

The optical system IL captures light incident from the outside of the image pickup apparatus 11 and forms an image such as an image circle by the captured light. The optical system IL constitutes, for example, a single focus lens or a zoom lens. The cylindrical surface 20 illustrated in FIG. 1 becomes rotationally asymmetric about the optical axis d0 of the image pickup apparatus 11. The image pickup apparatus 11 of the present embodiment implements a cylindrical projection method by using an optical system IL that guides light in a rotational asymmetry. Details of the optical system IL and the cylindrical projection method will be described later.

The image sensor 12 is, for example, a CCD or CMOS image sensor. The image pickup device 12 has an image pickup surface in which a plurality of pixels are two-dimensionally arranged at equal intervals. The image pickup device 12 is arranged in the image pickup apparatus 11 so that the image pickup surface is located on the image plane of the optical system IL. The image pickup device 12 takes an image formed on the image pickup surface via the optical system IL and generates an image signal indicating the image pickup image.

The image processing unit 13 performs predetermined image processing on the image captured by the image pickup device 11 based on the image signal from the image pickup device 12, and generates, for example, a panoramic image based on a plurality of captured images. The image processing by the image processing unit 13 includes stitching for connecting a plurality of images. Image processing may include, for example, gamma correction, distortion correction, and the like.

The image processing unit 13 includes, for example, a CPU or an MPU that realizes various functions by executing a program stored in an internal memory. The image processing unit 13 may include a dedicated hardware circuit designed to realize a desired function. The image processing unit 13 may include a CPU, MPU, GPU, DSP, FPGA, ASIC, or the like.

In the image pickup system of the present embodiment, the image pickup surface of the image pickup element 12 is configured, for example, in a rectangular shape. The image pickup surface of the image pickup device 12 will be described with reference to FIG.

FIG. 3 illustrates a case where the image pickup surface 12a of the image pickup element 12 is rectangular. The image pickup device 12 has a long side Dx and a short side Dy that define the image pickup surface 12a. In the example of FIG. 3, the long side Dx is orthogonal to the short side Dy and larger than the short side Dy. In the image sensor 12 of this example, the long side Dx is parallel to the X direction and the short side Dy is parallel to the Y direction.

FIG. 3 illustrates the positional relationship between the image circle Is by the optical system IL and the image pickup surface 12a of the image pickup device 12. The image circle Is of the present embodiment has a shape distorted from a circular shape to an ellipse or the like, and has a major axis Ix and a minor axis Iy. In the example of FIG. 3, the major axis Ix is orthogonal to the minor axis Iy and larger than the minor axis Iy. The optical system IL is arranged so that the major axis Ix of the image circle Is is parallel to the X direction and the minor axis Iy is parallel to the Y direction corresponding to the long side Dx and the short side Dy of the image pickup device 12. .. In the image sensor 12 of the present embodiment, the X direction is an example of the first direction, and the Y direction is an example of the second direction.

The image circle Is of the optical system IL has, for example, a portion not included in the range of the image pickup surface 12a of the image pickup element 12. In the example of FIG. 3, the major axis Ix of the image circle Is is larger than the long side Dx of the image pickup device 12. Further, the short diameter Iy of the image circle Is is larger than the short side Dy of the image pickup device 12. The image pickup device 12 captures an image by the image circle Is within the range of the image pickup surface 12a.

1-2. Optical system The details of the optical system IL in the present embodiment will be described with reference to FIG. FIG. 4 is a diagram illustrating the configuration of the optical system IL in the imaging system.

FIG. 4A shows a lens arrangement diagram in a YZ cross section of the optical system IL. FIG. 4B shows a lens arrangement diagram in the XZ cross section of the optical system IL. In FIG. 4B, various reference numerals are omitted.

Each lens arrangement diagram of FIGS. 4A and 4B shows the arrangement of various lenses and the like in the optical system IL. The YZ cross section of FIG. 4A is a virtual cross section along the Y direction and the optical axis d0 of the optical system IL. Hereinafter, the YZ cross section is also referred to as a vertical plane. The XZ cross section of FIG. 4B is a virtual cross section along the X direction, which is the horizontal direction, and the optical axis d0, and corresponds to the horizontal plane of the image pickup apparatus 11.

As shown in FIG. 4A, the optical system IL of the present embodiment includes a plurality of lens elements Ln and a diaphragm A. The plurality of lens elements Ln are arranged in order from the object side to the image plane side along the optical axis d0 in the optical system IL.

The diaphragm A is an aperture diaphragm and is arranged between a plurality of lens elements Ln. For example, the aperture A has an F value of less than 16. This makes it easy to avoid a situation where a good image cannot be obtained from the viewpoint of the influence of the diffraction limit. The F value of the aperture A may be set to less than 8 from the above viewpoint.

The plurality of lens elements Ln in the optical system IL include a free curved lens Lf. The free curved surface lens Lf is a lens element having a free curved surface on at least one of the object side and the image plane side. The free curved surface is a curved surface that is rotationally asymmetric with respect to the optical axis d0. In this example, the free-form surface lens Lf is arranged on the object side and the image plane side of the diaphragm A, respectively. Further, the lens element Ln in the optical system IL may include a rotationally symmetric lens Lr such as a spherical lens and an aspherical lens. The optical system IL may include a cover glass of the image pickup device 12.

In the optical system IL of the present embodiment, the free curved surface lens Lf has an asymmetric free curved surface between the X direction and the Y direction, as shown in FIGS. 4 (a) and 4 (b), for example. With such a free-form surface lens Lf, the optical system IL of the present embodiment controls the incident light rays in a rotationally asymmetric manner according to the cylindrical projection method. The free curved surface in the free curved surface lens Lf is, for example, an XY polynomial plane.

In the optical system IL of the present embodiment, various parameters for the lens element Ln including the free-form surface lens Lf can be set so that the cylindrical projection method is implemented. The various parameters include various coefficients that define the corresponding surface shape, for example, various coefficients of the XY polynomial that define the XY polynomial surface. Various parameters may include radius of curvature, interplanar spacing, refractive index, Abbe number, and the like.

The optical system IL of the present embodiment has, for example, a focusing distance larger than 10 times and smaller than 1000 times the focal length of the optical system IL. Below the above lower limit, it can be difficult to control curvature of field. Further, if the above upper limit value is exceeded, the image of a short-distance object may be blurred due to the depth of field.

Further, in the optical system IL of the present embodiment, the half angle of view θ1 / 2 of the horizontal angle of view θ1 is, for example, larger than 20 ° and smaller than 85 °. Below the above lower limit, the corresponding cylindrical surface 20 becomes too narrow. Further, if it exceeds the above upper limit value, it becomes difficult to control astigmatism in the optical system IL. From the above viewpoint, 40 ° <θ1 / 2 <70 ° may be used.

1-3. Cylindrical projection method The cylindrical projection method by the imaging system of the present embodiment will be described with reference to FIGS. 5 to 12.

FIG. 5 is a diagram for explaining a cylindrical projection method in an imaging system. FIG. 6 is a diagram illustrating the correspondence between the horizontal plane and the vertical plane in the cylindrical projection method. FIG. 7 is a diagram showing another example of the correspondence between the horizontal plane and the vertical plane.

FIG. 6A illustrates a projection relationship on a horizontal plane (that is, an XZ cross section) by a cylindrical projection type optical system IL. FIG. 6B illustrates a projection relationship on a vertical plane (that is, a YZ cross section) by a cylindrical projection type optical system IL.

The theoretical formula of the cylindrical projection method is ideally expressed as the following formulas (E1) and (E2).
x = fx * θx ... (E1)
y = fy * cosθx * tanθy ... (E2)

In the above equation (E1), x indicates the image height in the X direction and is defined for each image point with the optical axis d0 as a reference. θx is an angle indicating a half angle of view of the image point in the X direction (see FIG. 5). fx is the paraxial focal length in the X direction for the optical system IL. Similarly, in the above equation (E2), y indicates the image height of various image points in the Y direction, θy is an angle indicating the half angle of view of the image points in the Y direction, and fy is the paraxial focal length in the Y direction. The distance. The above equations (E1) and (E2) can be appropriately defined even in a region where various variables x, θx, y, and θy are negative.

As in the above equation (E1), the cylindrical projection type optical system IL controls (that is, guides) the incident light rays according to equidistant projection in the horizontal plane. As a result, as shown in FIG. 6A, for example, the images of the plurality of points P2a arranged on the cylindrical surface 20 for each angle width Δθ at the horizontal angle of view are formed on the imaging surface 12a via the optical system IL. An image is formed at each image point P1a arranged in the X direction. When the projection relationship of the optical system IL is ideal (see equation (E1)), the angular width Δθ at equal intervals along the arc direction d1 of the cylindrical surface 20 is the image points at equal intervals in the X direction on the imaging surface 12a. It corresponds to the distance Δx between P1a.

Along with the equidistant projection in the horizontal plane as described above, the cylindrical projection type optical system IL guides the vertical plane (that is, the YZ cross section) according to the central projection as in the above equation (E2). FIG. 6B shows the projection relationship of the optical system IL in the XZ cross section (that is, θx = 0) passing through the optical axis d0. For example, images of a plurality of points P2b arranged on the cylindrical surface 20 for each width Δd in the linear direction d2 are imaged on each image point P1b arranged in the Y direction on the imaging surface 12a via the optical system IL. .. When the projection relationship of the optical system IL is ideal (see equation (E2)), the equally spaced width Δd on the cylindrical surface 20 is the distance Δy between the equally spaced image points P1b in the Y direction on the imaging surface 12a. handle.

Further, the light guide of the central projection by the optical system IL changes for each angle θx in the horizontal angle of view as shown in the above equation (E2). FIG. 7 (a) shows the same projection relationship as in FIG. 6 (a). FIG. 7B illustrates the projection relationship of the optical system IL in the YZ cross section deviated from the optical axis d0 of FIG. 7A. According to the projection relationship of the ideal optical system IL, each point P2c on the cylindrical surface 20 approached as shown in FIG. 7 (b) by the amount of cos θx <1 in the above equation (E2) at various positions of θx ≠ 0. A central projection is guided between the image point P1c and the image point P1c as in FIG. 6 (b).

The optical system IL of the present embodiment has the above-mentioned optical characteristics peculiar to the cylindrical projection method. For example, the optical system IL has an asymmetric distortion between the X direction and the Y direction. The distortion of the optical system IL will be described with reference to FIG.

FIG. 8 is a diagram for explaining the optical characteristics of the optical system IL in the present embodiment. FIG. 8A illustrates the characteristics of the distortion aberration of the optical system IL. FIG. 8B exemplifies a subject that serves as a reference for distortion.

FIG. 8A corresponds to a captured image in which distortion occurs when the imaging device 11 of the present embodiment captures a square grid-shaped subject as shown in FIG. 8B. The origin O in FIG. 8A corresponds to the position of the optical axis d0. Note that FIG. 8A illustrates the distortion characteristics when the optical system IL is fx = fy.

In the optical system IL of the present embodiment, the characteristics of distortion in the X direction are similar to those of the equidistant projection method. For example, as shown in FIG. 8A, the optical system IL has a negatively large distortion as the distance from the optical axis d0 in the X direction increases. As a result, an image can be formed by associating the arc direction d1 of the cylindrical surface 20 with the X direction of the imaging surface 12a. Further, in the optical system IL of the present embodiment, the characteristic of distortion in the Y direction varies negatively depending on the position in the X direction, for example, as shown in FIG. 8A. As a result, an image can be formed by associating the linear direction d2 of the cylindrical surface 20 with the Y direction of the imaging surface 12a for each angle θx in the X direction.

In the above example, fx = fy makes it easy to obtain smooth distortion characteristics in both the X and Y directions. The optical characteristics peculiar to the cylindrical projection method in the optical system IL can be set by, for example, various parameters described above. Examples 1 to 3 of the optical system IL according to the present embodiment will be described with reference to FIGS. 9 to 12.

FIG. 9 is a scatter diagram showing the relationship between the angle of view and the image point P11 in the optical system IL of the first embodiment according to the present embodiment. The optical system IL of this embodiment satisfies the theoretical equations (E1) and (E2) of the cylindrical projection method. The optical system IL of this embodiment has a horizontal image height of 3 mm and a vertical image height of 1.85 mm as maximum values. The focal length of the optical system IL was fx = fy = 2.64 mm. The half-angle of view of the horizontal angle of view θ1 in the optical system IL was 65.0 °, and the half-angle of view of the vertical angle of view θ2 was 35.0 °.

In FIG. 9, the image point P11 in which the incident light is imaged on the image plane is plotted for each predetermined angle width over the entire angle of view of the optical system IL. The angle width was set to 5 ° in each of the horizontal and vertical directions. FIG. 9 illustrates the image point P11 in the first quadrant on the XY plane of the image plane whose origin is the position of the optical axis d0. Since the optical system IL of this embodiment is line-symmetrical with respect to the X-axis and the Y-axis, the same applies to the second to fourth quadrants as in FIG.

FIG. 10 is a scatter diagram illustrating the relationship between the angle of view and the image point P0 in the central projection method. In the example of FIG. 10, the image points P0 plotted in the same manner as in FIG. 9 are shown for an optical system that is symmetrical between the X and Y directions and follows a simple central projection method. According to FIG. 10, as the horizontal image height increases, the distance between the image points P0 for each predetermined angle width increases. As described above, in the case of the simple central projection method, the image quality is non-uniform in the horizontal direction.

On the other hand, according to the optical system IL of the present embodiment based on the cylindrical projection method, as shown in FIG. 9, the intervals of the image points P11 are evenly spaced in the horizontal direction, and uniform image quality is obtained. As described above, according to the imaging system of the present embodiment, it is possible to efficiently form an image on the entire image plane by the optical system IL of the cylindrical projection method.

The optical system IL in the imaging system of the present embodiment does not have to strictly satisfy the theoretical equations (E1) and (E2), and the cylindrical projection method is implemented by satisfying the following equations (1) and (2). May be good.
0.5 <x / (fx * θx) <2.0 ... (1)
0.5 <y / (fy * cosθx * tanθy) <2.0 ... (2)
A panoramic image can be formed with sufficient accuracy for practical use even by the projection relationship of the optical system IL satisfying the above equations (1) and (2).

FIG. 11 is a scatter diagram showing the relationship between the angle of view and the image point P12 in the optical system IL of the second embodiment. The optical system IL of this embodiment has a horizontal image height of 3 mm, a vertical image height of 2.14 mm, a paraxial focal length fx in the X direction fx = 2.64 mm, a paraxial focal length in the Y direction fy = 3.50 mm, and a horizontal half-angle. It has an angle θ1 / 2 = 65 ° and a vertical paraxial angle θ2 / 2 = 35 °.

In the optical system IL of this embodiment, the value of the middle side of the above equation (1) is in the range of 0.97 to 1.00, and the value of the middle side of the above equation (2) is 0.74 to 1. It is within the range of 00 and satisfies the above equations (1) and (2). As shown in FIG. 11, the optical system IL of this embodiment can also obtain uniform image quality in the horizontal direction.

FIG. 12 is a scatter diagram showing the relationship between the angle of view and the image point P13 in the optical system IL of the third embodiment. The optical system IL of this embodiment has a horizontal image height of 3.15 mm, a vertical image height of 2.14 mm, a paraxial focal length fx in the X direction fx = 2.64 mm, a paraxial focal length in the Y direction fy = 3.50 mm, and is horizontal. It has a half-angle of view θ1 / 2 = 50 ° and a vertical half-angle of view θ2 / 2 = 35 °.

In the optical system IL of this embodiment, the value of the middle side of the above equation (1) is in the range of 1.00 to 1.37, and the value of the middle side of the above equation (2) is 0.79 to 1. It is within the range of 00 and satisfies the above equations (1) and (2). As shown in FIG. 12, even with the optical system IL of this embodiment, uniform image quality can be obtained in the horizontal direction.

2. 2. Operation The operation of the image pickup system and the image pickup apparatus 11 configured as described above will be described below.

In the imaging system according to the present embodiment, for example, a panoramic image in which scenes of the imaging results for a plurality of times are continuously projected by performing imaging a plurality of times in the imaging apparatus 11 and synthesizing a plurality of captured images in the image processing unit 13. To generate. When the image pickup apparatus 11 generates each captured image, the optical system IL forms an image of a scene developed on the cylindrical surface 20 on the image pickup surface 12a of the image pickup element 12 according to the above-mentioned cylindrical projection method. As a result, the image processing unit 13 can easily perform stitching image processing for connecting a plurality of captured images, and can improve the image quality of the panoramic image.

2-1. Panorama Image Synthesis Processing In the imaging system of the present embodiment, the processing in which the image processing unit 13 synthesizes panoramic images will be described with reference to FIGS. 13 to 15.

FIG. 13 is a flowchart illustrating a panoramic image compositing process in an imaging system. Each process according to the flowchart of FIG. 13 is executed by the image processing unit 13 of the imaging system.

First, the image processing unit 13 of the image pickup system acquires a plurality of captured images from, for example, one image pickup device 11 (S1). The image pickup apparatus 11 captures the surrounding scene a plurality of times, for example, by time division, and outputs the plurality of captured images to the image processing unit 13. 14 (a) and 14 (b) show an example of the state of the image pickup of the image pickup apparatus 11 in step S1.

FIG. 14A illustrates how the image pickup apparatus 11 performs the first image pickup. FIG. 14B exemplifies a state in which the image pickup apparatus 11 performs a second image pickup following FIG. 14 (a). In the example of FIGS. 14A and 14B, a plurality of subjects 31 and 32 are arranged around the image pickup apparatus 11. The number of times of imaging by the image pickup apparatus 11 is not particularly limited to two, and may be three or more.

In step S1, as shown in FIGS. 14A and 14B, for example, the image pickup apparatus 11 sequentially performs imaging while changing the direction of the optical axis d0 with the center of the arcuate direction d1 of the cylindrical surface 20 as the rotation axis. .. The operation of such an image pickup apparatus 11 may be operated by a user or may be automatically controlled. The image pickup system may include a drive device for driving the image pickup device 11.

In step S1, the captured image acquired from the image pickup apparatus 11 follows a cylindrical projection method, as appropriate within the margin of error. The image processing unit 13 may perform image processing on the acquired captured image to correct an error with respect to the ideal cylindrical projection method.

Next, the image processing unit 13 performs stitching image processing on the acquired plurality of captured images (S2) to synthesize a panoramic image. The image processing of stitching by the image processing unit 13 will be described with reference to FIG.

FIG. 15 (a) illustrates the captured image 41 obtained at the time of imaging of FIG. 14 (a). FIG. 15 (b) illustrates the captured image 42 obtained during the imaging of FIG. 14 (b). FIG. 15 (c) illustrates the panoramic image 4 after stitching with respect to the captured images 41 and 42 of FIGS. 15 (a) and 15 (b).

In step S2, the image processing unit 13 connects the captured images 41 and 42 shown in FIGS. 15 (a) and 15 (b) to each other, and synthesizes the panoramic image 4 as illustrated in FIG. 15 (c). At this time, the image processing unit 13 may determine the overlapping region 40 between the captured images 41 and 42 by, for example, image analysis, and perform processing such as averaging or replacement of the determined region 40.

At the time of image processing in step S2, the image processing unit 13 indicates the orientation of each of the acquired captured images 41 and 42, such as the orientation of the optical axis d0 in each of the examples of FIGS. 14A and 14B. Information may be used. The information may be input by a user operation, for example, or may be acquired from a gyro sensor or the like incorporated in the image pickup apparatus 11.

Returning to FIG. 13, the image processing unit 13 outputs the combined panoramic image 4 to various display devices or storage devices (S3), and ends the processing according to this flowchart.

According to the above processing, the captured image (S1) acquired from the image pickup apparatus 11 projects the surrounding scene with uniform image quality along the arc direction d1 of the cylindrical surface 20 according to the cylindrical projection method. For example, the same subject 32 appears at a different position between the captured images 41 and 42 in FIGS. 15A and 15B. At this time, according to the cylindrical projection method, although the direction of the optical axis d0 with respect to the subject 32 is rotated (FIGS. 14 (a) and 14 (b)), the shape of the subject 32 reflected in the captured images 41 and 42 is It hasn't changed.

Therefore, during the stitching image processing (S2), the panoramic image 4 can be synthesized without performing a large coordinate transformation that deforms the subject 32 reflected in the captured images 41 and 42, and in the stitching image processing. The processing load can be reduced. Further, since the above-mentioned large coordinate conversion can be avoided, stitching of the captured image can be performed without deteriorating the image quality from the time of imaging, and the image quality of the panoramic image 4 can be improved.

3. 3. Summary As described above, the image pickup device 11 in the present embodiment includes the image pickup element 12 and the optical system IL. The image pickup device 12 constitutes an image pickup surface 12a having X and Y directions as an example of the first and second directions intersecting each other. The optical system IL is arranged so as to form an image on the image pickup surface 12a. The optical system IL includes a free curved lens Lf that is asymmetrical between the X and Y directions depending on the projection relationship with respect to the cylindrical surface 20 which is a virtual curved surface curved so as to cover the side opposite to the imaging surface 12a. The projection relationship of the optical system IL is such that the linear direction d2 intersecting the arc direction d1 on the cylindrical surface 20 and the Y direction of the image pickup surface 12a while associating the arc direction d1 in which the cylindrical surface 20 curves with the X direction of the image pickup surface 12a. Correspond.

According to the above image pickup apparatus 11, the distortion of the scene on the cylindrical surface 20 can be reduced and image pickup can be performed by the projection relationship of the optical system IL according to the cylindrical projection method with respect to the cylindrical surface 20, and it is possible to easily capture a panoramic image. can.

In the present embodiment, the optical system IL has an asymmetric distortion between the X and Y directions so as to correspond the linear direction d2 and the Y direction while making the arc direction d1 correspond to the X direction (see FIG. 8). .. The optical system IL, which has distortion characteristics peculiar to the cylindrical projection method, makes it easy to capture a panoramic image along the cylindrical surface 20.

Further, in the present embodiment, the projection relationship of the optical system IL is based on equidistant projection in the X direction and based on central projection in the Y direction (see FIGS. 6 and 7). This makes it possible to easily capture a panoramic image along each of the arc direction d1 and the linear direction d2.

Further, in the image pickup apparatus 11 of the present embodiment, the image height x in the X direction and the image height y in the Y direction on the image pickup surface 12a satisfy the following equations (1) and (2).
0.5 <x / (fx * θx) <2 ... (1)
0.5 <y / (fy * cosθx * tanθy) <2 ... (2)
here,
θx: Half angle of view of image height x in X direction θy: Half angle of view of image height y in Y direction fx: Paraxial focal length in X direction in optical system fy: Paraxial focal length in Y direction in optical system .. As a result, by implementing the cylindrical projection method as in the equations (1) and (2), it is possible to sufficiently easily capture a panoramic image in practice.

Further, in the image pickup apparatus 11 of the present embodiment, the half angle of view θx of the image height x in the X direction is larger than 20 ° and smaller than 85 °. As a result, it is possible to sufficiently secure the range of the cylindrical surface 20 to be imaged while controlling astigmatism.

Further, in the image pickup apparatus 11 of the present embodiment, the paraxial focal length fx in the X direction and the paraxial focal length fy in the Y direction may have the same magnitude. In this case, it is easy to obtain smooth distortion in both the X and Y directions.

Further, in the image pickup apparatus 11 of the present embodiment, the optical system IL has, for example, a focusing distance larger than 10 times of at least one of the paraxial focal lengths fx and fy in the X and Y directions and smaller than 1000 times. Have. With such a long focusing distance, it is easy to reduce blurring of an object near the image pickup apparatus 11 while controlling curvature of field.

Further, in the present embodiment, the optical system IL has an F value smaller than, for example, 16. This makes it easy to avoid a situation in which it becomes difficult to obtain a good image from the viewpoint of the diffraction limit.

Further, the image pickup system in the present embodiment includes an image pickup device 11 and an image processing unit 13. The image processing unit 13 generates a panoramic image 4 based on the captured images 41 and 42 captured by the image pickup device 11. According to the image pickup system of the present embodiment, the panoramic image 4 can be easily generated by the optical system IL of the image pickup device 11.

(Other embodiments)
As described above, the first embodiment has been described as an example of the technique disclosed in the present application. However, the technique in the present disclosure is not limited to this, and can be applied to embodiments in which changes, substitutions, additions, omissions, etc. are made as appropriate. It is also possible to combine the components described in each of the above embodiments into a new embodiment. Therefore, other embodiments will be exemplified below.

In the first embodiment described above, an example of an imaging system including one imaging device 11 has been described. The image pickup system of the present embodiment may include a plurality of image pickup devices 11. This modification will be described with reference to FIG.

FIG. 16 illustrates how two image pickup devices 11 included in the image pickup system perform image pickup. Each image pickup device 11 is connected to, for example, a common image processing unit 13 to form an image pickup system of this modification. The image pickup system of the present embodiment may include three or more image pickup devices 11.

In the example of FIG. 16, each image pickup device 11 is arranged so as to image different regions on a common cylindrical surface 20. In step S1 of FIG. 13, the image processing unit 13 acquires an image captured from each image pickup device 11 and performs stitching image processing (S2). Even in this case, since a uniform image taken along the arc direction d1 of the cylindrical surface 20 can be obtained in each image pickup device 11, the image processing unit 13 can easily perform stitching and obtain a high-quality panoramic image. Can be generated.

Further, in each of the above embodiments, the cylindrical projection method based on the cylindrical surface 20 has been described (see the theoretical formulas (E1) and (E2)). In the imaging system of the present embodiment, the cylindrical surface 20 may be appropriately set to a distorted curved surface within a tolerance. For example, the arc direction d1 of the cylindrical surface 20 does not have to be an arc in a perfect circle, and may be a direction due to an appropriately convex curve. Further, the linear direction d2 does not have to be a direction based on a strict straight line, and may be a direction more linear than the arc direction d1. Further, the arc direction d1 and the linear direction d2 may intersect each other from a right angle as appropriate within an angle range of the margin of error.

Further, in each of the above embodiments, the case where the optical system IL according to the cylindrical projection method is based on equidistant projection in the X direction and based on central projection in the Y direction is illustrated. The projection relationship of the optical system IL of the present embodiment may be based on each stereographic projection, orthographic projection, orthographic projection in the X direction, or may be based on a weak center projection or a pseudo-center projection in the Y direction.

Further, in each of the above embodiments, the rectangular image pickup surface 12a is illustrated in FIG. 3, but the image pickup surface 12a of the image pickup element 12 is not limited to this. In the present embodiment, the image pickup surface 12a of the image pickup element 12 may have various rectangular shapes other than a rectangle, or may be partially masked. Further, the image pickup surface 12a of the image pickup device 12 may be curved.

For example, the long side Dx and the short side Dy of the image pickup device 12 of the present embodiment may not be orthogonal to each other and may intersect at various angles. Further, the image pickup device 12 may have two sides having the same length instead of the long side Dx and the short side Dy. In the optical system IL of the present embodiment, the first and second directions of the major axis Ix and the minor axis Iy of the image circle Is do not have to be orthogonal to each other, and may intersect at various angles. Further, the lengths of the diameters in the first and second directions in the image circle Is may be the same. The image circle Is does not necessarily have to be distorted from a circle. Further, in the present embodiment, the image circle Is has a portion not included in the image pickup surface 12a of the image pickup element 12, but the image circle Is may include the image pickup surface 12a.

In each of the above embodiments, an XY polynomial plane is illustrated as an example of a free-form surface. In the present embodiment, the free curved surface is not limited to the above, and may include, for example, an anamorphic aspherical surface or a toric surface. Further, in the optical system IL of the present embodiment, the free-form surface lens may have a free-form surface that is not anamorphic. A non-anamorphic free-form surface contains an XY polynomial surface but does not contain an anamorphic aspherical surface. A non-anamorphic free-form surface does not have to have, for example, a plane of symmetry.

As described above, an embodiment has been described as an example of the technique in the present disclosure. To that end, the accompanying drawings and detailed description are provided.

Therefore, among the components described in the attached drawings and the detailed description, not only the components essential for problem solving but also the components not essential for problem solving in order to illustrate the above-mentioned technology. Can also be included. Therefore, the fact that those non-essential components are described in the accompanying drawings or detailed description should not immediately determine that those non-essential components are essential.

Further, since the above-described embodiment is for exemplifying the technique in the present disclosure, various changes, substitutions, additions, omissions, etc. can be made within the scope of claims or the equivalent thereof.

The present disclosure is applicable to various applications for imaging, and is applicable to, for example, mobile robots, Visual-SLAM, 360 ° cameras, conference systems, in-vehicle sensors, and the like.

11 Image pickup device 12 Image pickup element 13 Image processing unit 20 Cylindrical surface d1 Arc direction d2 Linear direction IL Optical system Ln Lens element Lf Free curved lens

Claims (9)

An image sensor constituting an image pickup surface having first and second directions intersecting with each other,
It is provided with an optical system arranged so as to form an image on the imaging surface.
The optical system comprises a free-form surface lens that is asymmetric between the first and second directions depending on the projection relationship to a cylindrical surface that is a virtual curved surface that is curved so as to cover the side opposite to the imaging surface.
The projection relationship of the optical system corresponds to the arc direction in which the cylindrical surface is curved and the first direction of the imaging surface, and the linear direction intersecting the arc direction in the cylindrical surface and the second direction of the imaging surface. Corresponding image pickup device. The first aspect of the optical system has an asymmetric distortion between the first and second directions so that the linear direction and the second direction correspond to each other while the arc direction and the first direction correspond to each other. The imaging device according to. The imaging device according to claim 1 or 2, wherein the projection relationship of the optical system is based on equidistant projection in the first direction and central projection in the second direction. On the imaging surface, the image height x in the first direction and the image height y in the second direction satisfy the following equations (1) and (2).
0.5 <x / (fx * θx) <2 ... (1)
0.5 <y / (fy * cosθx * tanθy) <2 ... (2)
here,
θx: Half-angle of view of image height x in the first direction θy: Half-angle of view of image height y in the second direction fx: Paraxial focal length in the first direction in the optical system: The imaging device according to any one of claims 1 to 3, which is a paraxial focal length in the second direction. The image pickup apparatus according to claim 4, wherein the half angle of view θx of the image height x in the first direction is larger than 20 ° and smaller than 85 °. The image pickup apparatus according to claim 4 or 5, wherein the paraxial focal length fx in the first direction and the paraxial focal length fy in the second direction have the same magnitude. Any of claims 4 to 6, wherein the optical system has a focusing distance larger than 10 times and less than 1000 times the paraxial focal lengths fx and fy in the first and second directions. The image pickup apparatus according to item 1. The imaging device according to any one of claims 1 to 7, wherein the optical system has an F value smaller than 16. The image pickup apparatus according to any one of claims 1 to 8.
An image pickup system including an image processing unit that generates a panoramic image based on an image captured by the image pickup device.

Images