JP6824143B2 - Imaging equipment, imaging systems, image processing methods, information processing equipment, and programs - Google Patents

Description

The present invention relates to an imaging device, an imaging system, an information processing device, and a program.

Conventionally, as a device for imaging omnidirectional (or omnidirectional), there are a device using a hyperboloid mirror, a device using a fisheye lens, and the like. These are special optical systems, and in order to make an image visible to the user, it is necessary to perform non-linear image conversion such as distortion correction and projective transformation.

A technique is already known in which a part of a distorted circular image obtained by recording a hemisphere obtained by wide-angle photography using a fisheye lens is cut out and image processing is performed by a computer to convert the distorted image into a regular plane image. ..

Further, when performing such image processing, when the center position of the circularly distorted image does not correctly match the zenith direction, the load at the time of image distortion correction is increased by having the user specify the tilt angle parameter. Techniques for mitigation also exist and are already known.

In this way, when the image pickup device is taken in an inclined state, there is a possibility that an omnidirectional (omnidirectional) image in the vertical direction is created, and an invention for solving this may already be known. There is.

Patent Document 1 or Patent Document 2 discloses a technique for creating a correct omnidirectional (omnidirectional) image in the vertical direction by adding a rotation conversion according to the tilt of the camera to the non-linear image conversion step. ing.

Further, in Patent Document 3 or Patent Document 4, distortion correction and projection are performed for the purpose of shortening the time required to generate a correct omnidirectional (omnidirectional) image in the vertical direction or reducing the calculation cost. In addition to the conversion, a conversion table for performing non-linear conversion collectively by adding rotation conversion according to the tilt of the camera is created in advance, and batch conversion using this conversion table is performed at the time of shooting in the vertical direction. A technique for creating a correct omnidirectional (omnidirectional) image at high speed is disclosed.

However, in the conventional spherical image pickup device, when a plurality of images taken from a plurality of shooting directions at one time are joined together and further converted into a plane regular image and displayed, the tilt angle at the time of shooting by the user is displayed. It is necessary to specify, and there is a problem that it is not possible to acquire the parameters necessary for correction at the time of shooting and automatically correct the image.

In addition, in the conventional omnidirectional (omnidirectional) imaging method that processes an image using a conversion table corresponding to tilt, if the amount or orientation of tilt changes, it is necessary to recreate the conversion table from the beginning. There is. In this case, there is a problem that processing takes time in order to generate a correct omnidirectional (omnidirectional) image in the vertical direction corresponding to an arbitrary tilt of the image pickup apparatus.

In particular, in the technique described in Patent Document 3 or Patent Document 4, image generation is performed using a conversion table including rotation conversion according to the tilt of the camera, but a conversion table according to a predetermined tilt amount is used. Is used. That is, if the amount or orientation of the tilt changes differently from the defined tilt, the conversion table is recreated from the beginning, so the correct omnidirectional (omnidirectional) vertical direction corresponds to any tilt of the imaging device. ) The problem that it takes a long time to generate an image has not been solved yet.

Therefore, the present invention has been made in view of the above-mentioned conventional problems, and is capable of generating a correct omnidirectional (omnidirectional) image in the vertical direction in response to an arbitrary tilt of the imaging device. The purpose is to provide the device.

In order to solve the above problems, the image pickup apparatus according to the present invention according to claim 1 is an image pickup apparatus provided with an image pickup element that captures an image input via a wide-angle lens, and each time the image pickup device captures an image. The tilt detecting means for detecting the tilt of the image pickup device with respect to the gravity direction and the conversion data for converting the plane image represented by the plane coordinate values taken by the image pickup element into the spherical image represented by the spherical coordinate values. A storage means for storing and a generation means for generating a spherical image converted by the conversion data are further provided, and the conversion data is tilted with respect to the gravity direction of the imaging device detected for each shooting by the tilt detecting means. It is characterized in that it is corrected for each shooting according to.

According to the present invention, in an omnidirectional (omnidirectional) imaging device, it is possible to generate a correct omnidirectional (omnidirectional) image in the vertical direction corresponding to an arbitrary tilt of the imaging device.

It is a schematic block diagram explaining the whole structure of the image pickup apparatus in embodiment. It is a figure which shows the appearance in the side view of the omnidirectional (omnidirectional) image pickup apparatus in embodiment. It is a flowchart explaining the operation flow of the image pickup apparatus in embodiment. It is a figure explaining the projection relationship of the fisheye lens used for the image pickup apparatus in an embodiment, (a) the appearance of the fisheye lens in the side view, and (b) the projection function f in the plan view of the photographed image. It is a figure explaining the omnidirectional (omnidirectional) image format photographed by the image pickup apparatus in the embodiment, when (a) is represented by a plane and (b) is represented by a spherical surface. The outline of the omnidirectional (omnidirectional) image generation process taken by the image pickup apparatus in the embodiment was converted by (a1) (a2) images taken by two fisheye lenses and (b1) (b2) conversion table. It is a figure explaining each of the image taken by a fisheye lens and (c) the image generated by combining the two images after conversion. Regarding the conversion table of the omnidirectional (omnidirectional) image taken by the imaging device in the embodiment, (a) a matrix of image coordinate values before and after conversion, (b) coordinate values of the image after conversion, and (c) an image before conversion. It is a figure explaining each of the coordinate values of. It is a flowchart explaining the operation flow of the correction processing of the conversion table with respect to the omnidirectional (omnidirectional) image taken by the image pickup apparatus in embodiment. It is the schematic explaining the inclination of the image pickup apparatus in an embodiment. It is a figure explaining the vertical correction calculation of the omnidirectional (omnidirectional) image taken by the image pickup apparatus in embodiment using (a) a camera coordinate system, and (b) a global coordinate system. It is a figure explaining another example of the vertical correction calculation of the omnidirectional (omnidirectional) image taken by the image pickup apparatus in embodiment, using (a) a camera coordinate system, (b) a global coordinate system. It is a figure explaining another example of the operation flow of the correction processing of the conversion table with respect to the omnidirectional (omnidirectional) image taken by the image pickup apparatus in embodiment. It is a figure explaining the whole structure of the image pickup system in another embodiment. It is a schematic block diagram explaining the whole structure of the electronic circuit of the transfer destination device of the image pickup system in another embodiment. It is a flowchart explaining the operation flow of the image pickup system in another embodiment. It is a figure explaining the structure of the multi-eye image pickup apparatus which captures all directions (omnidirectional) at once. It is a figure explaining the image obtained from one lens in the multi-eye image pickup apparatus of FIG. It is a figure explaining the image which joined after applying the distortion correction to the image obtained in FIG. It is a figure explaining the image at the time of taking a picture by tilting the multi-eye image pickup apparatus of FIG. It is a figure explaining the image at the time of performing distortion correction and image composition without considering the inclination of the image of FIG. It is a figure which illustrates typically the acceleration sensor which measures a tilt angle.

A mode for carrying out the present invention will be described in detail with reference to the drawings. In each figure, the same or corresponding parts are designated by the same reference numerals, and the duplicated description thereof will be appropriately simplified or omitted. In the present embodiment, when the omnidirectional (omnidirectional) image pickup device generates an omnidirectional (omnidirectional) image, the vertical direction is detected and the vertical direction is adjusted with respect to the conversion table used for image processing. Make corrections. In the present embodiment, by generating the omnidirectional (omnidirectional) image using this corrected conversion table, it is not necessary to recreate the conversion table from the beginning, and the processing time is shortened. The image pickup device means a single device such as a digital camera, and the image pickup system means that a plurality of devices, for example, a digital camera and an information processing device are used separately. In the present embodiment, unless otherwise specified, the imaging device conceptually includes an imaging system.

Here, a multi-lens imaging device that captures all directions (omnidirectional) at once will be described. FIG. 16 is a diagram illustrating a configuration of a multi-eye imaging system that captures all directions (omnidirectional) at once. In FIG. 16, an imaging system using a plurality of wide-angle lenses is created in order to photograph all directions (omnidirectional) at one time.

For example, in the multi-lens imaging device on the left side of FIG. 16, when a fisheye lens (ultra-wide-angle lens) having an angle of view of 180 degrees or more with one imaging lens is used, two eyes (two different imaging directions) are used as the minimum configuration. It is possible to shoot all directions (all celestial spheres) at once only by the configuration of the image pickup lens). If you want to reduce image distortion as much as possible even if you have other than two eyes, you can selectively use only the part with good image quality in the central part, and use the four eyes (four different imaging directions) shown on the right side of FIG. It is also possible to increase the number of image pickup lenses by adopting the configuration of the image pickup lens to have. In this four-eye imaging device, it is desirable to use a lens in which one eye has an angle of view of 90 degrees or more, preferably about 100 degrees.

In the following description, the basic concept can be applied to the four-eye configuration shown on the right side of FIG. 16, but in order to make the basic concept easier to understand, the following description illustrates an imaging device having a two-eye configuration. Used for.

In this image pickup apparatus, it is assumed that the vertical central axis of the image pickup apparatus coincides with the vertical axis of the celestial sphere. For example, an image obtained from each lens using two fisheye lenses is as shown in FIG. FIG. 17 is a diagram illustrating an image obtained through one lens in the two-eye imaging device on the left side of FIG. In this way, it is preferable that the image is taken so that the zenith recognized by the photographer and the horizon coincide with each other.

The 180-degree boundary in the figure means that in an imaging system having two lenses (two imaging directions), images are taken so as to have regions that overlap each other, and the overlapping regions are used to capture an image. Make a connection. The portion outside the 180-degree boundary is the overlapping region. Therefore, when two lenses are used, one lens side has an angle of view of 180 degrees or more, preferably about 190 degrees, and an overlapping region is provided. For example, in the case of four eyes, if the angle of view is around 100 degrees, an appropriate overlapping area can be set.

Even in the configuration using four lenses shown on the right side of FIG. 16, when the image pickup device is correctly oriented vertically and photographed, the horizontal line is correctly photographed horizontally. The position of the zenith also matches. A method of joining images taken in this way to perform distortion correction is widely known.

For example, when a plurality of images are corrected for distortion and the maps are connected in a Mercator projection manner, an image as shown in FIG. 18 below can be obtained. FIG. 18 is a diagram illustrating an image obtained by applying distortion correction to the image obtained in FIG. 17 and then joining the images.

When the image pickup device is correctly placed in the vertical direction and photographed without tilting, it is possible to obtain an image in which the horizontal line is correctly straight as shown in FIG. 18 simply by performing distortion correction and connecting. When the image pickup device is attached to some kind of fixture and the horizontal / vertical adjustment is performed using a spirit level or the like, the image can be correctly photographed in the vertical direction.

FIG. 19 is a diagram illustrating an image taken when the multi-eye and two-eye image pickup devices shown on the right side of FIG. 16 are tilted. In general, when a person holds it in his hand and shoots it, it is difficult to shoot horizontally and vertically. FIG. 19 is an example of images taken in such a situation, in which the zeniths of the images are displaced from each other and the horizon is distorted. When the image taken at an inclination is corrected for distortion and connected without considering the inclination in this way, as shown in FIG. 20, the distortion of the horizontal line appearing in FIG. 19 is reflected as it is. It ends up.

As shown in FIG. 20, when the image pickup device is tilted and the distortion correction and the image joining are performed without considering the tilt angle, the horizontal line becomes a trigonometric curve, and the angle is orthogonal. The sex also collapses. Therefore, it is necessary to make a correction in consideration of the inclination angle from the vertical direction.

Here, the principle of measuring the tilt angle will be described with reference to FIG. FIG. 21 is a diagram schematically illustrating an acceleration sensor that measures a tilt angle. An acceleration sensor as shown in FIG. 21 is built in the image pickup device, and the degree of inclination of the image pickup device from the vertical direction is measured by using the gravitational acceleration sensor.

FIG. 21 illustrates an outline of angle acquisition using a uniaxial accelerometer in order to show that only the inclination in the plane including the central axis of the lens surface is acquired in the binocular configuration for simplification. ing. However, when the user actually shoots, it is assumed that the shooting is performed at an angle twisted from the above-mentioned plane. Therefore, as the acceleration sensor, a three-axis one is used so that the twist angle from the plane including the two lens center planes can also be measured.

Next, the image pickup apparatus in this embodiment will be described in detail. FIG. 1 is a schematic block diagram illustrating an overall configuration of an electronic circuit of an imaging device according to an embodiment. In FIG. 1, the image sensor (hereinafter, also referred to as “digital camera”) 100 includes an image sensor 1 109, an image sensor 2 110, and an SDRAM (Synchronous Dynamic Random Access Memo).
ry) 111, the external storage 112, and the acceleration sensor 113 are connected to the controller 10, respectively.

Here, in the present embodiment, two image pickup elements (so-called two eyes) are used to acquire an omnidirectional image, but the number of the image pickup elements may be three or more. When there are three or more image pickup devices, the lens corresponding to the image pickup device described with reference to FIG. 2 does not need to have an angle of view of 180 degrees or more, and the angle of view can be adjusted as appropriate. In general, a wide-angle lens including a fisheye lens is used as the lens. Further, the image pickup device may be one that can acquire an image that covers 360 degrees in the horizontal direction, not limited to all directions.

The controller 10 includes a CPU (Central Processing Unit) 101, a SRAM (Static RAM) 102, a ROM (Read Only Memory) 103, an image processing block 104, an SDRAM I / F (Inter / Face) 105, and an external storage. It includes an I / F 106 and an external sensor I / F 107.

In this embodiment, the image processing block 104 performs general image processing such as distortion correction and pixel defect correction, the CPU reads a predetermined table or program, and corrects in the vertical direction by tilting the digital camera 100. The explanation is given by performing the processing. However, it goes without saying that the correction process in the vertical direction may be performed by the image processing block 104.

At the time of shooting, the image sensor 1 109 and the image sensor 2 110 input the digitized image data to the image processing block 104 of the controller 10. The input image data is image-processed using the image processing block 104, the CPU 101, the SRAM 102, the SDRAM 111, and the like, and finally stored in the external storage 112. Examples of the external storage include a compact flash (registered trademark), an SD (Secure Digital) memory, and the like.

Further, the controller 10 may be provided with a USB (Universal Serial Bus) connection I / F for connecting to an external device or a wired or wireless network I / F for connecting to a network. The conversion table, the correction processing program for the conversion table, and the processing program for the vertical correction calculation, which will be described later, are stored in the SRAM 102 and the SDRAM 111.

The acceleration sensor 113 is used to detect the tilt of the digital camera 100 during shooting. As a result, the tilt direction of the digital camera 100 can be detected instantly and easily.

The acceleration sensor 113 is a three-axis acceleration sensor that detects acceleration in three directions orthogonal to each other in the vertical, horizontal, and front-back directions with respect to the digital camera 100. For example, when the digital camera 100 is held by hand so as to be stationary, the acceleration sensor 113 detects only the gravitational acceleration.

Therefore, when the acceleration in the vertical direction only in the downward direction is detected, it can be seen that the vertical direction of the digital camera 100 and the vertical direction of the ground surface coincide with each other. That is, it can be seen that the digital camera is held horizontally as if the digital camera were generally operated.

When tilted from the top and bottom directions, the acceleration sensor 113 detects acceleration in the left-right and front-back directions according to the tilt direction, and depends on the ratio of the magnitudes of the up-down, left-right, and front-back accelerations. The tilt angle of the digital camera 100 can be obtained.

Next, an omnidirectional (omnidirectional) imaging device will be described. FIG. 2 is a diagram showing the appearance of the omnidirectional (omnidirectional) imaging device according to the embodiment in a side view.

In the present embodiment, it is intended to generate an omnidirectional (omnidirectional) image by an omnidirectional (omnidirectional) imaging device capable of photographing all directions from a shooting point. That is, the omnidirectional (omnidirectional) imaging device (digital camera) can shoot in all directions that can be seen from the shooting point.

The digital camera 100 (FIG. 1), which is an omnidirectional (omnidirectional) image pickup device, is composed of two image pickup elements 1 109 and image pickup elements 2 110 equipped with a fisheye lens, which is an example of a wide-angle lens having an angle of view exceeding 180 degrees. Take an image. Images taken by the two image sensors via the two fisheye lenses have portions that overlap each other. Therefore, an omnidirectional (omnidirectional) image is created by performing image processing such as predetermined distortion correction and then converting and joining the two images.

Next, the operation of the omnidirectional (omnidirectional) imaging device in the embodiment will be described. FIG. 3 is a flowchart illustrating an operation flow of the image pickup apparatus according to the embodiment. The operation from the input of the captured image to the storage of the image in the external storage 112 (FIG. 1) will be described with reference to FIG.

In step (hereinafter, referred to as “S”) 301, the acceleration sensor 113 detects the tilt angle of the digital camera 100. Then, in S302, the controller 10 reads, for example, the conversion table stored in the SDRAM 111 according to the tilt angle of the digital camera 100 detected in S301, and corrects the conversion table by a predetermined correction method. The correction method of the conversion table will be described later.

Next, in S303, two digitized fisheye images taken by the image sensor 1 (numbering 109) and the image pickup element 2 (numbering 110) are input to the image processing block 104. In this image processing block 104, general image processing such as distortion correction is performed. Then, in S304, the controller 10 converts the two captured fisheye images (images as obtained in FIG. 17) using the conversion table corrected in S302. The conversion method will be described later.

Next, in S305, the controller 10 generates a synthesized omnidirectional (omnidirectional) image by using the overlapping portion of the two images converted in S304. Then, in S306, the controller 10 stores the omnidirectional (omnidirectional) image generated in S305 in the external storage 112 via the external storage I / F 106.

Next, the projection relationship of a fisheye lens, which is an example of a wide-angle lens used in a digital camera in the present embodiment, will be described. FIG. 4 is a diagram illustrating (a) the appearance of the fisheye lens in the side view and (b) the projection function f in the plan view of the captured image, respectively, regarding the projection relationship of the fisheye lens used in the digital camera in the embodiment.

The image taken through the fisheye lens having an angle of view exceeding 180 degrees is a shot image of a scene substantially hemisphere from the shooting position as shown in FIG. Then, as shown in FIGS. 4A and 4B, an image is generated at an image height h corresponding to the incident angle θ, and the relationship between the incident angle θ and the image height h is determined by the projection function f. (H = f (θ)). The projection function differs depending on the properties of the fisheye lens.

Examples of the projection conversion method (function) include a central projection method, a stereographic projection method, an equidistant projection method, an equidistant angle projection method, and a normal projection method. The central projection method is a method used when shooting with a digital camera equipped with a lens having a normal angle of view, and the other four methods are digital cameras using a wide-angle lens having an ultra-wide angle of view such as a fisheye lens. This is the method used in.

Next, the format (expression method) of the omnidirectional (omnidirectional) image taken by the digital camera in the present embodiment will be described. FIG. 5 is a diagram illustrating an omnidirectional (omnidirectional) image format captured by a digital camera in the present embodiment when (a) is represented by a plane and (b) is represented by a spherical surface. ..

In FIG. 5, the format of the omnidirectional (omnidirectional) image is an angle having a horizontal angle of 0 to 360 degrees and a vertical angle of 0 to 180 degrees when developed on a plane, as shown in FIG. 5 (a). It is an image having pixel values corresponding to coordinates. The angular coordinates are associated with each of the coordinate points on the spherical surface shown in FIG. 5B, and are similar to the latitude and longitude coordinates on the globe.

For the relationship between the plane coordinate values of the image taken with the fisheye lens and the spherical coordinate values of the omnidirectional (omnidirectional) image, the projection function f (h = f (θ)) as described in FIG. 4 is used. Can be associated with. As a result, two images taken with a fisheye lens are converted and combined (combined) to create an omnidirectional (omnidirectional) image as shown in FIGS. 5 (a) and 5 (b). Can be done.

Next, the process of generating an omnidirectional (omnidirectional) image using an image actually taken with a fisheye lens will be described. FIG. 6 shows an outline of the omnidirectional (omnidirectional) image generation process taken by the digital camera in the embodiment, (a1) and (a2) images acquired by each image sensor via two fisheye lenses, (b1). (B2) The image converted by the conversion table (corresponding to FIG. 5 (a)), and (c) the image generated by combining (combining) the two converted images (in FIG. 5 (b)). Correspondence), is a figure explaining each.

In FIG. 6, the images (a1) and (a2) taken by each image sensor via the two fisheye lenses as shown in FIG. 17 are the processing of S304 described in FIG. 3, that is, the corrected conversion table. Is converted into (b1) and (b2), respectively, by the image conversion process using. At this point, the images (b1) and (b2) have an expression format that matches the omnidirectional (omnidirectional) image format, that is, an expression corresponding to FIG. 5A.

The process of S305 in FIG. 3, that is, the two converted images are combined to generate an omnidirectional (omnidirectional) image. That is, the image (c), that is, the omnidirectional (omnidirectional) image is generated by superimposing and synthesizing the overlapping portion of the images (b1) and (b2) as a key.

Next, the correction method of the conversion table described in S304 of FIG. 3 will be described. FIG. 7 shows a conversion table of omnidirectional (omnidirectional) images taken by a digital camera in the present embodiment.

FIG. 7A is a diagram for explaining a conversion table (also referred to as conversion data) showing a matrix of image coordinate values before and after conversion. Further, FIGS. 7 (b) and 7 (c) are diagrams for explaining the relationship between (b) the coordinate values of the image after conversion and (c) the coordinate values of the image before conversion.

As shown in FIG. 7A, the conversion table used for the image conversion described in S304 of FIG. 3 is the coordinate values (θ, φ) (pix: pixels) of the converted image and the corresponding pre-conversion. It has a dataset with the coordinate values (x, y) (pix) of the image for all the coordinate values of the converted image. Here, the table-like data structure is shown as the conversion table, but it does not necessarily have to be the table-like data structure. That is, it may be converted data.

A converted image is generated from the captured image (pre-conversion image) according to the conversion table shown in FIG. 7 (a). Specifically, as shown in FIGS. 7 (b) and 7 (c), each pixel of the converted image is divided into coordinate values (from the correspondence between the conversion table before conversion and the conversion table (FIG. 7 (a)) after conversion. It is generated by referring to the pixel values of the coordinate values (x, y) (pix) of the pre-conversion image corresponding to θ, φ) (pix).

Next, a process of correcting the conversion table for the omnidirectional (omnidirectional) image taken by the digital camera in the present embodiment according to the tilt of the digital camera will be described. FIG. 8 is a flowchart illustrating an operation flow of correction processing of a conversion table for an omnidirectional (omnidirectional) image taken by a digital camera in the present embodiment.

In FIG. 8, in S801, the controller 10 acquires camera tilt parameters (α, β) based on the tilt value of the digital camera 100 detected by the acceleration sensor 113 (FIG. 1). Here, in the camera tilt parameters (α, β), α and β each represent the angle of rotation, but the details will be described later.

Next, in S802, the input values (θ1, φ1) of the conversion table are set. In FIG. 8, the camera coordinate system is expressed as (θ0, φ0) and the global coordinate system is expressed as (θ1, φ1), and the parameter values (θ, φ) depending on the coordinate system are described separately. .. That is, here, the global coordinate system (θ1, φ1) is set.

Next, in S803, the global coordinate system (θ1, φ1) of the input value is converted to the camera coordinate system (θ0, φ0) by the vertical correction calculation by the controller 10. The vertical correction calculation will be described later.

Next, in S804, the converted camera coordinate system (θ0, φ0) is converted into the coordinate values (x, y) of the unconverted image using the conversion table before correction (FIG. 7 (a)). To. In addition, a conversion table for generating an accurate omnidirectional (omnidirectional) image created under the condition that the camera is not tilted is prepared in advance, and a predetermined storage unit (RAM) or It is necessary to store it in SDRAM).

Next, in S805, the controller 10 makes the input values (θ1, φ1) of the global coordinate system correspond to the finally calculated coordinate values (x, y) before correction in the converted conversion table after correction. Save as a coordinate set.

Next, in S806, the controller 10 receives an unprocessed input value (θ1, φ1), that is, an input value (θ1, φ1) of the global coordinate system for which the corresponding uncorrected coordinate value (x, y) has not been calculated. ) Remains or not. If it remains (S806: YES), the process returns to S802 so that the input values (θ1, φ1) of the global coordinate system are set as the next values.

If there is no remaining (S806: NO), the process ends. In this case, the controller 10 has the input values (θ1, φ1) of the global coordinate system as the coordinate values, and the coordinate values (x, y) before correction corresponding to each pixel of the omnidirectional (omnidirectional) image format. The calculation is complete.

Next, the inclination of the digital camera 100 in this embodiment will be described. FIG. 9 is a schematic view illustrating the inclination of the digital camera in the present embodiment.

In FIG. 9, the vertical direction coincides with the z-axis in Cartesian coordinates in the xyz three-dimensional direction of the global coordinate system. When this direction matches the direction of the digital camera shown in FIG. 9, the camera is not tilted. If they do not match, the digital camera is tilted.

That is, the tilt angle α from the gravity vector and the tilt angle β in the xy plane can be obtained by the following equations using the output of the acceleration sensor. Here, Ax is the value of the x0-axis direction component of the camera coordinate system of the acceleration sensor, Ay is the value of the y0-axis direction component of the camera coordinate system of the acceleration sensor, and Az is the value of the z0-axis direction component of the camera coordinate system of the acceleration sensor. The value.

Next, the vertical correction calculation will be described. FIG. 10 is a diagram illustrating a vertical correction calculation of an omnidirectional (omnidirectional) image taken by a digital camera in the present embodiment using (a) a camera coordinate system and (b) a global coordinate system. ..

In FIG. 10, the three-dimensional Cartesian coordinates of the global coordinate system are expressed as (x1, y1, z1), the spherical coordinates are expressed as (θ1, φ1), and the three-dimensional Cartesian coordinates of the camera coordinate system are expressed as (x0, y0, z0). The spherical coordinates are expressed as (θ0, φ0).

Using the equation shown in FIG. 10, conversion from spherical coordinates (θ1, φ1) to spherical coordinates (θ0, φ0) is performed. First, in order to correct the inclination, it is necessary to perform rotational conversion using three-dimensional Cartesian coordinates. Therefore, using equations (1) to (3) in FIG. 10, spherical coordinates (θ1, φ1) to 3 Convert to dimensional Cartesian coordinates (x1, y1, z1).

Next, using the camera tilt parameters (α, β), the global coordinate system (x1, y1, z1) is converted to the camera coordinate system (x0, y0) by the rotating coordinate transformation (equation (4)) shown in FIG. , Z0). In other words, this equation (Equation (4) shown in FIG. 10) defines the camera tilt parameters (α, β).

This means that the global coordinate system becomes a camera coordinate system when first α-rotated around the z-axis and then β-rotated around the x-axis. Finally, using the equation (5) shown in FIG. 10 and the equation (6) shown in FIG. 10, the three-dimensional Cartesian coordinates (x0, y0, z0) of the camera coordinate system are converted into spherical coordinates (θ0, φ0). ) Is converted back.

Next, another example of the vertical correction calculation will be described. FIG. 11 describes another example of vertical correction calculation of an omnidirectional (omnidirectional) image taken by a digital camera in the present embodiment using (a) a camera coordinate system and (b) a global coordinate system. It is a figure to do.

In this embodiment, the vertical correction calculation is speeded up. The calculation formulas (1) to (6) in the embodiment shown in FIG. 10 can also be written as the calculation formulas (7) to (14) in FIG.

That is, since the rotations α and γ about the z-axis are the rotations of θ in the spherical coordinates (θ, φ), the rotation conversion can be performed by simple addition and subtraction without converting to the Cartesian coordinate system. It enables high speed. Therefore, only the rotation transformation of the rotation β about the x-axis requires the transformation using the Cartesian coordinate system. As a result, the calculation speed can be increased.

Next, another example of the correction processing according to the tilt of the digital camera of the conversion table for the omnidirectional (omnidirectional) image taken by the digital camera in the present embodiment will be described. FIG. 12 is a diagram illustrating another example of the operation flow of the correction processing of the conversion table for the omnidirectional (omnidirectional) image taken by the digital camera in the present embodiment.

In this embodiment, the correction process for the conversion table can be speeded up. That is, also in the embodiment shown in FIG. 11, the correction process of the conversion table is performed by speeding up the vertical correction calculation, but in the present embodiment, the speed is increased by omitting the vertical correction calculation. Try.

Explaining the correction processing of the conversion table with reference to FIG. 12, in S1201, the camera tilt parameters (α, β) are acquired. This process is the same as the process of S801 of FIG.

Next, in S1202, the conversion table corresponding to the value of the camera tilt parameter (α, β) is acquired and the process is completed. That is, by holding a plurality of conversion tables so as to have different values according to the camera tilt parameters (α, β), the processing of the vertical correction calculation is omitted.

That is, the camera tilt parameter (α, β) is a three-dimensional real number vector in principle, but the conversion table is prepared only for a specific (α, β) and detected (α). , Β) By adopting the table closest to), it is possible to deal with all cases. Alternatively, it is also effective to extract a plurality of tables close to the detected (α, β) and perform interpolation operations such as weighting and difference according to the proximity. As a result, the conversion table can be corrected only by a relatively simple operation called an interpolation operation, and the processing required for the operation can be suppressed.

Next, the overall configuration of the imaging system according to another embodiment will be described. FIG. 13 is a schematic block diagram illustrating an overall configuration of an imaging system according to another embodiment. In the above-described embodiment, the tilt is corrected in the spherical image pickup device (digital camera), but it is not always necessary to correct the tilt in the spherical image pickup device.

As shown in FIG. 13, by adding a wired or wireless communication function to the all-sky imaging device, it is possible to perform image processing on a computer or a portable information terminal (smartphone or tablet PC). The image before correction is transferred to the information processing device. Then, it is also possible to perform a process of correcting the inclination in the information processing device (transfer destination device) of the transfer destination.

However, the tilt correction process requires tilt information of the spherical image pickup device. Since the tilt information of the spherical image pickup device can be detected only by the image pickup device main body, it is necessary to transfer the tilt information together with the image before correction. Further, the conversion table may be held in the image pickup device of the whole celestial sphere and transferred together with the image, or the transfer destination device recognizes the image pickup device in advance (for example, the transfer destination device recognizes the image pickup device) as a preparation before shooting. At that time, or when the image pickup apparatus recognizes the transfer destination device), the transfer may be performed to the transfer destination device (transfer destination device).

Further, the conversion table does not need to be transferred together with the image every time, and the transfer destination device may request the conversion table or check whether the conversion table is up-to-date before transferring. Furthermore, the conversion table does not necessarily have to be transmitted from the image pickup device, for example, it may be made available for download from the WEB site of the manufacturer selling the image pickup device and stored in the transfer destination device. .. The conversion table transferred from the imaging device may be customized to a conversion table suitable for each imaging device.

Next, the transfer destination device 200 of the imaging system according to another embodiment will be described. FIG. 14 is a schematic block diagram illustrating an overall configuration of an electronic circuit of a transfer destination device 200 of an imaging system according to another embodiment. In FIG. 14, the SDRAM 1411 and the external storage 1412 are connected to the controller 140, respectively.

The controller 140 includes a CPU 1401, a SDRAM 1402, a ROM 1403, an image processing block 1404, an SDRAM I / F 1405, and an external storage I / F 1406. Further, the controller 140 includes a USB connection I / F 1407 for connecting to an external device and a wired or wireless network I / F 1408 for connecting to a network.

In the above-described embodiment, it has been described that the image processing block 1404 performs general image processing such as distortion correction and pixel defect correction. Further, it has been described that the CPU 1401 reads a predetermined table or program and performs a correction process in the vertical direction by tilting the digital camera 100. However, it goes without saying that the correction process in the vertical direction may be performed by the image processing block 1404.

Next, the operation of the imaging system in another embodiment will be described. FIG. 15 is a flowchart illustrating an operation flow of the imaging system according to another embodiment. FIG. 15 explains that after the captured image is input, the image is stored in the storage of the transfer destination device.

First, processing in the digital camera 100 is performed. In S1501, the tilt angle of the digital camera 100 is detected by the acceleration sensor 113 (FIG. 1). Then, in S1502, two fisheye images (images as shown in FIG. 17) captured and digitized by the image sensor 1 (numbering 109) and the image sensor 2 (numbering 110) by the controller 10 are shown in FIG. As shown in the above, the image is transferred to the transfer destination device 200, for example, a mobile information terminal or a computer via the USB connection I / F 1407 or the network I / F 1408. At the same time, the tilt information of the digital camera and the conversion table are transferred to the transfer destination device 200 via the USB connection I / F 1407 or the network I / F 1408.

As described above, the conversion table may be transferred when the digital camera 100 and the transfer destination device 200 recognize each other. Further, it is sufficient that the conversion table is transferred to the transfer destination device 200 once, and the conversion table does not need to be transferred every time.

The conversion table is stored in, for example, an SDRAM 1411 connected to the digital camera 100, and is read and transferred from the conversion table. The process up to this point is the processing in the digital camera 100. From the next step, the process is executed in the transfer destination device 200.

In S1503, the conversion table is corrected by using a predetermined correction method in the controller 140 according to the tilt information which is the angle information of the transferred digital camera. The correction method of the conversion table is as described in the above-described embodiment. Next, in S1504, the fisheye image by the two transferred image sensors is input to the image processing block 1404 of the transfer destination device 200.

In this image processing block 1404, general image processing such as distortion correction is performed. Then, in S1505, the controller 140 converts two fisheye images using the conversion table corrected in S1503. The conversion method is the same as that of the above-described embodiment.

Next, in S1506, the controller 140 generates a combined omnidirectional (omnidirectional) image by using the overlapping portion of the two images converted in S1505. Then, in S1507, the omnidirectional (omnidirectional) image generated in S1506 is stored in the external storage 1412 by the controller 140 via the external storage I / F 1406.

The operations related to the flowcharts in the embodiments shown in FIGS. 3, 8, 12 and 15 can be executed by a program on a computer. That is, the CPU (control circuit) 101 (FIG. 1) that controls the operation of the image pickup apparatus or the CPU (control circuit) 1401 (FIG. 14) that controls the operation of the transfer destination device 200 is the ROM 103, 1403, SRAM 102, 1402. This is done by loading the program stored in the storage medium and sequentially executing the processing steps of the program.

As described above, in the omnidirectional (omnidirectional) imaging device or imaging system, the vertical direction is detected, the conversion table used for image processing is corrected according to the vertical direction, and the corrected conversion is performed. By generating omnidirectional (omnidirectional) images using the table, it is not necessary to recreate the conversion table from the beginning, so that the processing time can be shortened.

The above description has been made according to a preferred embodiment. Although specific examples have been described here, various modifications and changes can be made to these specific examples without departing from the broad purpose and scope defined in the claims.

10, 140 controller 100 image pickup device 101, 1401 CPU
102, 1402 SRAM
103, 1403 ROM
104, 1404 Image processing block 105, 1405 SDRAM I / F
106, 1406 External storage I / F
107 External sensor I / F
109 Image sensor 1
110 image sensor 2
111, 1411 SDRAM
112, 1412 External storage 113 Accelerometer 200 Transfer destination device 1407 USB connection I / F
1408 Network I / F

Japanese Unexamined Patent Publication No. 2003-223633 Japanese Unexamined Patent Publication No. 2006-059202 Japanese Unexamined Patent Publication No. 11-309137 Japanese Patent No. 4175832

Claims (10)

An image pickup device equipped with an image pickup device that captures an image input through a wide-angle lens.
A tilt detecting means for detecting the tilt of the image pickup device with respect to the gravity direction each time the image sensor takes a picture.
A storage means for storing conversion data for converting a plane image represented by plane coordinate values taken by the image sensor into a spherical image represented by spherical coordinate values.
A generation means for generating a spherical image converted by the conversion data is further provided.
The image pickup device is characterized in that the conversion data is corrected for each shooting according to the tilt of the imaging device with respect to the gravity direction detected for each shooting by the tilt detecting means. The slope is a slope vector in the three-dimensional direction with respect to the vertical direction. Conversion data corresponding to the slope vector is prepared in advance, and the conversion data corresponding to the slope vector closest to the detected slope is used. The imaging apparatus according to claim 1, wherein the imaging apparatus is extracted from conversion data prepared in advance. The slope is a slope vector in the three-dimensional direction with respect to the vertical direction. Conversion data corresponding to the slope vector is prepared in advance, and the conversion data corresponding to the slope vector closest to the detected slope is used. The imaging device according to claim 1, wherein the imaging device is extracted from the conversion data prepared in advance, and the difference between the extracted conversion data and the conversion data corresponding to the detected inclination is interpolated. The imaging device according to any one of claims 1 to 3, wherein the inclination is detected by an acceleration sensor. An imaging system including an imaging device and an information processing device connected to the imaging device via a network
An image pickup device equipped with an image pickup device that captures an image input through a wide-angle lens.
A tilt detecting means for detecting the tilt of the image pickup device with respect to the gravity direction each time the image sensor takes a picture.
A storage means for storing conversion data for converting a plane image represented by plane coordinate values taken by the image sensor into a spherical image represented by spherical coordinate values.
A generation means for generating a spherical image converted by the conversion data is further provided.
The image pickup system is characterized in that the conversion data is corrected for each shooting according to the tilt of the imaging device with respect to the gravity direction detected for each shooting by the tilt detecting means. The image pickup device transfers the tilt, the conversion data, and the image captured by the image pickup device to the information processing device.
The information processing apparatus corrects the conversion data according to the transferred inclination, performs coordinate conversion of the transferred image based on the corrected conversion data, and obtains the image after the coordinate conversion. The imaging system according to claim 5, wherein the imaging system is combined. A step of detecting the inclination of the image pickup device having the image pickup element with respect to the gravity direction by the tilt detection means for each image pickup by the image pickup element that captures the image input through the wide-angle lens.
A step of storing the conversion data for converting the plane image represented by the plane coordinate values taken by the image sensor into the spherical image represented by the spherical coordinate values in the storage means.
A step of generating a spherical image converted by the conversion data is provided.
An image processing method characterized in that the converted data is corrected for each shooting according to the tilt of the imaging device with respect to the gravity direction detected for each shooting by the tilt detecting means. The program to be executed by the image pickup device is
The conversion data for converting the plane image represented by the plane coordinate values taken by the image pickup element that captures the image input through the wide-angle lens into the spherical image represented by the spherical coordinate values is stored in the storage means. Processing and
Including a process of generating a spherical image converted by the conversion data.
The conversion data is a program characterized in that it is corrected for each shooting according to the inclination of the imaging device with respect to the gravity direction, which is detected for each shooting. An image sensor equipped with an image sensor that captures an image input via a wide-angle lens and an information processing device connected via a network are
A means for receiving the inclination of the image pickup device with respect to the gravitational direction, which is detected for each shooting by the image pickup device.
A storage means for storing conversion data for converting a plane image represented by plane coordinate values taken by the image sensor into a spherical image represented by spherical coordinate values.
A generation means for generating a spherical image converted by the conversion data is further provided.
An information processing device characterized in that the converted data is corrected for each shooting according to the inclination of the imaging device with respect to the gravity direction detected for each shooting. A program to be executed by an image sensor equipped with an image sensor that captures an image input via a wide-angle lens and an information processing device connected via a network
The process of receiving the inclination of the image pickup device with respect to the gravity direction, which is detected every time the image sensor takes a picture
A process of storing conversion data in a storage means for converting a plane image represented by plane coordinate values captured by the image sensor into a spherical image represented by spherical coordinate values.
A generation process for generating a spherical image converted by the conversion data is included.
A program characterized in that the conversion data is corrected for each shooting according to the inclination of the imaging device with respect to the gravity direction detected for each shooting.