JP2020039062A - Image processing apparatus, image processing method, and program - Google Patents

Abstract

To allow for generation of a composite image in which an angle of view according to a focal length of a photographing lens is maintained, when a plurality of photographed images are converted to cylindrical coordinates and composited.SOLUTION: The image processing apparatus includes: acquiring means (116) acquiring a plurality of images photographed in such a manner that the image include common areas in adjacent photographing directions while changing the photographing direction sequentially; determining means (123) determining a radius of a virtual cylinder; conversion means (116e) converting each of the images into an image projected on a virtual cylinder by cylindrical coordinate conversion; detecting means (116c) detecting an amount of movement between adjacent photographing directions; positioning means (116d) positioning each image obtained by performing the cylindrical coordinate conversion on the basis of the movement amount; and compositing means (116j) superimposing the common areas of the positioned images and compositing the images so as to sequentially connect the images to each other. A radius of the virtual cylinder is determined to be larger than a focal length of a photographing lens of an imaging apparatus.SELECTED DRAWING: Figure 1

Description

  The present invention relates to an image processing technique for combining a plurality of captured images.

Conventionally, a plurality of images are captured in such a manner that a common region is captured in adjacent imaging directions while sequentially changing the imaging direction of an imaging device, and the common regions of adjacent images are sequentially superimposed and synthesized to create a wide area. 2. Description of the Related Art A technique for generating an image of a shooting range (see Patent Document 1) is known.
On the other hand, when a plurality of images are combined to generate an image with a wide shooting range, it is important to set an appropriate synthesis processing method according to the shooting method. A technique for performing cylindrical coordinate conversion on a virtual cylindrical surface having a radius is disclosed.

JP 2005-328497 A JP-A-11-331696

  However, in the cylindrical coordinate conversion, the conversion amount from the center of the image is larger for a wide-angle lens having a shorter focal length than a lens having a longer focal length. For this reason, when a wide-angle lens is used as a photographing lens, a portion that can be used in the first and last photographed images among a plurality of sequentially photographed images is reduced, and a composite image finally generated is reduced. There is a problem that the effective height becomes narrow.

  In view of the above, an object of the present invention is to make it possible to generate a combined image that maintains an angle of view corresponding to the focal length of a photographing lens when a plurality of photographed images are converted by cylindrical coordinates and combined.

  The image processing apparatus according to the present invention includes: an acquiring unit configured to acquire a plurality of images photographed so as to include a common area in an adjacent photographing direction while sequentially changing a photographing direction of the image capturing apparatus; and determining a radius of a virtual cylinder. Means, a conversion means for converting the plurality of images into cylindrical coordinates converted to an image projected on the virtual cylinder, a detection means for detecting a movement amount between adjacent photographing directions, and based on the detected movement amount. Positioning means for performing positioning of each image subjected to the cylindrical coordinate conversion, and combining means for combining the images so as to sequentially connect the images by superimposing the common area of each of the images subjected to the positioning, Wherein the determining means determines a radius of the virtual cylinder to be larger than a focal length of a photographing lens of the imaging device.

  According to the present invention, when a plurality of captured images are converted into cylindrical coordinates and combined, it is possible to generate a combined image that maintains an angle of view corresponding to the focal length of the taking lens.

FIG. 1 is a block diagram illustrating a configuration example of an imaging device according to an embodiment. It is explanatory drawing of the concept of panorama imaging | photography. FIG. 3 is a diagram illustrating a relationship between a focal length and an effective height. FIG. 9 is an explanatory diagram of a difference between a cylindrical coordinate conversion result and a synthesis result depending on a focal length. 6 is a flowchart of a basic operation of the imaging apparatus at the time of shooting. 5 is a flowchart of a panoramic image capturing process according to the first embodiment. It is a data flow diagram at the time of panorama photography. It is a processing flowchart of panoramic imaging according to the second embodiment.

Hereinafter, preferred embodiments of the present invention will be described in detail with reference to the accompanying drawings.
<First embodiment>
FIG. 1 is a block diagram illustrating a schematic configuration of an imaging device as an application example of the image processing device according to the first embodiment. The imaging apparatus according to the present embodiment has various photographing functions, and for example, is a digital camera having a panoramic photographing function as one of them.
Before describing the detailed configuration of the imaging apparatus shown in FIG. 1, panoramic shooting will be described with reference to FIGS. 2A and 2B to FIG.

  In the panoramic shooting, a plurality of images constituting a part of the whole scene are sequentially and sequentially shot while sequentially changing the shooting direction of the imaging device to a desired direction (horizontal direction, vertical direction, or the like) (so-called panning shot sequence). Photo-taking). In other words, during panoramic shooting, the shooting direction of the imaging device is moved in a desired direction while the release button is pressed. In the imaging device, continuous shooting is performed while the release button is being pressed. Automatic photographing is performed at predetermined time intervals like photographing. Also, in panoramic shooting, some areas (common areas) overlap each other within adjacent shooting angles of view (adjacent shooting directions) in a desired direction in which the shooting direction is sequentially changed. Each image is taken. The desired direction when the photographing direction is sequentially changed in panoramic photographing is generally a horizontal direction parallel to the ground or the like, but may be another direction such as a vertical direction (gravity direction). . In the description of the present embodiment, panoramic imaging in which the desired direction in which the imaging direction is sequentially changed is a general horizontal direction is taken as an example. Further, as described later, the imaging apparatus performs a distortion correction process and a cylindrical coordinate conversion process on a plurality of images acquired by panoramic shooting, and further superimposes a common area of the images after the cylindrical coordinate conversion. A panorama image is generated by performing a combining process. The imaging apparatus of the present embodiment can generate a panoramic image of the entire shooting range wider than the angle of view of the shooting lens by such a panorama shooting function.

  FIG. 2A is a diagram schematically illustrating a state in which the imaging direction of the imaging device 201 is sequentially changed to a desired direction during panoramic imaging. As shown in FIG. 2A, at the time of panoramic shooting, for example, the user 200 sequentially changes the shooting direction of the imaging device 201 to a desired direction indicated by an arrow 210 as a center of rotation. The imaging direction may be changed by, for example, installing the imaging device 201 on a camera platform and rotating the camera platform in a horizontal direction or the like. In the following description, sequentially changing the shooting direction of the imaging device with the user (or pan head) as the center of rotation will be referred to as “swing”. Then, the imaging apparatus 201 captures a plurality of images in such a manner that some areas (common areas) are overlapped with each other within the adjacent imaging angle of view for each imaging direction changed to a desired direction by the swing. Shoot. Thereby, each image sequentially acquired by the imaging device 201 becomes an image having a common area between adjacent images in the swing direction. After that, the imaging apparatus 201 performs distortion correction and cylindrical coordinate conversion processing on each of the images, and combines the images so that the common area between adjacent images after the cylindrical coordinate conversion is overlapped to synthesize a panoramic image. Generate In other words, the synthesizing process of the panoramic image is a synthesizing process in which, from each image adjacent in the direction of the swing, partial images near the center are cut out in accordance with the rotation angle at the time of the swing, and the cut-out partial images are connected. ing.

  FIG. 2B illustrates a panoramic view of the entire scene 220, two adjacent images 221 and 222 taken by swinging the imaging device 201, and the two images 221 and 222. 9 shows an example of a synthesis result image 223 obtained by superimposing and synthesizing. Note that, in the example of FIG. 2B, only the first image 221 at the time of panoramic shooting and the second image 222 taken after the image 221 are shown. The image 221 and the image 222 each include a common area. When synthesizing a panoramic image, the synthesis is performed such that the common areas are overlapped and connected, thereby generating a synthesis result image 223. In the case of generating the panoramic image of the entire view 220 in FIG. 2B, a plurality of images that can cover the entire view 220 are captured in the same manner as described above, and the common areas of the adjacent images are sequentially overlapped and connected. Is performed.

  In addition, when generating a panoramic image, prior to the synthesizing process of each image, a process of correcting distortion or the like of a photographing lens of an imaging device is performed on each image, and a virtual cylinder having a radius corresponding to the focal length of the photographing lens. A cylindrical coordinate conversion process such as mapping above is performed. Further, for each image after the cylindrical coordinate conversion, a feature point is extracted from a common area of each image adjacent in the direction of the swing, and a movement vector indicating how much the feature point has moved is detected. The movement vector includes a horizontal component (X component, horizontal component) and a vertical component (Y component, vertical component), and is detected as a positioning parameter for matching the positions of adjacent images acquired by panoramic shooting. Is done. Then, for example, a coefficient of an affine transformation is calculated from the movement vector, and at the time of generating a panoramic image, a synthesizing process is performed based on the affine transformation coefficient such that common areas of adjacent images are overlapped so that feature points match. Done.

Hereinafter, the cylindrical coordinate conversion will be described in detail.
In the cylindrical coordinate conversion, the radius of the virtual cylinder is used as one of the cylindrical conversion parameters. It is preferable that the radius of the virtual cylinder in the cylindrical coordinate conversion be a radius corresponding to the focal length of the imaging lens of the imaging device. Therefore, the shorter the focal length of the taking lens, that is, the wider the lens, the smaller the radius of the virtual cylinder.

  In general, the coordinates (X, Y) of a point of interest after performing processing for correcting distortion of a photographing lens on a captured image are converted to coordinates (X · cos θ) on a virtual cylinder by performing cylindrical coordinate conversion. , Y · cos θ). That is, the captured image is converted into an image projected on the virtual cylinder by the cylindrical coordinate conversion. Here, assuming that the effective imaging area width of the imaging device of the imaging device is w [mm] and the focal length of the imaging lens is f [mm], the relationship between the image center, the right end of the image, and the angle θ [rad] is as follows. Equation (1).

  θ = arctan (w / 2 / f) Equation (1)

  Further, assuming that the size (pixel pitch) of one pixel of the image sensor is p [μm] and the number of pixels in the height direction is h [pix (pixel)], the effective height H [pix] of the image is represented by Expression (2). become that way.

  H = (h / 2) · cos θ Equation (2)

That is, for example, as the focal length f decreases, the angle θ increases, and thus the effective height H decreases.
As a specific example, it is assumed that the size of the imaging device of the imaging device is a so-called APS-C size having a width of 22.32 mm and a height of 14.88 mm, a pixel pitch of 3.72 μm, and a number of pixels of 6000 × 4000 pixels. In this example, when the focal length f of the photographing lens is, for example, 11 mm, the effective height H is 2809 pixels from Equations (1) and (2), which is about 70% of the height (4000 pixels) of the image sensor. become. Further, for example, when the focal length f of the photographing lens is 18 mm, the effective height H is 3400 pixels, which is about 85% of the height (4000 pixels) of the image sensor. FIG. 3 plots the corresponding point between the focal length f of the imaging lens and the ratio of the effective height, with the horizontal axis representing the focal length f and the vertical axis representing the ratio [%] of the effective height to the height of the image sensor. FIG. As shown in FIG. 3, as the focal length f of the photographing lens becomes shorter, the ratio of the effective height to the height of the image sensor decreases. In other words, this means that as the focal length of the photographing lens becomes shorter, the amount of coordinate conversion in the cylindrical coordinate conversion becomes larger. This means that the area that can be used as an image after cylindrical coordinate conversion becomes smaller as the shooting lens is a wide-angle lens. On the other hand, when performing panoramic shooting, the user attempts to shoot a wide-angle lens with a wide angle of view, that is, a short shooting distance with the focal length of the shooting lens, in order to shoot a wide range. The available effective height will be reduced.

  FIG. 4 is a diagram illustrating an example in which the effective height changes according to the focal length of the photographing lens, and the size (effective height) of the panoramic image in the height direction changes. In FIG. 4, when the focal length f of the photographing lens is 18 mm, for example, the radius of the virtual cylinder is changed to 18 mm with respect to the original image 400 before the cylindrical coordinate conversion. An image 401 represents an image obtained by converting the original image 400 into a cylindrical coordinate with the radius of the virtual cylinder being 18 mm. For example, when the focal length of the taking lens is 11 mm, cylindrical coordinate conversion is performed on the original image 400 with the radius of the virtual cylinder set to 11 mm. An image 402 represents an image obtained by converting the original image 400 into a cylindrical coordinate with the radius of the virtual cylinder being 11 mm. As can be seen by comparing these two images, the image 402 after the cylindrical coordinate conversion when the radius of the virtual cylinder is 11 mm is shorter than the image 401 after the cylindrical coordinate conversion when the radius of the virtual cylinder is 18 mm. However, an image is greatly deformed (curved) in the height direction. Therefore, the image 402 has a smaller area that can be used at the time of composition in the height direction than the image 401. That is, the panoramic image 405 obtained by combining the plurality of images 402 obtained by performing the cylindrical coordinate conversion with the radius of the virtual cylinder being 11 mm is more effective height than the panoramic image 404 obtained by combining the respective images 401 obtained by performing the cylindrical coordinate conversion with the virtual cylinder having a radius of 18 mm. Becomes a small image.

  Therefore, the imaging apparatus of the present embodiment performs the cylindrical coordinate conversion by increasing the radius of the virtual cylinder from the focal length of the photographing lens, and increases the effective height as compared with the cylindrical coordinate conversion in which the focal length is the radius of the virtual cylinder. It is possible to do. In the present embodiment, the radius of the virtual cylinder is determined based on at least the amount of movement between adjacent imaging directions when the imaging apparatus is swung to perform panoramic imaging and the focal length of the imaging lens. The movement amount between the adjacent photographing directions may be calculated based on information from an angular velocity sensor, for example, or may be calculated from each photographed image. Details of the case where the movement amount between the adjacent photographing directions is calculated based on the information of the angular velocity sensor or the case where it is calculated based on the photographed image will be described later.

Hereinafter, a process of calculating the radius of the virtual cylinder based on the amount of movement between adjacent shooting directions and the focal length of the shooting lens during panoramic shooting will be described.
At the time of panoramic shooting, as described with reference to FIG. 2A, the imaging device 201 is swung so as to rotate around the user 200 and the like, and the imaging direction is sequentially changed, and continuous imaging such as continuous shooting is performed. Image taking is performed. Then, the panoramic image is generated by cutting out partial images near the center of each image in accordance with the rotation angle β during the swing between the adjacent photographing directions from each image acquired in the panoramic shooting and sequentially combining them. You. Here, it is assumed that the size of one pixel of the image sensor is, for example, p [μm], the component (size) of the movement vector in the X direction is x [pix], and the radius of the virtual cylinder is R [mm]. Then, the angle β is represented by Expression (3).

  β = arctan (x · p / R / 1000) Equation (3)

  Therefore, the effective height H [pix] in the partial image used for synthesizing the panoramic image is represented by Expression (4).

  H = (h / 2) · cos {arctan (x · p / R / 1000)} Equation (4)

  In the present embodiment, for example, the effective height of the partial image is substantially the same when the radius R of the virtual cylinder is equal to the focal length f and when the radius R of the virtual cylinder is larger than the focal length f. If not changed, the radius R of the virtual cylinder can be changed to a value different from the focal length f. In the present embodiment, it is assumed that the effective height of the partial image does not substantially change when there is no positional deviation at the boundary portion between the adjacent images or when the positional deviation is small enough to be unnoticeable. For example, if the combined panoramic image is output at the same magnification, the effective height of the partial image must be substantially unchanged when the displacement at the boundary is, for example, less than about 0.5 to 1 pixel. I do. Also, for example, if the panoramic image is reduced and output, if the displacement at the boundary after the reduction is less than about 0.5 to 1 pixel, the effective height of the partial image does not substantially change. I do. Note that these numerical values are merely examples, and can be changed as appropriate.

  In the present embodiment, the number of pixels in the height direction of the image sensor is h [pix], the pixel pitch is p [μm], the radius of the virtual cylinder is R [mm], the focal length is f [mm], and the movement vector X is When the component is x [pix], the determination is made based on the difference between Expressions (5) and (6). That is, when the difference between the calculated value of Expression (5) and the calculated value of Expression (6) is less than a predetermined threshold (for example, less than one pixel), it is assumed that the effective height of the partial image does not substantially change. .

(h / 2) · x · cos {arctan (x · p / R / 1000) / 2} Equation (5)
(h / 2) · x · cos {arctan (x · p / f / 1000) / 2} Equation (6)

  In the above-described example, the number of pixels in the height direction of the image sensor is set to h pixels. However, since so-called camera shake also occurs at the time of shooting, it is difficult to use all the h pixels in consideration of the camera shake. Few. That is, assuming that the amount of movement due to camera shake is, for example, about 10% to 30% with respect to the height direction of the image sensor, the actual panoramic image is, for example, 90 pixels with respect to the number h of pixels in the height direction of the image sensor. % To 70%. Therefore, h in Expressions (5) and (6) may be used as 90% to 70% of the number of pixels in the height direction of the image sensor in consideration of camera shake in advance.

  As an example, the number of pixels of the image sensor is 6000 × 4000 pixels, the size of each pixel is 3.72 μm, the amount of movement due to camera shake is about 20% on the image sensor, and the amount of movement between adjacent shooting directions due to swing is calculated as the number of pixels. Assume that it is equivalent to 145 pixels. In this case, the number of pixels EP [pix] of the effective height in consideration of the camera shake is expressed by Expression (7).

  EP = 4000/2 × 0.8 · cos {arctan (145 · 3.72 / R / 1000)} formula (7)

  At this time, when the radius R of the virtual cylinder is, for example, 11 mm or 14 mm, the difference between the effective heights is 0.8 pixels, which is less than the above-described threshold (less than 1 pixel). Therefore, in this case, even if the image is photographed at the focal length f of 11 mm, there is no problem even if the cylindrical coordinate conversion with the radius of the virtual cylinder is 14 mm.

  However, for example, when the speed of the swing is increased and the movement amount between the adjacent photographing directions is equivalent to 290 pixels which is twice the above-described 145 pixels in terms of the number of pixels, the case where the radius R of the virtual cylinder is 11 mm and the case where the radius R of the virtual cylinder is 14 mm The difference between the effective heights is 3.3 pixels. That is, the difference in effective height (3.3 pixels) at this time is larger than the threshold value (for example, 1 pixel). In this case, when the panoramic images are combined, the boundary between the images becomes visible as a step. Therefore, if the swing speed during panoramic shooting is high and the amount of movement between adjacent shooting directions is large, it is necessary to keep the radius of the virtual cylinder unchanged or to reduce the amount of change in radius. Becomes When performing panoramic imaging, the imaging device is not swung before the start of imaging, and the imaging device starts swinging immediately after the start of imaging. That is, since the swing speed is slow at the start of the panoramic shooting, it is considered that the problem that the difference between the effective heights becomes large as described above hardly occurs.

  On the other hand, if the speed of the swing gradually increases after the start of the panorama shooting, the above-described problem of the difference in the effective height becomes remarkable. For this reason, in the present embodiment, the radius of the virtual cylinder at the time of cylindrical coordinate conversion is changed according to the amount of movement between adjacent photographing directions, that is, the speed of the swing. For example, when shooting with a shooting lens having a focal length of 11 mm, the radius of the virtual cylinder is set to 15 mm at the time of cylindrical coordinate conversion of the first image when the swing speed is low immediately after the start of shooting, and the radius is set to 13 mm for the second image. After the first sheet, the radius is changed so that the radius is 11 mm. By doing so, when the swing speed increases after the start of shooting, the radius of the virtual cylinder can be returned to the same value as the focal length of the shooting lens, so that it is not affected by the swing speed.

  Also, when ending panoramic photography, it is desirable to change the radius of the virtual cylinder according to the swing speed, as described above. That is, when ending the panoramic shooting, the swing speed becomes slow. For this reason, for example, when shooting is performed with a shooting lens having a focal length of 11 mm, the radius of the virtual cylinder is gradually increased to 11 mm and 13 mm as the swing speed decreases to end panoramic shooting. Then, for the last image at the time of panorama shooting, the radius of the virtual cylinder is set to, for example, 15 mm.

  In the present embodiment, as described above, the radius of the virtual cylinder is variable, and the radius of the first and last shot images is increased to increase the effective height of the panorama image. Can be prevented from being reduced in the effective height. For images in the middle of shooting other than at the start and end of panoramic shooting, the focal length of the shooting lens is set to the radius of the virtual cylinder.

  Further, the decrease in the effective height as described above becomes remarkable when the focal length (the radius of the virtual cylinder) is about 20 mm or less as shown in FIG. Therefore, a threshold value for the focal length may be prepared in advance, and the method according to the present embodiment may be applied only when photographing is performed with a wide-angle lens having a focal length shorter than the threshold value. Also, it is one of the preferable modes to increase the radius of the virtual cylinder on the ultra-wide-angle side and the wide-angle side where the focal length is shorter than the threshold value, or to reduce the coordinate conversion amount near the threshold value.

Hereinafter, the configuration of the imaging apparatus of the present embodiment illustrated in FIG. 1 will be described.
In FIG. 1, a photographing lens 101 has a focus lens and a zoom lens, and forms an optical image of a subject or the like on an imaging surface of an imaging sensor 112. The taking lens 101 may be an interchangeable lens that can be attached to and detached from the imaging device main body, or may be a lens fixed to the imaging device. The diaphragm 103 is driven by a diaphragm driving circuit 104 to adjust the amount of incident light via the photographing lens 101. The aperture drive circuit 104 changes the optical aperture value of the aperture 103 based on the aperture drive amount calculated by the microcomputer 123. The AF (autofocus) drive circuit 102 includes, for example, a DC motor and a stepping motor, and drives the focus lens of the photographing lens 101 to perform focusing based on a focus control signal from the microcomputer 123.

  The main mirror 105 is a mirror for switching a light beam incident from the photographing lens 101 between the finder unit side and the image sensor 112 side. The main mirror 105 is normally disposed so as to reflect the light beam toward the finder unit side. However, when an image is taken or a live view display is performed, the light beam from the photographing lens 101 is reflected by the image sensor 112. Is jumped upward so as to be incident on. The mirror driving such as flipping up the main mirror 105 is performed by the mirror driving circuit 107 under the control of the microcomputer 123. The main mirror 105 is a half mirror so that a central portion thereof can transmit a part of light.

  The pentaprism 108 constitutes a part of a finder section, and is a prism for guiding a light beam incident from the imaging lens 101 and reflected by the main mirror 105 to an eyepiece (not shown). The eyepieces (not shown) are each provided with a focusing plate (not shown), an eyepiece lens, and the like.

  The sub-mirror 106 reflects a light beam transmitted through a half mirror provided at the center of the main mirror 105, and detects a focus detection sensor (not shown) and an exposure amount (not shown) provided in the exposure amount calculation circuit 109. This is a mirror for guiding to the detection sensor.

  The exposure amount detection sensor photoelectrically converts incident light transmitted through the half mirror at the center of the main mirror 105 and reflected by the sub-mirror 106, and sends the light to the exposure amount calculation circuit 109. The exposure amount calculation circuit 109 calculates an exposure amount from the detection output of the exposure amount detection sensor, and outputs a signal of the calculated exposure amount to the microcomputer 123. The microcomputer 123 controls the aperture driving amount, the shutter speed, the exposure time, and the like based on the signal of the exposure amount.

  The focus detection sensor receives the light flux transmitted through the central half mirror of the main mirror 105 and reflected by the sub mirror 106, and sends the sensor output to the microcomputer 123. The microcomputer 123 calculates a defocus amount from the sensor output, performs a focus calculation based on the defocus amount, generates a focus control signal, and controls the AF drive circuit 102 based on the focus control signal.

  The focal plane shutter (hereinafter, referred to as a shutter 110) is driven by a shutter driving circuit 111 under the control of the microcomputer 123. That is, the opening time of the shutter 110 is controlled by the microcomputer 123.

  The image sensor 112 is an image sensor composed of a CCD, a CMOS sensor, or the like, which is driven and controlled by the microcomputer 123, and converts a subject image formed by the photographing lens 101 into an electric signal. The AD converter 115 converts an analog output signal from the image sensor 112 into a digital signal under the control of the microcomputer 123. The digital signal output from the AD converter 115 is sent to the image processing circuit 116.

  The image processing circuit 116 performs filter processing, color conversion processing, gamma processing, and the like on the digitized image data, performs compression processing to JPEG or the like, and outputs the result to the memory controller 119. At this time, the image processing circuit 116 can also temporarily store the image data being processed in the buffer memory 122 via the memory controller 119. Further, the image processing circuit 116 can output image data captured by the image sensor 112 and image data input from the memory controller 119 to the display unit 118 through the display driving circuit 117. Switching of these functions in the image processing circuit 116 is performed according to an instruction from the microcomputer 123. Further, the image processing circuit 116 can output information such as exposure information and white balance at the time of imaging by the imaging sensor 112 to the microcomputer 123 as necessary. Based on the information, the microcomputer 123 issues a white balance or gain adjustment instruction.

  When a continuous shooting operation such as the panoramic shooting described above is performed, the image processing circuit 116 reads out the unprocessed image data once stored in the buffer memory 122 via the memory controller 119 and then reads out the processed image data. And compression processing. The storage in the buffer memory 122, the image processing, and the compression processing are continued while a continuous photographing operation such as panoramic photographing is performed. It should be noted that the number of images that can be subjected to continuous image capturing depends on the capacity of the buffer memory 122 and the number of images that are captured during panoramic image capturing.

  The image processing circuit 116 is realized by a logic device such as a gate array, and has a luminance adjustment circuit 116a, a gamma correction circuit 116b, a development circuit 116k, a compression / decompression circuit 116l, and the like. The image processing circuit 116 also includes a movement amount calculation circuit 116c, a positioning circuit 116d, a geometric conversion circuit 116e, a scaling circuit 116f, a trimming circuit 116g, a combining circuit 116j, and the like. The developing circuit 116k performs a developing process. The brightness adjustment circuit 116a adjusts brightness by a digital gain. The gamma correction circuit 116b adjusts luminance according to gamma characteristics. The compression / decompression circuit 116l performs conversion into a general image format such as JPEG. The moving amount calculation circuit 116c calculates the moving amount between the adjacent photographing directions, for example, when panoramic photographing is performed. The movement amount calculation circuit 116c also calculates the amount of shake of the imaging device. For example, when panoramic shooting is performed, the positioning circuit 116d performs image positioning as described above based on the movement amount calculated by the movement amount calculation circuit 116c. The alignment circuit 116d also performs alignment for blur correction. The geometric transformation circuit 116e performs distortion correction of the taking lens 101, affine transformation, projection transformation, cylindrical coordinate transformation, and the like. In particular, when panoramic photography is performed, the geometric conversion circuit 116e performs the above-described distortion correction and cylindrical coordinate conversion. The scaling circuit f changes the size of the image. The trimming circuit 116g cuts out a part of the image. The combining circuit 116j combines a plurality of images. In particular, when panoramic shooting is performed, the synthesizing circuit 116j synthesizes a panoramic image by overlapping the common areas and connecting the images as described above. The operation of the movement amount calculation circuit 116c, the positioning circuit 116d, the geometric conversion circuit 116e, the scaling circuit 116f, the trimming circuit 116g, and the combining circuit 116j when panoramic shooting is performed in the imaging apparatus of the present embodiment will be described later. .

  The memory controller 119 controls writing and reading of data to and from the memory 120, temporary storage of data in the buffer memory 122, and the like. The memory 120 may be a removable card type memory or the like. The memory controller 119 stores unprocessed image data that has not been processed by the image processing circuit 116 in the buffer memory 122, and stores digital image data processed by the image processing circuit 116 in the memory 120. Further, the memory controller 119 outputs the image data read from the buffer memory 122 or the memory 120 to the image processing circuit 116. Further, the memory controller 119 can output an image stored in the memory 120 to an external device such as a computer via the external interface 121.

  The display unit 118 is a display device such as a TFT or an organic EL. The display drive circuit 117 receives display data stored in the buffer memory 122 also used as a VRAM via the memory controller 119 and the image processing circuit 116, drives the display unit 118 with the display data, and displays an image on the screen. And so on.

  The operation unit 124 is connected to various buttons and switches, detects the states of the buttons and switches, and transmits the state detection signals to the microcomputer 123. The microcomputer 123 controls each unit according to a state detection signal from the operation unit 124.

  Among various switches connected to the operation unit 124, a switch 125 (hereinafter, referred to as SW1) and a switch 126 (hereinafter, referred to as SW2) are switches that are turned on / off by operating a release button. The state in which only SW1 is ON is a so-called half-pressed release button state. When the release button is half-pressed, the microcomputer 123 starts the autofocus operation of the imaging apparatus and starts the photometric operation. The state where both SW1 and SW2 are on is the so-called full-press state of the release button. When the release button is fully depressed, the microcomputer 123 causes the imaging device to capture and record an image. Further, while the release button is fully pressed, the microcomputer 123 causes the imaging device to perform a continuous shooting operation. At the time of the panoramic shooting described above, the release button is fully pressed.

  The operation unit 124 includes buttons and switches (not shown) such as an exposure compensation button, an aperture button, an ISO setting button, a menu button, a set button, a flash setting button, and a single / continuous / self-timer switching button. Is also connected. Buttons (not shown) connected to the operation unit 124 include, for example, a move + (plus) button and a move-(minus) button for moving a menu or a reproduced image, a display image enlargement button, a display image reduction button, A playback switch, an erase button, an information display button, and the like are also included. The aperture button is a button operated when the aperture 103 is narrowed down to a preset aperture value. The delete button is a button operated when deleting a captured image. The information display button is a button that is operated when displaying information regarding shooting and playback. Further, a rotary dial may be connected to the operation unit 124, for example. For example, each function of the move + (plus) button and the move-(minus) button may be provided in a rotary dial. By rotating the rotary dial, numerical values and functions can be selected more easily.

  The liquid crystal drive circuit 127 drives the external liquid crystal display unit 128 and the liquid crystal display unit 129 in the finder. The microcomputer 123 sends display contents to the liquid crystal drive circuit 127, and the liquid crystal drive circuit 127 drives the external liquid crystal display unit 128 and the liquid crystal display unit 129 in the finder, so that the operation state of the image pickup apparatus, messages, and the like are displayed in characters and the like. Display using an image. Further, a backlight such as an LED (not shown) is disposed in the liquid crystal display unit 129 in the finder, and the LED is also driven by the liquid crystal driving circuit 127.

  The microcomputer 123 checks the capacity of the memory 120 via the memory controller 119 based on the predicted value data of the image size according to the ISO sensitivity, the image size, and the image quality set before the photographing, and then sets the remaining photographable amount. Numbers can be computed. Then, the microcomputer 123 causes the display unit 118 to display the remaining photographable number. The microcomputer 123 can also display the remaining number of images that can be shot on the external liquid crystal display unit 128 and the finder liquid crystal display unit 129 as necessary.

  The nonvolatile memory 130 is an EEPROM, and can store the stored contents even when the power of the imaging device is not turned on. The operation program of the microcomputer 123 is stored in the nonvolatile memory 130. When image processing for an image acquired by, for example, panoramic photography is realized by a software configuration, the image processing program according to the present embodiment is also stored in the nonvolatile memory 130, and the microcomputer 123 executes the image processing program. .

The gyro sensor 133 is, for example, a biaxial or triaxial angular velocity sensor, detects the angular velocity of rotation in the imaging device, and outputs the detection signal (hereinafter, referred to as gyro information) to the microcomputer 123.
The thermometer 134 detects the temperature and outputs a detection signal to the microcomputer 123. In the case of the present embodiment, the thermometer 134 is preferably arranged near the gyro sensor 133, but is generally installed near the imaging sensor 112, the memory 120, and the like for protection of the imaging device, the memory 120, and the like. The existing temperature sensor may be diverted.

The external interface 121 connects the imaging device of the present embodiment to an external device such as a computer.
The power supply unit 131 includes, for example, a detachable battery, and supplies necessary power to each unit of the imaging device of the present embodiment.
The internal clock 132 outputs time information and time information necessary for the operation of the imaging device, and outputs the information to the microcomputer 123. The microcomputer 123 can add or superimpose the photographing time data based on the time information from the internal clock 132 to the image file recorded in the memory 120, for example.

  Hereinafter, a basic photographing operation in the imaging apparatus of the present embodiment will be described with reference to the flowchart of FIG. The photographing operation shown in the flowchart of FIG. 5 is performed by the microcomputer 123 controlling each unit of the imaging device. Before the photographing operation is started, it is assumed that the exposure amount is calculated in advance by the exposure amount calculation circuit 109, and the aperture amount, the accumulation time (shutter speed), the ISO sensitivity, and the like are determined. In the following description, the processing steps S501 to S507 in the flowchart of FIG. 5 are abbreviated as S501 to S507.

  When the release button is fully pressed by the user and both SW1 and SW2 are turned on, the microcomputer 123 starts controlling the photographing operation shown in the flowchart of FIG.

  When the control of the photographing operation is started, the microcomputer 123 controls each unit to perform the following series of operations in S502. First, the microcomputer 123 notifies the aperture driving circuit 104 of a predetermined aperture amount, and sets the aperture 103 to a target aperture amount. In addition, the microcomputer 123 prepares for shooting in which the image sensor 112, the AD converter 115, and the like are operable. When the preparation for photographing is completed, the microcomputer 123 controls the mirror driving circuit 107 to flip the main mirror 105 upward, and controls the shutter driving circuit 111 to open the front curtain (not shown) of the shutter 110. . As a result, the subject image via the photographing lens 101 is formed on the image sensor 112. Subsequently, the microcomputer 123 controls the shutter drive circuit 111 to close the rear curtain (not shown) of the shutter 110 after a preset accumulation time. As a result, light enters the image sensor 112 only during the accumulation time. In S502, the exposure of the image sensor 112 is performed by this series of operations.

  Subsequently, in step S <b> 503, the microcomputer 123 causes the image data output from the image sensor 112 and converted by the AD converter 115 to be stored in the buffer memory 122 via the image processing circuit 116 and further via the memory controller 119. Further, as S504, the microcomputer 123 sends the image data read from the buffer memory 122 via the memory controller 119 to the image processing circuit 116, and causes the developing circuit 116k to perform the developing process. In step S504, the microcomputer 123 may control the image processing circuit 116 to perform image processing such as white balance processing and processing for applying a gain to a dark part by the gamma correction circuit 116b.

  Next, in step S505, under the control of the microcomputer 123, the image processing circuit 116 converts the image data on which the image processing has been performed into a general-purpose data format such as JPEG by the compression / decompression circuit 116l. To record. Thereafter, if the release button is not fully pressed, the microcomputer 123 ends the control of the photographing operation in the flowchart of FIG. 5 in S507.

  It should be noted that the image data recorded in the memory 120 may be data obtained by reversibly compressing RAW data that has not been subjected to image processing and development processing in the image processing circuit 116 by the compression / decompression circuit 116l. The microcomputer 123 determines whether the image data to be recorded in the memory 120 is JPEG image data or lossless-compressed RAW data based on an instruction from the user via the operation unit 124.

Next, the operation of the imaging apparatus according to the present embodiment illustrated in FIG. 1 when panoramic imaging is performed will be described.
The imaging device of the present embodiment can operate in various shooting modes including the panorama shooting mode, and the above-described panorama shooting can be executed by the user setting the shooting mode of the imaging device to the panorama shooting mode. Becomes

  When a user inputs an instruction to switch to the panoramic imaging mode via the operation unit 124, the microcomputer 123 sets the imaging apparatus to the panoramic imaging mode and supplies power to the imaging sensor 112 and the AD converter 115. Control and make initial settings. Further, the microcomputer 123 controls the mirror driving circuit 107 to raise the main mirror 105, and controls the shutter driving circuit 111 to open the shutter 110, so that a subject image or the like by the photographing lens 101 is formed on the image sensor 112. So that Thus, the image signal read from the image sensor 112 and digitally converted by the AD converter 115 is sent to the image processing circuit 116. Further, the microcomputer 123 controls the image processing circuit 116 to perform the development processing by the development circuit 116k, the image processing by the luminance adjustment circuit 116a and the gamma correction circuit 116b, and the magnification processing by the magnification circuit 116f. The image scaled to an image size suitable for display by the scaling circuit 116f is sent to the display unit 118 and displayed. In the imaging apparatus, a process from imaging by the imaging sensor 112 to display on the display unit 118 is repeatedly performed 24 to 60 times per second, so that a so-called live view is displayed.

  Here, when performing panoramic imaging with the imaging apparatus of the present embodiment, the user moves the imaging lens 101 to a desired angle of view (desired focus) while checking the live view display on the display unit 118 as a preparation work for starting panoramic imaging. Distance). Further, the user presses SW1 (presses the release button halfway) with the photographic lens 101 facing the main subject within the photographic range of the whole view 220. When the user presses SW1, the microcomputer 123 controls each unit of the imaging apparatus in preparation for panoramic shooting, calculates an appropriate exposure amount for a subject within the angle of view, and focuses on the subject within the angle of view. And make it match. For example, when the live view display is performed, the microcomputer 123 causes the exposure amount calculation circuit (not shown) included in the image processing circuit 116 to calculate an optimal exposure amount, and acquires information on the exposure amount. For example, when the live view is not displayed, the microcomputer 123 causes the exposure amount calculation circuit 109 to calculate an optimal exposure amount according to the light reception signal of the light reflected by the sub-mirror 106, and the calculated exposure amount Get the information of. The microcomputer 123 performs drive control of the aperture 103 via the aperture drive circuit 104, sensitivity control of the image sensor 112, and control of the accumulation time based on the information on the exposure amount. Further, the microcomputer 123 controls the focusing lens to be focused on the main subject within the angle of view by driving the photographing lens 101 via the AF driving circuit 102. When the preparation for starting the panorama shooting is completed, the microcomputer 123 outputs a notification sound from a speaker or the like (not shown) or displays a notification to notify the user that the preparation for starting the panorama shooting is completed.

Next, the user turns the photographing direction (photographing lens 101) in a direction in which the user wants to start panoramic photographing in the whole view 220 of the image capturing apparatus, and then presses down SW2 (fully presses the release button). As a result, the image capturing apparatus actually starts panoramic image capturing and starts capturing an image. Then, when the panoramic shooting is started, the imaging device starts the processing of the flowchart shown in FIG.
Hereinafter, the flow of processing in the imaging apparatus at the time of panoramic shooting will be described using the flowchart of FIG. 5 and the data flow diagram shown in FIG. The operations shown in the flowchart of FIG. 5 and the data flow diagram of FIG. 6 are performed by the microcomputer 123 controlling each unit of the imaging apparatus. Note that the processing of the flowchart illustrated in FIG. 5 may be executed by a hardware configuration, or may be partially realized by a software configuration and the remaining may be realized by a hardware configuration. When the processing is executed by the software configuration, the processing is realized, for example, by the microcomputer 123 executing a program stored in the nonvolatile memory 130. The program according to the present embodiment may be prepared in the nonvolatile memory 130 in advance, may be read from a removable semiconductor memory or the like, or may be downloaded from a network such as the Internet (not shown).

  When panoramic photography is started in S601, the microcomputer 123 first acquires lens information and sensor information in S602. The lens information includes lens correction data used for correcting distortion and a decrease in peripheral light amount in the photographing lens 101, and information on a focal length and an angle of view used for cylindrical coordinate conversion. The lens information is stored in, for example, a memory (not shown) built in the photographing lens 101 or a nonvolatile memory 130, and the microcomputer 123 stores the obtained lens information in an internal memory. The sensor information includes the size of the vertical and horizontal widths of the effective imaging area of the imaging sensor 112, the number of pixels, the size per pixel, the pixel pitch, and the like. The sensor information may be stored in advance in, for example, the nonvolatile memory 130 or the like, or may be calculated by the microcomputer 123. The microcomputer 123 stores the acquired sensor information in the internal memory.

  Next, in step S603, the microcomputer 123 controls each unit of the imaging device to capture the first image. At this time, since the image sensor 112 and the AD converter 115 are set to, for example, drive for live view, the microcomputer 123 switches the drive to drive for still image shooting. Further, the microcomputer 123 adjusts the aperture 103 to the predetermined exposure amount via the aperture drive circuit 104, and further opens and closes the shutter 110 via the shutter drive circuit 111, so that a subject image is formed on the image sensor 112. To be imaged. Thus, a still image is captured by the image sensor 112, and the image signal read from the image sensor 112 is converted into a digital signal by the AD converter 115. Then, the image processing circuit 116 performs a minimum image process such as a shading correction process of the image sensor 112 on a digital signal from the AD converter 115 by a circuit (not shown). The image data subjected to the minimum processing by the image processing circuit 116 is stored in the buffer memory 122 via the memory controller 119 as data of the first RAW image.

  In step S604, the microcomputer 123 can detect how much the imaging apparatus has been swung (rotated) from the time when the first image is shot until the time when the next second image is shot. The gyro sensor 133 is initialized (reset).

  The first RAW image data stored in the buffer memory 122 in S603 is read out by the memory controller 119 and sent to the developing circuit 116k of the image processing circuit 116. The development circuit 116k performs a development process on the RAW image to convert the RAW image into a first YUV image including luminance (Y) and color difference (UV), and converts the YUV image data into a geometric conversion circuit 116e and a scaling circuit. Send to 116f. If the first RAW image captured in S603 is the N-th RAW image 705 in the data flow diagram of FIG. 7, the first YUV image is the N-th RAW image 705 of FIG. Is an N-th post-development image 706 developed by the developing circuit 116k.

  In the scaling circuit 116 f of the image processing circuit 116, a reduction process according to the number of pixels of the display unit 118 is performed on the first YUV image, and the reduced image obtained by the reduction process is stored in the VRAM area of the buffer memory 122. Is stored in Assuming that the first YUV image is the N-th developed image 706 in FIG. 7, a reduced image obtained by reducing the N-th developed image 706 by the scaling circuit 116f is stored in the VRAM area 708. Is done. Then, the first YUV image read from the VRAM area is displayed on the screen of the display unit 118 via the display drive circuit 117. By looking at this display, the user can confirm the first captured image in panoramic photography.

  Further, the geometric conversion circuit 116e of the image processing circuit 116 performs a process of correcting the distortion of the photographic lens 101 on the YUV image using the lens correction data obtained via the microcomputer 123 as the process of S605. The processing for correcting the distortion is an existing technique, and a description thereof will be omitted. Assuming that the first YUV image is the N-th developed image 706 in FIG. 7, the image after the N-th developed image 706 in FIG. This is the Nth geometrically transformed image 707. The N-th post-geometrically transformed image 707 is stored in the buffer memory 122, and the next image is photographed. From the buffer memory 122.

  Next, in step S606, the microcomputer 123 controls each unit of the imaging device to perform the second still image shooting. In the case of capturing the second image, similarly to the first image, the microcomputer 123 controls the exposure amount and the shutter drive. Then, similarly to the first image signal, the second image signal read from the image sensor 112 is also converted into a digital signal by the AD converter 115 and subjected to the minimum image processing by the image processing circuit 116. The data is stored in the buffer memory 122 as the data of the RAW image of the eye. When the second RAW image is captured, the second RAW image becomes the N-th RAW image 705 in the data flow diagram of FIG.

  In step S606, the second RAW image data stored in the buffer memory 122 is developed by the developing circuit 116 of the image processing circuit 116 and converted into the second YUV image in the same manner as the first RAW image data. Assuming that the second RAW image captured in S606 is the N-th RAW image 705 in FIG. 7, the second YUV image is developed from the N-th RAW image 705 in FIG. 7 by the developing circuit 116k. An N-th developed image 706 is obtained. Similarly to the first YUV image, the second YUV image is reduced by the scaling circuit 116f of the image processing circuit 116, stored in the VRAM area, and then displayed on the display unit 118 via the display drive circuit 117. Will be displayed on the screen. By looking at this display, the user can confirm the second captured image in the panoramic imaging.

  Further, in S607, the microcomputer 123 acquires gyro information (gyro information 724 in the data flow diagram of FIG. 7) from the gyro sensor 133 and stores the gyro information in an internal memory (not shown). Here, as the gyro information, two-axis information of the imaging apparatus in the yaw (Yaw) direction and the pitch (Pitch) direction is acquired, but the three-axis information obtained by adding the roll (Roll) direction that is the rotation around the optical axis is added. It is preferable to obtain information. Note that the output itself from the gyro sensor 133 is information on the angular velocity. However, in panoramic imaging, it is necessary to detect how much the swing has occurred between adjacent imaging directions, that is, between the previous imaging and the current imaging. For this reason, the microcomputer 123 executes a calculation for integrating the angular velocity due to the swing from the previous shooting to the current shooting at the time of shooting the second and subsequent images, respectively, and executes the rotation angle between the adjacent shooting directions. Is calculated and stored in an internal memory (not shown).

  Next, in step S608, the movement amount calculation circuit 116c acquires, from the microcomputer 123, information on the above-described focal length and angle of view of the lens information, sensor information, and information on the rotation angle between adjacent shooting directions during a swing. Then, the movement amount calculation circuit 116c converts the rotation angle of the swing from the previous shooting to the current shooting, that is, between adjacent shooting directions, into a movement amount expressed in pixel units based on the information. . In the present embodiment, the movement amount obtained in S608 is a movement amount calculated based on gyro information from the angular velocity sensor.

  Here, assuming that the focal length of the imaging lens 101 is f [mm] and the effective imaging area width of the imaging sensor 112 is w [mm], the angle of view α [°] after the distortion correction (or the lens having no distortion). The angle of view α) can be calculated by the following equation (8).

  α = 2 × arctan (w / 2 / f) Equation (8)

Further, assuming that the size per pixel of the image sensor 112 is p [μm] and the swing angle from the previous shooting to the current shooting is β [°], the moving amount d [pix] is given by Expression (9). Can be calculated.
d = tan (β / 2) × f / p × 1000 Equation (9)

Next, in step S609, the geometric conversion circuit 116e performs a process of correcting distortion on the second YUV image in the same manner as the first YUV image.
Subsequently, in S610, the microcomputer 123 calculates the radius of the virtual cylinder used when the geometric conversion circuit 116e performs the cylindrical coordinate conversion.
Here, as described above, when the photographing lens 101 is a wide-angle lens having a short focal length, when the focal length is set to the radius of the virtual cylinder and cylindrical coordinate conversion is performed, the coordinate conversion amount near the left and right ends of the image is obtained. And the number of effective pixels in the height direction decreases.

  Therefore, in the present embodiment, as described above, the case where the focal length f is set to the radius R of the virtual cylinder and the case where the radius R of the virtual cylinder is set to be larger than the focal length f are substantially the same at the upper and lower ends of the image. The cylindrical coordinate conversion is performed by increasing the radius of the virtual cylinder within a range in which there is no significant difference. The process of calculating the radius R of the virtual cylinder may be performed only once at first, and the radius of the virtual cylinder may be a fixed value thereafter, or may be performed each time an image is captured as in the present embodiment. Furthermore, it is one of the preferable embodiments that the radius of the virtual cylinder is gradually made closer to the value of the focal length f every time an image is sequentially taken from the calculation result of the radius of the virtual cylinder performed first. .

Subsequently, in step S611, cylindrical coordinate conversion is performed by the geometric conversion circuit 116e. In this embodiment, since both the cylindrical coordinate conversion and the distortion correction are performed by the geometric conversion circuit 116e, the geometric conversion circuit 116e may perform the cylindrical coordinate conversion and the distortion correction separately as described above, or at the same time. May go.
When the second YUV image is the N-th developed image 706 in FIG. 7, the N-th developed image 706 in FIG. 7 is subjected to cylindrical coordinate conversion and distortion correction by the geometric conversion circuit 116e. The image becomes an N-th post-geometric transformation image 707. The Nth geometrically transformed image 707 is stored in the buffer memory 122, and when the next image (third image) is photographed and the movement amount is calculated by the movement amount calculation circuit 116c, N-1 images are calculated. The image 703 is read from the buffer memory 122 as the image 703 after the eye geometric conversion.

  Next, in step S612, the movement amount calculation circuit 116c performs a movement amount calculation process using the N-th image after geometric conversion 707 and the (N-1) -th image after geometric conversion 703. A known method can be used for the process of calculating the movement amount from the image. In the present embodiment, the moving amount calculation circuit 116c detects edges from the image, extracts some feature points, and samples the feature points to calculate affine transformation coefficients. As described above, the movement amount calculation circuit 116c calculates the movement amount of the captured image by detecting the edge and further extracting the feature points.

  Here, for example, the coordinates of the feature point 1 are shifted from (x1, y1) to (u1, v1) and the coordinates of the feature point 2 are (x2, y2) between the image taken last time and the image taken this time. ) To (u2, v2), and the coordinates of the feature point 3 are shifted from (x3, y3) to (u3, v3). In this case, if simultaneous equations are created based on the above-mentioned equation (1), the equations are as shown in equations (10) and (11).

  By solving the equations (10) and (11), the affine coefficients a to f in the equations can be calculated. Then, when four or more feature points are extracted, the movement amount calculation circuit 116c excludes feature points having a short distance among them, and normalizes the remaining feature points by the least square method. On the other hand, when three feature points cannot be extracted, when the detected three feature points are arranged in a straight line, or when two of the three points are close to each other, the movement amount calculation circuit 116 c It is determined that the calculation of the movement amount has failed. If the movement amount (affine coefficient) calculated based on the feature points of the image is significantly different from the movement amount calculated based on the gyro information 724 (for example, if the difference between the movement amounts exceeds a threshold), It is possible that the image contains a repetitive pattern or a moving object. For this reason, when the movement amount based on the image is significantly different from the movement amount based on the gyro information, the movement amount calculation circuit 116c cancels the movement amount calculated based on the feature points of the image, and under another condition. The movement amount may be calculated again. If the movement amount based on the image is significantly different from the movement amount based on the gyro information, the microcomputer 123 controls the next photographing (S606) with the photographed image as a failed image, or controls the panoramic photographing itself. The processing in FIG. 6 may be terminated as a failure. On the other hand, if the difference between the movement amount based on the image and the movement amount based on the gyro information is smaller than the threshold value, and the two movement amounts substantially match, the process proceeds to S613.

  In step S613, the positioning circuit 116d performs positioning of the Nth image and the (N-1) th image based on the moving amount (affine coefficient) calculated based on the image by the moving amount calculation circuit 116c. Let it do. In the case of the present embodiment, the positioning circuit 116d performs positioning by affine transformation using affine coefficients. In the case of the example of FIG. 7, a registration image 711 after the registration by the positioning circuit 116d is obtained.

  Next, as S614, the synthesizing circuit 116j performs a synthesizing process using the alignment image 711 and the synthesized image that has been subjected to the synthesizing process. That is, when the N-th (N> 2) image processing is being performed in the image processing circuit 116, the combining circuit 116j performs processing on the N-th alignment image 711 and the (N-1) -th image before that. Are synthesized with the synthesized image 710 synthesized in step (1). The composite image 712 by the composite circuit 116j corresponds to a composite image in which the above-described common areas are overlapped and connected. Note that if there is a moving object over the whole area such as the water surface in the panorama shooting range, the quality of the synthesized result may be degraded. To improve the quality.

  Next, in step S615, the microcomputer 123 determines whether or not the switch SW2 is pressed. If the microcomputer 123 determines that the switch SW2 is pressed, the process returns to step S606, and the next image is captured. On the other hand, if it is determined in step S615 that the SW2 has not been pressed, the microcomputer 123 determines that the image capturing for panoramic capturing has ended, and proceeds to step S616.

In step S616, the microcomputer 123 causes the compression / expansion circuit 116l of the image processing circuit 116 to perform processing for compressing the composite image 712 (panoramic image) into a general-purpose format such as JPEG.
After that, in S617, the microcomputer 123 stores the compressed data of the composite image 712 in the memory 120 via the memory controller 119. Note that, before the compression processing, the gamma correction may be performed by the gamma correction circuit 116b in order to make the dark portion of the composite image 712 easier to see, or the color tone may be corrected to unify the color tone of the entire image. When the size of the synthesized image is large, the scaling process by the scaling circuit 116f may be performed so that the size of the synthesized image becomes the size designated by the user in advance. Further, in consideration of camera shake and the like, compression processing and storage processing may be performed after the trimming circuit 116g cuts out a maximum inscribed rectangle or a predetermined area.

  Here, as shown in FIG. 2A, an example in which the user 200 swings in the horizontal direction while holding the imaging device 201 at the normal position (horizontal position) has been described. This embodiment also includes a case where the player swings in the horizontal direction while holding the player. In addition, the present embodiment also includes a case where the user swings the imaging device in another direction such as the gravity direction (vertical direction).

  As described above, in the present embodiment, the movement amount between the adjacent photographing directions in the imaging device is calculated based on the gyro information, and the radius of the virtual cylinder is determined based on the calculated movement amount. In this embodiment, the radius of the virtual cylinder is set to be larger than the focal length of the photographing lens, so that the effective height is not reduced by the cylindrical coordinate conversion. Thus, according to the present embodiment, in a case where a plurality of images taken while sequentially changing the shooting direction are combined by cylindrical coordinate conversion and combined, a wide angle of view is maintained while using a lens with a short focal length. Images can be generated.

<Second embodiment>
Next, a second embodiment will be described. Since the configuration and the like of the imaging apparatus according to the second embodiment are the same as those described above, the illustration and description thereof are omitted.
In the first embodiment described above, as described in the flowchart of FIG. 6, the swing amount (movement amount) is calculated based on the gyro information, and the radius of the virtual cylinder is determined based on the movement amount. On the other hand, in the second embodiment, the movement amount is calculated from the captured image, and the radius of the virtual cylinder is determined based on the movement amount. FIG. 8 shows a flowchart of the second embodiment including a process of calculating a movement amount between adjacent shooting directions from an image captured by panorama shooting and determining a radius of the virtual cylinder based on the movement amount. In FIG. 8, S801 to S805 are the same as S601 to S605 described above, and S807 is substantially the same as S606, S808 is S609, and S813 to S819 are substantially the same as S612 to S619. Description is omitted. Hereinafter, processing different from the flowchart of FIG. 6 will be mainly described.

In the case of the flowchart of FIG. 8, the microcomputer 123 advances the processing to S806 after the distortion correction processing in S805.
In step S806, the microcomputer 123 determines the actual focal length of the imaging lens 101 to be the radius of the virtual cylinder, and causes the geometric conversion circuit 116e to perform cylindrical coordinate conversion on the first image. Note that the cylindrical coordinate conversion in S806 is performed in order to calculate an accurate movement amount in a movement amount calculation performed in S810 described later. After S806, the second and subsequent images are captured in S807 in the same manner as in S606, and the distortion correction process is performed in S808 in the same manner as in S609.

  In S809 after S808, the microcomputer 123 also determines the actual focal length of the photographing lens 101 to be the radius of the virtual cylinder for the second and subsequent images, and performs cylindrical coordinate conversion by the geometric conversion circuit 116e of the image processing circuit 116. Let it do. Similar to S806, the cylindrical coordinate conversion in S809 is also performed in order to calculate an accurate movement amount by a movement amount calculation performed in S810 described later.

  Next, in step S810, the movement amount calculation circuit 116c calculates the Nth (current) image and the (N-1) th image after the cylindrical coordinate conversion is performed using the virtual cylindrical radius corresponding to the actual focal length as described above. Using the (previous) image, the movement amount is calculated in the same manner as in S612.

  Next, in step S811, the microcomputer 123 calculates the radius of the virtual cylinder based on the movement amount calculated from the image in step S810, in the same manner as in the first embodiment. That is, similar to the first embodiment, the case where the focal length f is set to the radius R of the virtual cylinder and the case where the radius R of the virtual cylinder is set to be larger than the focal length f are substantially at the upper and lower ends of the image. When the difference does not occur, that is, when the difference is less than the threshold, the radius of the virtual cylinder is increased.

  Next, in step S812, the geometric conversion circuit 116e uses the virtual cylinder radius calculated in step S811 to perform cylindrical coordinate conversion in the same manner as in step S611 in FIG. That is, the geometric transformation circuit 116e performs cylindrical coordinate transformation on each of the sequentially photographed images using the virtual cylinder radius calculated in S811.

  Next, in step S813, the movement amount calculation circuit 116c uses the Nth (current) image and the (N-1) th (previous) image after the cylindrical coordinate conversion has been performed in step S812, in the same manner as in the above-described step S612. And recalculate the movement amount. The movement amount is recalculated in S813 because the movement amount calculated in S810 is obtained from an image obtained by performing cylindrical coordinate conversion with the virtual cylinder radius of the actual focal length, and the virtual cylinder radius at that time is the virtual cylinder calculated in S811. This is because, unlike the radius, the amount of movement also differs as a result. Therefore, if the virtual cylinder radius used in the cylindrical coordinate conversion in S809 and S812 is the same or a difference in an allowable range, such as when the swing speed is high, the recalculation in S813 can be omitted. In addition, the movement amount (coefficient of the affine transformation) calculated in S809 may be transformed in consideration of the difference in the cylindrical coordinate transformation.

  After that, in steps S814 to S819, in the same manner as in steps S613 to S618 described above, each process of positioning, combining, ending determination, compression, saving, and further ending panoramic shooting is performed.

  As described above, in the second embodiment, the amount of movement between adjacent photographing directions during a swing is calculated from each photographed image, and the radius of the virtual cylinder is determined based on the amount of movement. In the case of the second embodiment, as in the first embodiment, the radius of the virtual cylinder is made larger than the focal length of the photographing lens 101 to minimize the reduction of the effective height due to the cylindrical coordinate conversion. It can be suppressed.

  The preferred embodiments of the present invention have been described above, but the present invention is not limited to these embodiments, and various modifications and changes can be made within the scope of the gist. The present embodiment is not provided with an image sensor or a photographic lens, and operates an image processing apparatus that acquires an image and lens information at the time of image capturing from an external memory such as an SD card and combines them, and operates these processes on a computer. And a recording medium. Further, the imaging apparatus according to the embodiment can be applied to not only a digital camera but also various mobile terminals such as a smartphone and a tablet terminal having a camera function, a vehicle-mounted camera, an industrial camera, and the like.

  In the present embodiment, the present invention can also be realized by supplying a storage or recording medium storing program codes of software for realizing the functions of the above-described embodiments to an image processing apparatus. A computer (or CPU, MPU, or the like) of the calculation unit reads out the program code stored in the storage medium and executes the above-described function. In this case, the program code itself read from the storage medium implements the functions of the above-described embodiment, and the program and the storage medium storing the program constitute the present invention. By installing the program of the present invention in a computer of an imaging apparatus including a predetermined imaging optical system, a predetermined imaging unit, and a computer, the imaging apparatus can detect a distance with high accuracy. The computer of the present invention can be distributed via the Internet in addition to the storage medium.

  Each of the above-described embodiments is merely an example of a specific embodiment for carrying out the present invention, and the technical scope of the present invention should not be interpreted in a limited manner. That is, the present invention can be implemented in various forms without departing from the technical idea or the main features.

  101: photographing lens, 112: image sensor, 116: image processing circuit, 116c: movement amount calculation circuit, 116d: position adjustment circuit, 116e: geometric conversion circuit, 116f: variable magnification circuit, 116j: synthesis circuit, 116k: development circuit , 116l: compression / expansion circuit, 123: microcomputer, 133: gyro sensor

Claims (15)

Acquisition means for acquiring a plurality of images captured so as to include a common area in adjacent imaging directions while sequentially changing the imaging direction of the imaging device,
Determining means for determining the radius of the virtual cylinder;
A conversion unit that converts the plurality of images into cylindrical coordinates into an image projected on the virtual cylinder,
Detecting means for detecting an amount of movement between adjacent photographing directions;
Based on the detected amount of movement, a positioning unit that performs positioning of each image subjected to the cylindrical coordinate conversion,
Synthesizing means for superimposing the common area of each image subjected to the alignment and synthesizing the images so as to sequentially connect the images,
The image processing apparatus according to claim 1, wherein the determining unit determines a radius of the virtual cylinder to be larger than a focal length of a photographing lens included in the imaging device.   The image processing apparatus according to claim 1, wherein the determination unit calculates a radius of the virtual cylinder based on at least the movement amount and a focal length of the photographing lens.   The image processing apparatus according to claim 2, wherein the determining unit determines the radius of the virtual cylinder to be larger than the focal length when the moving amount is smaller than a first threshold.   The determining means performs the cylindrical coordinate conversion by setting the focal length of the imaging lens to the radius of the virtual cylinder, and performs the cylindrical coordinate conversion by setting the radius of the virtual cylinder to a radius larger than the focal length of the imaging lens. The method according to claim 1, wherein when the position shift at the boundary of the image in the synthesis is smaller than a second threshold, the radius of the virtual cylinder is determined to be larger than the focal length. 3. The image processing device according to 2. The determining means comprises:
The height of the image is h [pix], the pixel pitch is p [μm], the radius of the virtual cylinder is R [mm], the focal length is f [mm], and the X component of the vector representing the moving amount is x [ pix],
(h / 2) · x · cos {arctan (x · p / R / 1000) / 2},
When the difference from the expression of (h / 2) · x · cos {arctan (x · p / f / 1000) / 2} is smaller than the second threshold, the radius of the virtual cylinder is set to be smaller than the focal length. The image processing apparatus according to claim 4, wherein the radius is determined to be larger.   2. The method according to claim 1, wherein the determining unit determines a radius of the virtual cylinder to be larger than the focal length when a focal length of the photographing lens is smaller than a third threshold value. 6. The image processing apparatus according to any one of claims 1 to 5,   The image processing apparatus according to claim 1, wherein the determining unit changes a radius of the virtual cylinder for each of the images.   The deciding means determines the virtual cylinder by using at least one of an image taken first and an image taken last, and each image taken between the first and last images. The image processing apparatus according to claim 7, wherein the radius of the image processing device is changed.   The apparatus according to claim 1, wherein the detection unit acquires information from an angular velocity sensor included in the imaging device, and calculates the movement amount based on information from the angular velocity sensor. The image processing apparatus according to any one of the preceding claims. The detection means further calculates a movement amount based on feature points of images taken in adjacent shooting directions,
In the positioning unit, when a difference between the movement amount calculated based on the information of the angular velocity sensor and the movement amount calculated based on the feature point is less than a fourth threshold value, The image processing apparatus according to claim 9, wherein the positioning is performed using a movement amount calculated based on a point.   The detection unit is configured to determine the amount of movement based on feature points of images photographed in adjacent different photographing directions, a focal length of the photographing lens, and information on the size and the number of pixels of an image sensor included in the image capturing apparatus. The image processing apparatus according to claim 1, wherein is calculated. The detection means calculates a movement amount based on the feature points of the image after the cylindrical coordinate conversion by the virtual cylinder having the focal length as the radius,
The determining unit determines the radius of the virtual cylinder larger than the focal length based on the calculated movement amount, the focal length, and information on the size and the number of pixels of the image sensor.
12. The apparatus according to claim 11, wherein the detection unit recalculates the movement amount based on a feature point of the image after the cylindrical coordinate conversion by the virtual cylinder having the determined radius is performed. Image processing device. An image sensor that captures an optical image by a taking lens;
Control means for performing control to continuously capture images at predetermined time intervals,
An image processing apparatus according to any one of claims 1 to 12,
An imaging device comprising: An acquisition step of acquiring a plurality of images photographed so as to include a common area in adjacent photographing directions while sequentially changing the photographing direction of the imaging device,
A determining step of determining the radius of the virtual cylinder;
A conversion step of converting the plurality of images into cylindrical coordinates into an image projected on the virtual cylinder,
A detecting step of detecting a moving amount between adjacent photographing directions;
Based on the detected movement amount, a positioning step of performing positioning of each image subjected to the cylindrical coordinate conversion,
A synthesizing step of superimposing the common area of each of the images subjected to the alignment and synthesizing the images so as to sequentially connect the images,
In the determining step, a radius of the virtual cylinder is determined to be larger than a focal length of a photographing lens included in the imaging device.   A program for causing a computer to function as each unit of the image processing apparatus according to any one of claims 1 to 13.

Images