JP2013046270A - Image connecting device, photographing device, image connecting method, and image processing program - Google Patents

Abstract

PROBLEM TO BE SOLVED: To generate a panoramic composite image having joint portions where mismatch of connection of images is less conspicuous.SOLUTION: A plurality of developed image data 152, obtained by continuously photographing with a photographing device moved along a horizontal plane so as to draw an arc, are input in an image conversion processing part 122 together with camera posture data 154. A plurality of spherical-surface projected image data 158 are generated at the image conversion processing part 122 by projecting each of two-dimensional images based on the plurality of developed image data 152 on a virtual spherical surface having a pole axis as a reference axis by using a processing parameter determined on the basis of the camera posture data 154. A motion vector 160 between images based on the plurality of spherical-surface projected image data 158 is derived by a motion vector computing part 124. Motion compensation between the images based on the plurality of spherical-surface projected image data 158 is performed in accordance with the motion vector 160, and necessary parts are extracted from the plurality of spherical-surface projected image data 158 and connected so as to generate a panoramic composite image 162.

Description

  The present invention relates to a technique for combining a plurality of images and a technique for generating a panoramic image having a wide angle of view.

  There is a panorama synthesis technique for generating a wide-field image by combining a plurality of images obtained by shooting while changing the direction of the camera. At this time, it is known that an image obtained by projection conversion to a virtual curved surface, such as cylindrical conversion or spherical projection, is generated and bonded, so that the image can be smoothly joined with reduced image shift at the joined portion. As a panorama composition process using cylindrical transformation, there is one disclosed in Patent Document 1. As a panorama synthesis process using spherical projection, there is one disclosed in Patent Document 2. Also, in recent years, several tens to hundreds of images are taken at high speed while moving (swinging) the camera around the photographer so as to draw an arc, and the resulting images are cut into strips and pasted together. Thus, a technique for generating a panoramic image is also known (FIG. 8). As this prior art document, there is one disclosed in Patent Document 3.

The processing contents of the cylinder conversion are shown in FIG. The cylindrical conversion process is a process for obtaining an image as if a two-dimensional image based on image data obtained by photographing was projected onto a cylindrical virtual curved surface. Below, it demonstrates as a process which projects the image formed on an imaging surface to the cylindrical-surface-like virtual curved surface which touches this imaging surface. In this case, the radius r of the virtual curved surface is the actual focal length of the photographic lens. When the point on the imaging surface is (s, t), the s coordinate uses the actual focal length r and the incident angle θ,

s = r · tan θ (1)

It can be expressed as.

The light travels from each point on the imaging surface so as to concentrate on one point on the central axis of the cylinder, and reaches a virtual curved surface, and is projected onto the virtual curved surface and the formed image is developed on a plane. Then, the coordinate position (s ′, t ′) on the plane where the image on the virtual cylindrical surface corresponding to the point (s, t) on the imaging surface is developed is

s ′ = r · θ (2)
t ′ = t · cos θ (3)

It can be expressed as.

  Projecting an image formed on the imaging surface onto a virtual virtual curved surface by converting each coordinate value of the pixels constituting the image using the above-described equations (1), (2), and (3) And an image as if it were developed can be obtained.

Next, spherical projection processing will be described with reference to FIG. The spherical projection process is a process for obtaining an image as if a two-dimensional image based on image data obtained by photographing was projected onto a spherical virtual curved surface. Hereinafter, description will be given as a process of projecting an image formed on the imaging surface onto a spherical virtual curved surface in contact with the imaging surface. In this case, the radius r of the virtual curved surface is the actual focal length of the photographic lens. First, when the point on the imaging surface is (s, t), the actual focal length is r, the rotation angle about the x axis is θV, and the rotation angle about the y axis is θH.

s = r · tan θH (4)
t = r · tan θV / cos θH (5)

It is represented by

It is assumed that light travels from each point on the imaging surface so as to concentrate on the center of the spherical surface and reaches a virtual curved surface. Then, a point corresponding to the point (s, t) on the imaging surface is projected onto a spherical virtual curved surface, and this is projected by an equirectangular projection (development display projection in which latitude and longitude are orthogonal to each other at equal intervals). The plane coordinates (s ′, t ′) after expansion are

s ′ = r · θH (6)
t ′ = r · θV (7)

It is represented by

  By converting the coordinate values of the pixels constituting the image using the above-described equations (4), (5), (6), and (7), the image formed on the imaging surface is converted into a spherical virtual image. It is possible to obtain an image as if it was projected onto a curved surface and developed.

  FIG. 11A and FIG. 11B conceptually show the effects of cylindrical transformation and spherical projection. Consider a case where a plurality of images are photographed while rotating the camera around the photographer. The sizes W1 and W2 of the image formed on the imaging surface are compared between the case where the subject image is located at the center of the screen and the case where the subject image is located near the edge of the screen. When the subject image is located near the edge of the screen, the size W2 of the subject image formed on the imaging surface is larger than the size W1 when it is formed at the center of the imaging surface. As described above, the size of the subject image changes depending on the position on the imaging surface even when the subject is the same and the photographing distance is the same.

  On the other hand, the size of the subject image varies depending on the position on the imaging surface by performing processing to convert the image on the imaging surface as if it was projected onto a curved surface, such as cylindrical transformation or spherical projection. Can be suppressed. This is shown as dimensions W1 'and W2' in FIGS. 11 (a) and 11 (b). It can be seen that fluctuations in the dimensions W1 'and W2' are suppressed compared to fluctuations in the dimensions W1 and W2.

  12 (a) to 12 (d) correspond to the examples shown in FIGS. 11 (a) and 11 (b), and images obtained by actually photographing and processing these images are processed. An example of an image obtained in this way is shown. FIGS. 12A and 12B show an image obtained by actually capturing an image and an enlarged main subject in the image. In FIG. 12A, the chair image as the main subject is located closer to the center of the screen. In FIG.12 (b), the image of a chair is located near the screen edge part. In the enlarged images of the chairs in the respective drawings, when the width dimensions of the backrest portion are W1 and W2, the dimension W2 is larger than the dimension W1.

  Images obtained as a result of cylindrical transformation of the images shown in FIGS. 12 (a) and 12 (b) are shown in FIGS. 12 (c) and 12 (d). In these magnified images of the chair in FIGS. 12C and 12D, the width dimensions W1 'and W2' of the backrest are substantially equal. That is, fluctuations in the dimensions W1 ′ and W2 ′ shown in FIGS. 12C and 12D are suppressed as compared with fluctuations in the dimensions W1 and W2 of the images shown in FIGS. 12A and 12B. You can see that.

  From the above, when combining multiple images obtained by shooting while moving the camera along the arc direction around the photographer, by performing curved surface projection processing such as cylindrical transformation and spherical projection, Regardless of the position, the size of the subject can be made substantially constant. Therefore, it is possible to smoothly join the images only by performing a relatively simple alignment process for the entire image in the vertical and horizontal biaxial directions.

Japanese Patent No. 3205571 Japanese Patent No. 3817302 JP 2007-174301 A

  The stitching of images described above may not function well depending on the posture of the camera at the time of shooting. Here, an example that does not function well will be described. When the camera is shaken at the time of shooting, if the axis along the vertical direction of the camera is not parallel to the axis when the camera is shaken, the above-described image stitching may not function well. For example, this is the case when continuous shooting is performed while swinging the camera along the horizontal direction with the camera looking down from the top of the mountain. Alternatively, the same applies to the case where the vertical direction (tilt angle) of the camera is determined so as to look up from a low place, and continuous shooting is performed while swinging the camera along the horizontal direction in that state. In this specification, a surface including the movement trajectory of the camera when the camera is shaken while performing continuous shooting is referred to as a swing surface, and that the optical axis of the camera photographing lens is not parallel to the swing surface is referred to as camera tilt. Called.

  FIGS. 13A and 13B show examples of two images obtained by shaking the camera in the absence of camera tilt. FIGS. 13C and 13D show examples of two images obtained by shaking the camera in a state where a downward camera tilt exists. FIGS. 13 (a), (b), (c), and (d) all show spherical projections.

  In the images shown in FIGS. 13A and 13B taken without camera tilt, the linear subject image extending in the horizontal direction and present in the vicinity of the middle in the vertical direction in the screen is distorted. No. On the other hand, in the images shown in FIGS. 13 (c) and 13 (d) taken with the camera tilt downward, the portion corresponding to the subject image described above exists in the vicinity of the middle in the vertical direction in the screen. , A straight subject image extending in the horizontal direction) is distorted into a medium-high curve. As a result, even if the same subject is shot at the same shooting distance, the tilt, size, etc. of the subject image change according to the position in the screen.

  When partial images are cut out from the images shown in FIGS. 13 (c) and 13 (d) and connected by simple alignment in the XY two-axis directions, the distortion of the subject image according to the position in the screen described above can be reduced. Changes may cause discontinuities. This is shown in FIG. FIG. 14 is an enlarged view of a part of an image obtained by the stitching process. In FIG. 14, the discontinuous part G1 is seen in the side edge part of the seat surface of the chair. In the drawing, a discontinuous portion G2 is seen in the vicinity of the ground contact portion of the leg portion located on the right side.

  With reference to FIGS. 15A and 15B, a mechanism for causing the above-described distortion when an image captured by swinging the camera along a horizontal plane in the presence of camera tilt is spherically described. . Here, let us consider a case where the photographer holding the camera is positioned at the center of rotation and the camera is photographed at different positions (orientations) while swinging so as to draw an arc on a horizontal plane. Further, it is assumed that the subject is a single point existing in the object scene, and when the camera shutter is opened and the camera is shaken as described above, an image of a point formed on the imaging surface (hereinafter, referred to as “image”). Consider the trajectory of a “point image”. When the camera is not tilted, only the longitude of the position of the point image changes. Therefore, when the coordinate transformation of an image is performed as if a spherical virtual curved surface is developed by an equirectangular projection (development projection in which latitude and longitude are orthogonal to each other at equal intervals), the horizontal axis direction of the obtained image (see FIG. The positional deviation can be corrected only by positioning in the 10 S-axis direction). On the other hand, when shooting with a camera shake in the presence of camera tilt, both latitude and longitude fluctuate. In the example shown in FIG. 15B, the image is distorted so that the latitude increases in the vicinity of the center in the left-right direction of the image. For example, if you are shooting a scene that includes a horizontal line, project that image onto a spherical virtual curved surface, and then expand it with equirectangular projection, the image of the horizontal line above the center in the vertical direction in the image When is located, the image of the horizon produces a medium-high distortion (as if it swelled up near the center in the left-right direction).

  As described above, in the conventional technique using curved surface projection such as cylindrical transformation or spherical projection described above, if there is no tilt in the tilt direction of the camera with respect to the moving plane (swing plane) of the camera at the time of shooting, simple cylindrical transformation or spherical By projecting, images can be smoothly joined, but if there is a tilt in the tilt direction of the camera with respect to the swing surface during shooting, the image will be distorted by simple cylindrical transformation or spherical projection. It becomes difficult to join smoothly. In the prior art, sufficient consideration has not always been given to removing such image distortion caused by tilting in the tilt direction of the camera.

  The present invention has been made in view of the above-described problems. Even when there is a tilt in the tilt direction of the camera with respect to the moving plane (swing plane) of the camera at the time of shooting, the registration residual in image pasting It is an object to suppress smoothness and obtain a smooth panoramic image.

The image pasting apparatus according to the first aspect of the present invention includes a plurality of pieces of image data obtained by continuously performing a plurality of photographing operations while moving the photographing device so as to draw an arc along a horizontal plane. The partial image data is cut out and pasted together to generate panorama composite image data. This image pasting device
An image acquisition unit for acquiring the plurality of image data;
Camera posture data for acquiring camera posture data including information relating to a tilt angle that is a tilt angle in the tilt direction of the photographing device detected before, after, or during the time when the plurality of photographing operations are performed by the photographing device. An acquisition unit;
A process of projecting a two-dimensional image based on the plurality of image data onto a virtual sphere having a reference axis as a polar axis, using processing parameters determined based on information about the tilt angle included in the camera posture data. An image conversion processing unit for generating a plurality of spherical projection image data,
A motion vector arithmetic processing unit for deriving a motion vector between the images based on the plurality of spherical projection image data;
A process of performing motion compensation between images based on the plurality of spherical projection image data based on a motion vector derived by the motion vector calculation processing unit, and cutting out and pasting a necessary portion from the plurality of spherical projection image data And a synthesis processing unit that generates panorama synthesized image data.

The imaging device according to the second aspect of the present invention can generate a panorama composite image data by combining a plurality of image data,
An image sensor that continuously performs a plurality of image capturing operations when the image capturing apparatus is shaken so as to draw an arc along a horizontal plane, and generates a plurality of image data;
Before, after or during the plurality of shooting operations performed by the image sensor, a tilt angle that is a tilt angle in the tilt direction of the imaging device is measured, and camera posture data including information about the tilt angle is obtained. A posture detection unit to generate;
A process of projecting a two-dimensional image based on the plurality of image data onto a virtual sphere having a reference axis as a polar axis, using processing parameters determined based on information about the tilt angle included in the camera posture data. An image conversion processing unit for generating a plurality of spherical projection image data,
A motion vector arithmetic processing unit for deriving a motion vector between the images based on the plurality of spherical projection image data;
A process of performing motion compensation between images based on the plurality of spherical projection image data based on a motion vector derived by the motion vector calculation processing unit, and cutting out and pasting a necessary portion from the plurality of spherical projection image data And a synthesis processing unit that generates panorama synthesized image data.

In the image pasting method according to the third aspect of the present invention, each of a plurality of image data obtained by continuously performing a plurality of photographing operations while moving the photographing device so as to draw an arc along a horizontal plane. A partial image data cut out and pasted together to create panorama composite image data,
An image data acquisition step of acquiring the plurality of image data;
Camera posture data for acquiring camera posture data including information relating to a tilt angle that is a tilt angle in the tilt direction of the photographing device detected before, after, or during the time when the plurality of photographing operations are performed by the photographing device. An acquisition step;
A process of projecting a two-dimensional image based on the plurality of image data onto a virtual sphere having a reference axis as a polar axis, using processing parameters determined based on information about the tilt angle included in the camera posture data. An image conversion processing step for generating a plurality of spherical projection image data,
A motion vector deriving step for deriving a motion vector between the images based on the plurality of spherical projection image data;
A process of performing motion compensation between images based on the plurality of spherical projection image data based on the motion vector derived in the motion vector deriving step, and cutting out and pasting necessary portions from the plurality of spherical projection image data Panorama synthesis step for generating panorama synthesized image data.

The image processing program according to the fourth aspect of the present invention is based on each of a plurality of image data obtained by continuously performing a plurality of photographing operations while moving the photographing device so as to draw an arc along a horizontal plane. An image processing program for performing a process of generating partial panoramic composite image data by cutting out and pasting partial image data on a computer,
An image data acquisition procedure for acquiring the plurality of image data;
Camera posture data for acquiring camera posture data including information relating to a tilt angle that is a tilt angle in the tilt direction of the photographing device detected before, after, or during the time when the plurality of photographing operations are performed by the photographing device. Acquisition procedure;
A process of projecting a two-dimensional image based on the plurality of image data onto a virtual sphere having a reference axis as a polar axis, using processing parameters determined based on information about the tilt angle included in the camera posture data. Image conversion processing procedure for generating a plurality of spherical projection image data,
A motion vector derivation procedure for deriving a motion vector between the images based on the plurality of spherical projection image data;
A process of performing motion compensation between images based on the plurality of spherical projection image data based on the motion vector derived in the motion vector derivation procedure, and cutting out and pasting a necessary portion from the plurality of spherical projection image data Then, a panorama synthesis procedure for generating panorama composite image data is executed.

  According to the present invention, when a plurality of still images are obtained by continuously shooting while changing the direction of the camera so as to draw an arc along the swing surface, a camera tilt exists with respect to the camera swing surface. However, by performing an image conversion process in consideration of the camera tilt, it is possible to suppress the occurrence of inconsistent portions when a plurality of images are combined, and to obtain a smooth panoramic image.

It is a block diagram which shows the structural example at the time of incorporating an image bonding apparatus in an electronic camera. It is a flowchart which illustrates roughly the process sequence of the continuous imaging | photography and image synthesis process performed with the electronic camera of 1st Embodiment. It is a figure which illustrates the process of spherical projection conversion notionally, (a) is a figure which shows the process when a camera tilt angle is not considered, (b) is a figure which respectively shows the process when a camera tilt angle is considered. . It is a figure explaining the definition of the variable in the process of spherical projection conversion. It is a figure explaining the effect at the time of processing which considered the camera tilt angle at the time of processing of spherical projection conversion, (a) is an example when a camera tilt angle is not considered, (b) is a camera tilt angle considered It is a figure which shows the example of a case, respectively. It is a figure which expands and shows a part of image produced | generated by bonding the image by which spherical projection conversion processing was performed in consideration of the camera tilt angle. It is a flowchart which illustrates roughly the process sequence of the continuous imaging | photography and image synthesis process performed with the electronic camera of 2nd Embodiment. It is a figure which illustrates notionally that a continuous shooting is performed while shaking a camera along a horizontal swing surface, and the obtained image is cut out in a strip shape and pasted to generate a panorama composite image. It is a figure explaining the principle of cylinder conversion. It is a figure explaining the principle of spherical projection conversion, (a) is a figure which shows the process of spherical projection conversion notionally, (b) is the definition of a variable in the process of spherical projection conversion, and coordinate conversion formula derivation | leading-out. It is a figure explaining a principle. It is a figure explaining the relationship between the size of the subject image on the imaging surface and the size of the subject image on the projection curved surface in the cylindrical transformation and spherical projection according to the prior art, (a) when the subject is in the center of the screen For example, (b) is a diagram showing an example where the subject is at the edge of the screen. It is a figure explaining the effect by the processing of cylindrical conversion and spherical projection according to the prior art, (a), (b) is the size of the image changes depending on the position existing in the screen of the subject in the image before processing (C), (d) is a figure explaining a mode that the size of an image becomes substantially the same irrespective of the position which exists in the screen of a to-be-photographed object in the image after a process. It is a figure explaining a mode that it receives to the influence of a camera tilt in the process of cylindrical transformation and spherical projection which concern on the prior art, (a), (b) is the image obtained by image | photographing in the state without a camera tilt. (C), (d) is a figure which shows the example at the time of processing, and the example at the time of processing the image acquired by image | photographing in a state with a camera tilt, respectively. It is a figure which expands and shows a part of image generated by pasting together the image after processing of cylindrical conversion and spherical projection according to the prior art. It is a figure which shows the example which has arisen. It is a figure explaining the mechanism which produces the distortion of an image under the influence of a camera tilt in the process of the spherical projection conversion based on a prior art, (a) is a mode when there is no camera tilt, (b) is a camera tilt. It is a figure explaining a mode in case there is.

-First embodiment-
(Internal structure)
FIG. 1 is a block diagram illustrating a configuration example when the image bonding apparatus 120 according to the first embodiment is incorporated into the electronic camera 100. The electronic camera 100 may be a camera in which a photographing lens and a camera body are integrated, or a photographing lens exchangeable camera. The electronic camera 100 may be a still camera or a movie camera capable of taking a still image.

  The electronic camera 100 includes a photographing optical system 102, an image sensor 104, a memory 106, an image processing unit 110, an image pasting device 120, a control unit 140, and an image memory 142. The image pasting device 120 includes an angular velocity sensor 128, an image conversion processing unit 122, a motion vector calculation processing unit 124, and a synthesis processing unit 126. In the figure, solid arrows indicate data flow, and broken lines indicate control signal transmission paths.

  The control unit 140 includes a CPU and the like, and comprehensively controls the operation of the electronic camera 100. That is, the operation mode of the electronic camera 100 is switched to the image recording mode, the image reproduction mode, and the like, and the operation mode and the like are set by the user at that time. Operations performed in each of the processing unit 110 and the image memory 142 are comprehensively controlled. Hereinafter, a case where the electronic camera 100 is set to the panoramic image generation mode and a release operation is performed by the user will be described as an example. That is, the photographer switches the operation mode of the electronic camera 100 to the panoramic image generation mode, and then shakes (swings) the electronic camera 100 so as to draw an arc around the photographer, and continuously photographs while shifting the photographing direction. I do. As a result, a series of still image data of a plurality of frames is obtained, and each component in the electronic camera 100 performs processing described below.

  The image memory 142 is configured by a flash memory, a hard disk drive, or the like. The image memory 142 may be built in the electronic camera 100, or may be configured to be detachable from the electronic camera 100.

  The image sensor 104 is composed of a CCD or CMOS image sensor, for example, and photoelectrically converts a subject image formed on the image area of the image sensor 104 by a photographing optical system 102 having a photographing lens, a diaphragm, etc. Generate and output. In this specification, the image sensor 104 is a CMOS image sensor that includes an amplifier, an A / D converter, and the like, and is configured to be able to output a digital image signal. The image sensor 104 is also assumed to be a single-plate image sensor in which a Bayer array on-chip color filter is arranged on pixels arranged in the image area. A digital image signal output from the image sensor 104 is temporarily stored in the memory 106 as pre-development image data 150.

  Although not shown in FIG. 1, the electronic camera 100 includes a shutter that physically opens and closes such as a lens shutter, a focal plane shutter, and a liquid crystal shutter, and the shutter is continuously opened and closed during continuous shooting operation. You may make it operate | move. Alternatively, the image sensor 104 itself may have a sensor shutter function, and the above-described shutter that physically opens and closes may be configured to be capable of continuous shooting while maintaining the open state. At that time, it is desirable that the image sensor 104 is configured to be able to perform an exposure operation by a global shutter system (a system in which the exposure operation is started and stopped almost simultaneously in all pixels on the light reception of the image sensor 104). The reason is that the global shutter system can suppress image distortion that occurs when a moving object is imaged by the rolling shutter system.

  The pre-development image data 150 includes R (Red) signal, G (Green) signal, and B (Blue) for each pixel based on the array (Bayer array) of on-chip color filters formed on the light receiving surface of the image sensor 104. This is image data in the so-called Raw format having information of any one of the signals. This pre-development image data can be recorded in the image memory 142 as a raw image data file as necessary.

  The image processing unit 110 performs processing such as demosaicing, noise reduction, white balance, color correction, and gradation conversion on the pre-development image data 150 stored in the memory 106 to generate post-development image data 152. The developed image data 152 can be compressed and recorded as an image file in the image memory 142. However, in this embodiment, the electronic camera 100 performs a series of shooting operations in the panorama image generation mode, and the obtained post-development image data 152 of a plurality of frames is processed to generate one panorama composite image. To do. In this case, the developed image data 152 generated by the image processing unit 110 is temporarily recorded in the memory 106.

  The image pasting device 120 processes the post-development image data 152 of a plurality of frames generated by the image processing unit 110 as described below to generate one panorama composite image data 162.

  The angular velocity sensor 128 is a sensor configured to be able to detect the tilt angle of the electronic camera 100. The tilt angle means the tilt angle of the optical axis of the photographing optical system 102 with respect to the horizontal plane. In the following description of the tilt direction, the tilt angle takes a positive value when the optical axis of the imaging optical system 102 is upward from the horizontal plane, and takes a negative value when the optical axis is downward from the horizontal plane. A measurement operation is performed by the angular velocity sensor 128 when a series of frames are captured in the panoramic image generation mode, and camera attitude data 154 representing the tilt angle of the electronic camera 100 is output from the angular velocity sensor 128.

  The image conversion processing unit 122 performs projection processing on the virtual spherical surface and planar development processing on the developed image data 152 temporarily recorded in the memory 106 to generate spherical projection image data 158 to generate the memory. 106 is temporarily recorded. When generating the spherical projection image data 158 described above, the image conversion processing unit 122 uses shooting parameters 156 such as a focal length set at the time of shooting and camera posture data 154 obtained by measuring a tilt angle at the time of shooting. refer. As described above, since the post-development image data 152 is temporarily recorded in the memory 106 for a plurality of frames, the image conversion processing unit 122 performs image conversion on the post-development image data 152 of each frame. Perform the process. As a result, spherical projection image data 158 for a plurality of frames is temporarily recorded in the memory 106.

  The motion vector calculation processing unit 124 obtains a motion vector 160 representing the amount of positional deviation between the images for the plurality of spherical projection image data 158. That is, the motion vector calculation processing unit 124 extracts a target image block from each of a plurality of images based on a plurality of spherical projection image data 158, and the amount of movement of the target image block between a plurality of temporally adjacent images. Then, the motion vector 160 is obtained by obtaining the moving direction. The motion vector 160 between the adjacent images obtained by the motion vector calculation processing unit 124 is temporarily recorded in the memory 106. As a process for obtaining the motion vector 160, for example, there is a method in which a plurality of motion vector measurement regions are set in a reference image, and template matching processing with a reference image is performed using the image of the motion vector measurement region as a template. . Then, a motion vector 160 corresponding to each motion vector measurement region is derived.

  The template matching process is a process of calculating a matching index while scanning a template in a search area in a reference image and detecting a position where the matching index is the highest. The amount of misalignment between the position where the matching index is the highest in the reference image and the corresponding position in the standard image (the position where the template exists) is calculated and output as a motion vector that defines the direction and amount of misalignment Is done. The coincidence index uses known techniques such as SAD (Sum of Absolute Intensity Difference), SSD (Sum of Squared Intensity Difference), NCC (Normalized Cross-Correlation), etc. And can be derived.

  The synthesizing unit 126 corrects misalignment between the plurality of images by referring to the motion vector 160 obtained by the motion vector arithmetic processing unit 124 for the spherical projection image data 158 of the plurality of frames recorded in the memory 106. The panorama composite image data 162 is generated by performing the pasting process while being stored in the memory 106. At this time, if necessary, the peripheral portion of the image is cut out and stored in the memory in order to make the contour shape rectangular. The image that has been subjected to various types of image processing including development processing as described above is compressed and recorded in the image memory 142 as an image file. Further, it is displayed on a display device (not shown) or the like as necessary.

(Processing sequence)
FIG. 2 is a flowchart illustrating the contents of the image combining process performed by the electronic camera 100 including the image combining apparatus 120 according to the first embodiment. For example, when the electronic camera 100 is set to a panoramic image generation mode for generating a panoramic image, and the user performs a pressing operation on the release button, the processing shown in FIG. 2 is started.

  In step S <b> 200, the tilt in the tilt direction of the electronic camera 100 when the user presses the release button is measured and stored in the memory 106 as camera posture data 154.

  In step S202, continuous shooting is performed with exposure conditions such as focal length, shutter speed, ISO sensitivity, aperture value, etc. fixed. At this time, an arrow or the like indicating the direction in which the electronic camera 100 is shaken is displayed on the electronic camera 100, and the photographer performs shooting while swinging the electronic camera 100 so as to draw an arc according to the arrow. That is, photographing is performed while swinging the electronic camera 100 along the horizontal plane.

  The order of the processes in S200 and S202 described above may be reversed, and the camera tilt angle may be measured after continuous shooting. Alternatively, the camera tilt angle may be measured at a predetermined timing after the start of the continuous shooting operation.

  In step S204, processing such as demosaicing, noise reduction, white balance, gradation conversion, and color correction is performed on each of the plurality of pre-development image data 150 obtained by continuous shooting in step S202, and after development. Processing for generating image data 152 is performed. This process corresponds to the process performed by the image processing unit 110 described with reference to FIG.

  In step S206, for each of the plurality of post-development image data 152 obtained by the development processing in step S204, the shooting parameters 156 such as the focal length and the camera tilt angle (camera posture data 154) measured in S200. The spherical projection conversion and planar development are performed with reference to FIG. 5 to generate a plurality of frames of spherical projection image data 158. This process corresponds to the process performed by the image conversion processing unit 122 described with reference to FIG.

  The processing contents of the spherical projection processing will be described with reference to FIGS. 3 (a) and 3 (b). Here, a case where the electronic camera 100 is swung in a tilted state during shooting will be described as an example. When spherical projection is performed without taking this camera tilt into consideration, the trajectory of the point image corresponding to one point in the object scene changes not only in longitude but also in latitude as shown in FIG. This is the same as described with reference to FIG.

FIG. 3B is a diagram conceptually showing the processing content of the spherical projection processing in the embodiment of the present invention. In the process shown in FIG. 3B, the spherical axis (polar axis) on which the image is projected is tilted by the tilt angle measured by the angular velocity sensor 128. The radius of the spherical surface is assumed to be the actual focal length. The x, y, z3 axis orthogonal coordinates are set for a spherical surface whose polar axis is tilted by the tilt angle of the electronic camera 100, the rotation angle about the x axis is θx, and the rotation angle about the y axis is θy (see FIG. 4) The relationship shown below between the point (x, y, z) on the spherical surface with the radius r and the rotation angles θx, θy around the x axis and around the y axis,

(X, y, z) = (r · cosθx · sinθy, r · sinθx, r · cosθx · cosθy)
... (8)

Holds.

Since the normal coordinate system (not considering the tilt angle) (x ′, y ′, z ′) and the coordinate system (x, y, z) considering the tilt angle are shifted by the tilt angle φ, The following relationship,

Holds.

When the rotation angles around the x ′ axis and the y ′ axis in the camera-based coordinate system (not considering the tilt angle) are θ′x and θ′y,

θ′x = arcsin (y ′) (10)
θ′y = arctan (x ′ / z ′) (11)

Holds.

When it is assumed that light is emitted radially from one point and incident on the imaging surface, the corresponding position (s, t) on the imaging surface is obtained using the camera-based spherical seat system (θ′x, θ′y). ,

s = r · tan (θ′y) (12)
t = r · tan (θ′x) / tan (θ′y) (13)

It is represented by

From the equations (8) to (13) described above, the correspondence between the position (s, t) on the imaging surface and the spherical coordinates (θx, θy) considering the camera tilt angle can be obtained. it can. The spherical coordinates (θx, θy) in consideration of the camera tilt angle are expressed by the following equations (14) and (15):

s ′ = r · θy (14)
t ′ = r · θx (15)

Can be developed into plane coordinates (s ′, t ′) indicated by equirectangular projection.

  As described above, an image (a two-dimensional image based on image data) formed on the imaging surface can be projected onto a virtual curved surface of a spherical coordinate system in consideration of the camera tilt angle.

  Returning to the description of FIG. In step S208, a process for obtaining a motion vector 160 representing a positional deviation between images is performed on the plurality of spherical projection image data 158 obtained through the process of S206.

  In step S210, panorama composite image data is obtained by performing a pasting process on the plurality of spherical projection image data 158 generated in S206 while correcting the positional deviation between the images using the motion vector 160 derived in S208. Processing to generate 162 is performed.

  As described above, the camera tilt angle is measured at the time of pressing the release button, and a spherical projection image corrected by the tilt angle is generated, thereby performing image pasting by a relatively simple alignment process in the biaxial direction. It is possible to reduce the registration residual.

  In the present embodiment, it is assumed that the camera tilt angle after starting continuous shooting does not change significantly. An example in which the camera tilt angle can be changed during continuous shooting will be described in the second embodiment.

  An example of an image generated by performing the processing described above will be described with reference to FIGS. FIG. 5A shows an example of an image obtained by processing for generating a spherical projection image without considering the camera tilt angle. FIG. 5B shows an example of an image obtained by processing for generating a spherical projection image in consideration of the camera tilt angle. As apparent from a comparison between FIG. 5A and FIG. 5B, the curvature of the image of the linear object extending in the horizontal direction by the spherical projection conversion processing according to the embodiment of the present invention. It can be seen that is suppressed.

  FIG. 6 shows an enlarged view of a part of the image obtained by pasting the images shown in FIG. In the image shown in FIG. 6, the discontinuity as described with reference to FIG. 14 is hardly noticeable. At this time, since it is possible to perform smooth bonding only by simple alignment in two axes directions without performing processing such as image rotation and magnification conversion, the processing load of image bonding can be reduced. It becomes possible.

-Second Embodiment-
In the second embodiment, as will be described in detail below, when continuous shooting is performed, the camera tilt angle is measured corresponding to each shooting operation, and the measurement result is used as camera posture data 154. To be recorded. Thereafter, when each image is subjected to spherical projection conversion, the camera tilt angle obtained by measurement when the image is captured is referred to.

  The image bonding apparatus 120 according to the second embodiment of the present invention also has the same configuration as the image bonding apparatus 120 according to the first embodiment described with reference to FIG. Also, the image bonding apparatus 120 is applied to the electronic camera 100 as in the first embodiment. What is different from the first embodiment is a processing sequence described below.

(Processing sequence)
FIG. 7 is a flowchart for explaining the procedure of the image combining process performed by the electronic camera 100 including the image combining device 120 according to the second embodiment. Similar to the first embodiment, for example, when the electronic camera 100 is set to a panoramic image generation mode for generating a panoramic image and the user performs a pressing operation of the release button, the processing illustrated in FIG. 2 is performed. Is started.

  In step S <b> 700, the tilt of the electronic camera 100 in the tilt direction is measured and stored in the memory 106 as attitude data 130 of the electronic camera 100.

  In step S702, shooting for one frame is performed with exposure conditions such as focal length, shutter speed, ISO sensitivity, aperture value, etc. fixed. At this time, an arrow or the like indicating the direction in which the electronic camera 100 is shaken is displayed on the electronic camera 100, and the photographer performs shooting while swinging the electronic camera 100 so as to draw an arc according to the arrow. That is, photographing is performed while swinging the electronic camera 100 along the horizontal plane.

  The order of the processes in S700 and S702 described above may be reversed, and the camera tilt angle may be measured after each shooting operation. Alternatively, the photographing operation and the camera tilt angle measurement may be performed at substantially the same timing.

  In step S704, it is determined whether or not the release switch has been turned off. While the determination in S704 is negative, the measurement of the camera tilt angle in S700 and the shooting in S702 are repeated. In S706, which is a branch destination when the determination in S704 is affirmative, demosaicing, noise reduction, white balance, gradation conversion, and the like for each of the plurality of pre-development image data 150 obtained by the photographing operation in S702, Processing such as color correction is performed to generate post-development image data 152. This process corresponds to the process performed by the image processing unit 110 described with reference to FIG.

  In step S708, for each of the plurality of post-development image data 152 obtained by the development processing in step S706, the shooting parameters 156 such as the focal length and the camera tilt angle (camera posture data 154) measured in S700 are obtained. The spherical projection conversion and planar development are performed with reference to FIG. 5 to generate a plurality of frames of spherical projection image data 158. This process corresponds to the process performed by the image conversion processing unit 122 described with reference to FIG.

  Hereinafter, the processes of S710 and S712 are the same as the processes of S208 and S210 described with reference to FIG. 2 in the first embodiment, and thus detailed description thereof is omitted.

  In the first embodiment, the camera tilt angle is measured before the start of continuous shooting, and then the camera tilt angle is uniformly applied when spherical projection conversion processing is performed on all images obtained after continuous shooting. Is done. On the other hand, in the second embodiment, the camera tilt angle is measured once in response to the shooting operation of one frame, and a tilt angle suitable for each of the shot images is set, and spherical projection conversion is performed. This is different. As a result, even when the tilt angle of the camera fluctuates during continuous shooting, it is possible to correct misalignment between images by relatively simple alignment processing in the vertical and horizontal biaxial directions, and to prevent inconsistencies in the joints. It is possible to obtain a smooth panoramic image that is not noticeable.

  As described above, in the first and second embodiments, the example in which the image bonding apparatus 120 is mounted on the electronic camera 100 has been described, but an image is processed by the information processing apparatus and a program executed on the information processing apparatus. The bonding apparatus 120 may be realized. In that case, a program for acquiring a plurality of image data and camera posture data and for the information processing apparatus to execute the processing procedure from S204 to S210 in FIG. 2 or the processing procedure from S706 to S712 in FIG. Stored in an auxiliary storage device in the processing device. These programs are read from the auxiliary storage device, loaded onto the memory in the information processing device, and executed by the information processing device. In this case, the pre-development image data 150 or the post-development image data 152 is recorded as an image file in the image memory 142 of the electronic camera 100 together with the camera attitude data 154 and the shooting parameters 156, and the image file is input to the information processing apparatus. An image based on the generated panorama composite image data can be output to a display device or the like connected to the information processing device. It is also possible for the information processing apparatus to generate and record an image file that contains the generated panorama composite image data.

  The information processing apparatus may be a general-purpose computer, an apparatus dedicated to image processing, a video recorder, or the like.

  Although an example in which panoramic composite image data is generated based on a plurality of still image data obtained by continuously performing a series of still image shooting operations has been described above, the present invention is not limited thereto. For example, there is a case where a landscape is shot while panning the camera when shooting a moving image. When the user reproduces the moving image and finds a favorite scene from the moving image, the user selects a scene in a specific time interval and issues a command for generating panorama composite image data. The image pasting apparatus 120 can cut out a plurality of still images from moving image data in a time interval selected by the user, and perform image conversion, motion vector derivation, and panorama synthesis processing. In this case, it is assumed that the angular velocity sensor 128 measures the tilt angle of the electronic camera 100 at an appropriate time interval during moving image shooting, and the measurement result is added to the moving image data as camera posture data 154.

  The image bonding apparatus according to the present invention can be incorporated in a digital still camera or a digital movie camera. Furthermore, it can be mounted on a personal computer, a video recorder, a mobile phone, a portable information terminal device, or the like.

DESCRIPTION OF SYMBOLS 100 ... Electronic camera 102 ... Shooting optical system 104 ... Image pick-up element 106 ... Memory 110 ... Image processing part 120 ... Image pasting apparatus 122 ... Image conversion processing part 124 ... Motion vector arithmetic processing part 126 ... Composition processing part 128 ... Angular velocity sensor 140 ... Control unit 142 ... Image memory 150 ... Pre-development image data 152 ... Post-development image data 154 ... Camera attitude data 156 ... Shooting parameters 158 ... Spherical projection image data 160 ... Motion vector 162 ... Panorama composite image data

Claims (7)

The partial image data is cut out and pasted from each of a plurality of image data obtained by continuously performing a plurality of photographing operations while moving the photographing device so as to draw an arc along the horizontal plane, and the panoramic composite image data is obtained. An image pasting device to be generated,
An image acquisition unit for acquiring the plurality of image data;
Camera posture data for acquiring camera posture data including information relating to a tilt angle that is a tilt angle in the tilt direction of the photographing device detected before, after, or during the time when the plurality of photographing operations are performed by the photographing device. An acquisition unit;
A process of projecting a two-dimensional image based on the plurality of image data onto a virtual sphere having a reference axis as a polar axis, using processing parameters determined based on information about the tilt angle included in the camera posture data. An image conversion processing unit for generating a plurality of spherical projection image data,
A motion vector arithmetic processing unit for deriving a motion vector between the images based on the plurality of spherical projection image data;
A process of performing motion compensation between images based on the plurality of spherical projection image data based on a motion vector derived by the motion vector calculation processing unit, and cutting out and pasting a necessary portion from the plurality of spherical projection image data And a composition processing unit for generating panorama composite image data. The processing parameter defines an angle formed by the reference axis of the virtual sphere and the surface of the two-dimensional image,
The image conversion processing unit performs a process of projecting on the virtual spherical surface in which the reference axis is inclined so that an angle formed by the reference axis and the plane of the two-dimensional image is equal to the tilt angle. The image pasting apparatus according to claim 1, wherein the spherical projection image data is generated.   The image pasting apparatus according to claim 1, wherein the tilt angle is detected corresponding to each of the plurality of photographing operations.   The image pasting device according to any one of claims 1 to 3, wherein the motion vector calculation processing unit derives a motion vector between the respective images using a template matching method. An imaging device capable of generating a panorama composite image data by combining a plurality of image data,
An image sensor that continuously performs a plurality of image capturing operations when the image capturing apparatus is shaken so as to draw an arc along a horizontal plane, and generates a plurality of image data;
Before, after or during the plurality of shooting operations performed by the image sensor, a tilt angle that is a tilt angle in the tilt direction of the imaging device is measured, and camera posture data including information about the tilt angle is obtained. A posture detection unit to generate;
A process of projecting a two-dimensional image based on the plurality of image data onto a virtual sphere having a reference axis as a polar axis, using processing parameters determined based on information about the tilt angle included in the camera posture data. An image conversion processing unit for generating a plurality of spherical projection image data,
A motion vector arithmetic processing unit for deriving a motion vector between the images based on the plurality of spherical projection image data;
A process of performing motion compensation between images based on the plurality of spherical projection image data based on a motion vector derived by the motion vector calculation processing unit, and cutting out and pasting a necessary portion from the plurality of spherical projection image data And a compositing processing unit for generating panoramic composite image data. The partial image data is cut out and pasted from each of a plurality of image data obtained by continuously performing a plurality of photographing operations while moving the photographing device so as to draw an arc along the horizontal plane, and the panoramic composite image data is obtained. An image pasting method to be generated,
An image data acquisition step of acquiring the plurality of image data;
Camera posture data for acquiring camera posture data including information relating to a tilt angle that is a tilt angle in the tilt direction of the photographing device detected before, after, or during the time when the plurality of photographing operations are performed by the photographing device. An acquisition step;
A process of projecting a two-dimensional image based on the plurality of image data onto a virtual sphere having a reference axis as a polar axis, using processing parameters determined based on information about the tilt angle included in the camera posture data. An image conversion processing step for generating a plurality of spherical projection image data,
A motion vector deriving step for deriving a motion vector between the images based on the plurality of spherical projection image data;
A process of performing motion compensation between images based on the plurality of spherical projection image data based on the motion vector derived in the motion vector deriving step, and cutting out and pasting necessary portions from the plurality of spherical projection image data And a panorama synthesizing step for generating panorama synthesized image data. The partial image data is cut out and pasted from each of a plurality of image data obtained by continuously performing a plurality of photographing operations while moving the photographing device so as to draw an arc along the horizontal plane, and the panoramic composite image data is obtained. An image processing program for performing processing to be generated by a computer,
An image data acquisition procedure for acquiring the plurality of image data;
Camera posture data for acquiring camera posture data including information relating to a tilt angle that is a tilt angle in the tilt direction of the photographing device detected before, after, or during the time when the plurality of photographing operations are performed by the photographing device. Acquisition procedure;
A process of projecting a two-dimensional image based on the plurality of image data onto a virtual sphere having a reference axis as a polar axis, using processing parameters determined based on information about the tilt angle included in the camera posture data. Image conversion processing procedure for generating a plurality of spherical projection image data,
A motion vector derivation procedure for deriving a motion vector between the images based on the plurality of spherical projection image data;
A process of performing motion compensation between images based on the plurality of spherical projection image data based on the motion vector derived in the motion vector derivation procedure, and cutting out and pasting a necessary portion from the plurality of spherical projection image data And a panorama composition procedure for generating panorama composite image data.