JP2020036249A - Image processing device that performs tilt correction - Google Patents

Abstract

To provide an image processing device which can determine whether tilt correction is necessary or not when shooting with a tilt angle on a camera and can obtain a high-quality panoramic image.SOLUTION: An image processing device obtains a tilt correction coefficient on the basis of posture information of an imaging unit or an arbitrary set value, creates a determination temporary image in which tilt correction has been performed on at least two captured images, calculates a correspondence between feature points for each of the determination temporary image and the image before the tilt correction is performed, compares the number of correspondence between the determination temporary image and the image before the tilt correction is performed to determine the magnitude relationship, and evaluates the reliability of posture information and correspondence calculation. Further, the image processing device determines a tilt correction coefficient from any of the posture information and the magnitude relationship of the number of correspondences on the basis of the evaluation of the reliability, corrects the tilt of the image, and synthesizes the tilt-corrected image.SELECTED DRAWING: Figure 4

Description

  The present invention relates to an image processing apparatus and method for obtaining a wider range of images, that is, a panoramic image, and a program by synthesizing continuously shot images while sequentially changing the imaging direction so that a common region is captured. In particular, the present invention relates to an image processing apparatus and method capable of obtaining a high-quality panoramic image, and a program.

  Conventionally, a unit image forming a part of a shooting range is picked up by sequentially changing a shooting direction, and an image region of a predetermined size forming a part of the unit image picked up in the shooting step is cut out so as to generate an overlapping region. A technique has been disclosed for generating a panoramic image by successively superimposing cut-out image areas (Patent Document 1).

  In the panoramic shooting, since the vertical displacement of the swing causes a vertical displacement, an image in which the vertical direction is cropped after the panoramic image is synthesized is output. FIG. 9A shows a result obtained by aligning and cropping a panoramic image captured without tilt. Since the captured four images do not interrupt the cropping range, a panoramic image connecting all the images is finally output.

  On the other hand, if a shift occurs beyond the crop area, a panoramic image having an area without an image is generated. Therefore, the image before the shift beyond the crop area occurs is used for synthesis. The resulting image has a shorter angle of view than intended.

  FIG. 8A is a schematic diagram showing the positional relationship between a camera and a subject when panoramic shooting is performed by tilting, and the shape of a captured panoramic image. When the photographer 800 swings the camera 802 along the horizon to photograph an object 801 such as a towering building group, the camera 802 is often tilted up and down by an angle of 805 so as to look up at the object 801 as shown in FIG. In some cases, the camera is tilted. At this time, if the image is taken in a state where the axis 803 for rotating the camera is not parallel to the image element surface 804, the obtained image becomes a sector 807 attached to the narrowed cylinder 806, which is developed two-dimensionally. Then, a fan-shaped horizontally long image 808 is obtained. In this case, the image to be captured is shown in FIG. 9B, and when the acquired images are aligned, the cropping range within the dotted line is changed at the time when the fourth image in the figure is combined due to distortion due to tilt. Deviate. For this reason, only the third image can be used for synthesis, and the image has an angle of view shorter than the photographer intends. On the other hand, FIG. 9C shows a case where the image of FIG. 9B subjected to the tilt correction is synthesized, and a panoramic image having a longer range than that of FIG. 9B because distortion of the image is reduced by the tilt correction. Can be obtained.

  Patent Literature 2 discloses a method of calculating a distortion evaluation value from a tilt angle detected by a gyro sensor and synthesizing image data from the evaluation value when combining a plurality of image data. .

  Patent Literature 3 discloses an image processing method for evaluating images synthesized by a plurality of types of image synthesis methods and selecting an appropriate synthesis method from the images. In the evaluation method, a corresponding point is extracted in an overlapping area between images to calculate a combining error, and the combining method is evaluated by comparing the extracted value with a predetermined value or comparing the magnitude of the combining error.

JP 2005-328497 A Japanese Patent No. 4174122 Japanese Patent No. 5875248

  However, in panoramic shooting, even when shooting is performed with a tilt angle, an image is not always improved by tilt correction. For example, as shown in FIG. 8B, the photographer 810 shoots the camera 812 with a tilt angle 815, but if the axis 813 for rotating the camera 812 is parallel to the image element surface 814, The resulting image is a rectangle 817 attached to a cylinder 816, and when this is developed two-dimensionally, it becomes a rectangular image 818. The captured image is not distorted, and the tilt correction is not required. As shown in FIG. 8C, similarly to FIG. 8A, the photographer 820 photographs the camera 822 with the rotation axis 823 not parallel to the image element surface 824 and with an angle 825. However, when the object 821 to be photographed has a surface shape (for example, a dome) substantially similar to the locus of rotation, the photographed image 828 is not distorted as a result, and the tilt correction is unnecessary. When the distortion of the panoramic image is determined only by the tilt angle of the gyro sensor as in Patent Literature 1, the tilt correction is performed even in a scene that does not require the tilt correction as shown in FIGS. 8B and 8C. However, there is a problem that the image is deteriorated.

  On the other hand, according to the method described in Patent Literature 3, the scene of FIG. 8A that requires tilt correction and the scene of FIG. 8B and FIG. It is possible to perform the synthesis. However, in a scene where the number of corresponding points in an image is small, there is a problem that the accuracy of the determination is reduced.

  In view of the above, it is an object of the present invention to be able to discriminate between scenes requiring tilt correction (a in FIG. 8) and unnecessary scenes (b and c in FIG. 8) when shooting with a tilt angle on the camera. Another object of the present invention is to provide an image processing apparatus and method, and a program that can prevent a decrease in determination accuracy and obtain a high-quality panoramic image even in a scene where the reliability of corresponding point extraction of an image is insufficient.

In order to achieve the above object, an image processing apparatus according to the present invention includes:
An image processing apparatus for sequentially inputting images captured by an imaging device and connecting the input images to generate a composite image, wherein an imaging unit configured to capture an image using the imaging device and attitude information of the image processing device are acquired. An orientation information acquiring unit, an image correcting unit for geometrically deforming an image captured by the imaging unit, and calculating a correspondence between a feature point of the reference image and a feature point of the target image, and calculating the correspondence. An image alignment unit that performs image alignment based on the position information, a reliability evaluation unit that evaluates the reliability of the posture information and the correspondence calculation, a tilt correction coefficient setting unit that sets a tilt correction coefficient, and the position An image synthesizing unit for synthesizing the combined images, at the start of the photographing, the posture information obtaining unit obtains the posture information of the imaging unit, and the correction coefficient setting unit sets the posture information or an arbitrary set value. The tilt correction coefficient is input to the image correction unit, and the image correction unit creates a determination temporary image in which tilt correction has been performed on the captured at least two images, and the image alignment unit determines The correspondence between feature points is calculated for each of the temporary image for determination and the image before the tilt correction is performed, and the corresponding number is compared between the temporary image for determination and the image before the tilt correction is performed. A relationship is obtained, the reliability evaluation unit evaluates the reliability of the posture information and the correspondence calculation, and inputs the reliability to the tilt correction coefficient setting unit, and the tilt correction coefficient setting unit evaluates the reliability based on the reliability evaluation. Inputting a tilt correction coefficient to the image correction unit from any of the attitude information of the imaging unit and the magnitude relationship of the number of correspondences, tilt-corrects the image captured based on the tilt correction coefficient by the image correction unit, and performs the image synthesis. In the department By joining the corrected image and generating a composite image.

  ADVANTAGE OF THE INVENTION According to this invention, even in the scene where the reliability of the corresponding point extraction of an image is inadequate, an image processing apparatus and method, and a program which can perform a tilt correction appropriately and can obtain a high-quality panoramic image Can be provided.

1 is a conceptual diagram illustrating a configuration of an image processing device according to an embodiment. FIG. 4 is a flowchart illustrating a processing flow according to the first embodiment. It is a conceptual diagram about exchange between the components of a 1st embodiment. FIG. 3 is a flowchart for explaining a processing flow of the first embodiment in detail; It is a conceptual diagram about the tilt angle of an acceleration sensor and a difference image processing apparatus. FIG. 4 is a conceptual diagram related to image alignment and the reliability of a gyro sensor. It is a conceptual diagram regarding calculation of reliability of image alignment. It is a conceptual diagram about the subject which the invention solves. It is a conceptual diagram about the subject which the invention solves.

  Hereinafter, preferred embodiments of the present invention will be described in detail with reference to the accompanying drawings.

(Embodiment 1)
FIG. 1 is a block diagram illustrating a basic configuration of the image processing apparatus according to the present embodiment.

  The image processing apparatus 100 may be a camera such as a digital camera or a digital video camera, or may be any electronic device having a camera function, such as a mobile phone with a camera function and a computer with a camera.

  The optical system 101 includes a lens, a shutter, and an aperture, and forms light from a subject on the image sensor 102 under the control of the CPU 103. The lenses included in the optical system 101 include, for example, a focus lens, a zoom lens, a shift lens, and the like. The shift lens is a correction unit that is used to optically correct blur (image blur) generated in a captured image due to a shake applied to the image processing apparatus 100. CPU is an abbreviation for Central Processing Unit. An image sensor 102 such as a CCD image sensor or a CMOS image sensor converts light focused through the optical system 101 into an image signal. CCD is an abbreviation for Charge Coupled Device. CMOS is an abbreviation for Complementary Metal Oxide Semiconductor.

  The posture information acquisition unit 105 may be a gyro sensor or an acceleration sensor, detects the angular velocity indicating the moving amount of the image processing apparatus 100 or the inclination of the apparatus with respect to the gravitational acceleration, converts the detected inclination into an electric signal, and converts the inclination into an electric signal. Communicate to The CPU 103 realizes the functions of the image processing apparatus 100 by controlling each unit constituting the image processing apparatus 100 according to an input signal or a program stored in advance. The primary storage device 104 is, for example, a volatile device such as a RAM, stores temporary data, and is used for the operation of the CPU 103. RAM is an abbreviation for Random Access Memory. The information stored in the primary storage device 104 is used by the image processing unit 106 or recorded on the recording medium 107.

  The secondary storage device 108 is a non-volatile storage device such as an EEPROM, for example, and stores a program (firmware) for controlling the image processing apparatus 100 and various kinds of setting information, and is used by the CPU 103. EEPROM is an abbreviation for Electronically Erasable Programmable Read Only Memory.

  The recording medium 107 records data of an image obtained by shooting, which is stored in the primary storage device 104. Note that the recording medium 107 is detachable from the imaging device 100 like a semiconductor memory card, for example, and the recorded data can be read out by attaching it to a personal computer or the like. That is, the image processing apparatus 100 has a mechanism for attaching and detaching the recording medium 107 and a read / write function. The display unit 109 displays a viewfinder image at the time of shooting, displays a shot image, and displays a GUI image for interactive operation. GUI is an abbreviation for Graphical User Interface. The operation unit 110 is an input device group that receives a user's operation and transmits input information to the CPU 103, and may be, for example, a button, a lever, a touch panel, or an input device using a voice or a line of sight.

  The temperature information acquisition unit 111 is installed in the image processing apparatus 100 and monitors the temperatures of the posture information acquisition unit 105 and the CPU 103.

  Note that the image processing apparatus 100 has a plurality of image processing patterns applied to the captured image by the image processing unit 106, and the pattern can be set as the imaging mode from the operation unit 110. The image processing unit 106 performs not only image processing called so-called development processing, but also adjustment of color tone according to the shooting mode. Note that at least a part of the function of the image processing unit 106 may be implemented by the CPU 103 as software.

  In the panoramic photographing of the present invention, the photographing is performed by moving the camera continuously or by moving the camera with an automatic camera platform or the like. The photographing is performed so that the plurality of photographed images include the common area of the subject. A feature point of a common area of each image is extracted, and a movement vector is detected as to how much the feature point has moved. The movement vector is converted into, for example, an affine transformation coefficient, and the two images are superimposed so that the feature points coincide with each other, whereby an image in which a portion other than the common area is expanded is obtained. By repeating this a plurality of times, a panoramic image in a wider range than that of a single image is generated.

  The details of the panoramic photographing of the present invention will be described mainly with reference to FIGS. Note that although FIG. 3 shows the tilt correction coefficient setting units 302a and 302b and the image correction units 303a and 303b, these are the same, and are separately illustrated for clarity of explanation. In FIG. 3, each component is described as directly exchanging information with each other, but actually exchanges information via the primary storage device 104 and the CPU 103.

  In the panoramic photography according to the present invention, the flow chart of the image processing of the first few images that have been photographed is different from the flow chart of the image processing shown in FIG. Here, the first two sheets and the third and subsequent sheets will be described as examples. For the first and second sheets, it is determined whether or not the tilt correction is performed. Specifically, a coefficient for performing the tilt correction is set. However, when the tilt correction is not desired, a coefficient is set so that the correction is not performed. For the third and subsequent sheets, the execution determination of the tilt correction is not performed, and the composition is performed according to the contents determined for the previous two sheets. In other words, when the execution of the tilt correction is determined for the first and second images, the images subjected to the tilt correction similar to those of the first and second images are combined in the third and subsequent images. Hereinafter, the processing for the first and second sheets and the processing for the third and subsequent sheets will be described separately.

  First, the processing flow for the first and second sheets will be described. The imaging unit 300 has an optical system for guiding light to the image sensor, an image sensor for converting received light into an electric signal, and a developing system for converting the electric signal into image data such as YUV. On the other hand, the posture information acquisition unit 301 acquires posture information of the imaging device per unit time in parallel with the photographing of the imaging unit. As an example of the posture information acquisition unit 301, an acceleration sensor and a gyro sensor are suitably used.

  The S201 image data that has been captured by the imaging unit 300 (S200, S400) and developed is input to the image correction unit 303a 3001, and image preprocessing A (S202) is performed. At the same time as shooting, the value of the posture information obtaining unit 301 is obtained and stored in the primary storage device 104 in step S401. The image preprocessing A (S202) is a process for correcting image distortion, and specifically includes cylindrical transformation (S2020, S402) and tilt correction (S2021, S403). In panoramic imaging, it is preferable to perform cylindrical conversion (S2020, S402) so that positioning can be performed easily. An existing method is used for the cylinder conversion. It is also preferable to correct the distortion of the optical system 101 at this time.

  On the other hand, in the tilt correction (S2021, S403), the tilt correction coefficient setting unit 302b creates a temporary determination image used to determine whether to perform tilt correction. The tilt correction coefficient setting unit 302a calculates the tilt angle of the first image captured from the attitude information of the imaging device acquired from the attitude information acquisition unit 301, and inputs the angle to the image correction unit 303a. As a method of calculating the tilt angle of the imaging apparatus, an acceleration sensor will be described as an example with reference to FIG. As shown in FIG. 5A, the horizontal direction in the imaging apparatus is the x axis, the optical axis is the y axis, and the direction of gravity is the z axis. As shown in FIG. 5B, the tilt angle of the imaging device is the rotation angle of the yz axis around the x axis, that is, the angle between the horizontal direction and the y-axis direction of the imaging device is the tilt angle θ. is there. The tilt angle θ is obtained from the following equation 1.

  In Expression 1, Ay and out are outputs of the acceleration sensor in the y-axis direction, and Az and out are outputs of the acceleration sensor in the z-axis direction. As a method of obtaining the inclination of the posture information, there is a method of taking the offset with the gyro sensor in a horizontal state before imaging and integrating and calculating the output. However, when the integration time is long, the integration error due to the stability of the bias is required. Therefore, an acceleration sensor is preferably used.

  Next, the image correction unit 303a obtains the focal length d of the imaging system from the optical system 101 and the pixel pitch p of the imaging device from the secondary storage device 108 using the following Expressions 2 and 3, and sets the tilt correction coefficient. A conversion coefficient for the following tilt correction is obtained in combination with the (3003) tilt angle θ input from the unit 302a.

  The image correction unit 303a performs the tilt correction by applying the tilt correction coefficient obtained by Expression 2 to the first and second image data subjected to the cylindrical conversion (S2020, S402) (S2021, S403). The image correcting unit 303a transfers the image after the tilt correction to the image positioning unit 304 (3004). Here, the coefficient of the tilt correction is derived from the projective transformation, but another conversion method may be used as long as the tilt correction is possible.

  The image positioning unit 304 performs a corresponding point calculation process (S203, S404) on each of the two pairs of images, the image subjected to the tilt correction and the image not subjected to the tilt correction.

  The corresponding point calculation process is to detect a corresponding point between two images. The detection of the corresponding points is performed by extracting the feature points common to the images. A known method can be used for extracting the feature points. As an example, a method of extracting only luminance information of two images that have been subjected to the cylindrical transformation (S2020, S402), subtracting one or several pixels from the original image, and calculating the absolute value thereof (template matching) The edge is extracted by a method of subtracting an image obtained by applying a high-pass filter to an image obtained by extracting only luminance information from the original luminance information image. A plurality of edge images are subtracted while being shifted, and a position where the difference is reduced is calculated. If the difference is equal to or less than an arbitrary threshold value, it is regarded as a match and is counted as a corresponding point. In extracting feature points, it is preferable to divide an image into a plurality of blocks and extract feature points. This is to avoid erroneously detecting similar similar feature points that are actually different in the image as corresponding points by dividing the image into blocks. Although the block division depends on the number of pixels and the aspect ratio of the image, generally, 12 × 4 × 3 blocks or 96 × 64 blocks are preferable. If the blocks are too fine, the feature points approach each other and include an error. Conversely, if the blocks are too large, different similar feature points are erroneously detected as corresponding points as described above. Thus, the optimal number of blocks can be appropriately selected according to the number of pixels, the ease of finding feature points, the angle of view of the subject, and the like. The image alignment unit 304 performs the above-described corresponding point calculation process on each of the two pairs of images, the image that has been tilt-corrected and the image that has not been tilt-corrected, and calculates the number of corresponding points of each pair (the number of blocks is set to the maximum value. Is input to the tilt correction coefficient setting unit 302b (3008).

  Next, the tilt correction coefficient setting unit 302b sets a tilt correction coefficient based on the posture information of the imaging device acquired from the posture information acquisition unit 301 separately from the above-described calculation of the corresponding points (S412, S414). In a scene requiring tilt correction (FIG. 8A), the imaging device is rotating in the roll direction. On the other hand, in a scene (b and c in FIG. 8) in which the tilt correction is unnecessary, the imaging device is not rotating in the roll direction. The rotation in the roll direction can be detected by, for example, a gyro sensor. The posture information acquisition unit 301 inputs the amount of rotation of the gyro sensor in the roll direction between the first and second images to the tilt correction coefficient setting unit (3005). Note that the output from the gyro sensor itself is the angular velocity, but in panoramic shooting, it is necessary to determine how much the camera has been swung (rotated) since the previous shooting. The angular velocity is integrated up to and the rotation angle from the previous time is calculated when the second image is taken. It is preferable that the output value of the gyro sensor before imaging is set as the camera shake amount and subtracted when calculating the rotation amount in the roll direction (not shown).

  The tilt correction coefficient setting unit 302b that has received the number of corresponding points from the image positioning unit 304 compares the number of corresponding points between pairs in S407, and sets the larger tilt correction coefficient in the image correction unit 303b in S408. In other words, when the pair subjected to the tilt correction has a larger number of corresponding points than the pair of the original image, the tilt correction coefficient setting unit 302b inputs the tilt correction coefficient used for calculating the determination image to the image correction unit 303b. Conversely, when the original image pair has more corresponding points than the pair that has undergone the tilt correction, the tilt correction coefficient setting unit 302b inputs the following coefficient 1 to the image correction unit 303b as the tilt correction conversion coefficient. So that the tilt correction is not performed.

  When the rotation amount of the imaging device in the roll direction is received from the posture information acquisition unit 301, the tilt correction coefficient setting unit 302b causes the image correction unit 303b to calculate the tilt image when the rotation amount is equal to or larger than the threshold. A correction coefficient is input (S412). Conversely, when the gyro sensor is not rotating in the roll direction between the first and second images, the tilt correction coefficient setting unit 302b inputs the coefficient 1 to the image correction unit 303b as a tilt correction conversion coefficient ( S414).

  Whether the tilt correction coefficient set by the tilt correction coefficient setting unit 302b is set by the tilt correction coefficient calculated based on the number of corresponding points or the tilt correction coefficient from the attitude control information depends on the reliability of the image registration and attitude information. Determined by gender. Hereinafter, a method of calculating the reliability of image alignment and posture information (S204, S405) will be described.

  The reliability of the corresponding point calculation and the reliability of the posture control information (especially the gyro sensor) are determined based on the information reliability received from the information input unit (3005) from the image alignment unit 304 and the information input unit (3006) from the posture control unit 301. The judgment is made by the evaluation unit 305, and the judgment is passed to the tilt correction coefficient setting unit 302b (3007). The reliability is divided into three reliability sections separated by two large and small thresholds (alignment: 600, 601; attitude control: 610, 611). That is, when the values are sufficiently reliable (alignment: Gr 604, attitude control: Pr 614), when the values are not reliable (alignment: Gur 602, attitude control: Pur 612), and in the middle (position alignment: Gpr 603, Posture control: Ppr 613).

  For example, in the case of a gyro sensor, what happens to the above classification will be described. There are signals output even when the gyro sensor is stationary, and this is called an offset. This offset is different for each individual, and the intensity generally changes depending on the temperature and the like. Normally, the offset amount at the time of standing at normal temperature is stored in the secondary storage device 108 in the imaging device, and a value obtained by subtracting the offset from the output signal when used is used. However, under a high or low temperature, the offset is different from a normal temperature, so that the offset cannot be accurately removed. In addition, in panoramic imaging, dozens of images may be taken at one time, and the temperature of the imaging device is likely to rise. In such a case, the amount of movement acquired from the gyro sensor information cannot be accurately calculated because the offset cannot be completely removed. For this reason, the reliability is considered to be lower as the gyro sensor temperature at the time of capturing the first or second image obtained from the temperature information obtaining unit 111 is farther from the normal temperature. Specifically, when the temperature of the gyro sensor is out of the operating temperature range guaranteed by the manufacturer, the value cannot be used for the determination, and thus the classification is classified as unreliable (Pur). On the other hand, within the operating temperature range, particularly when the temperature is around room temperature, the reliability is classified as high (Pr), and when the temperature is between Pur and Pr, the reliability is classified as medium (Ppr).

  The reliability of the corresponding point calculation is based on at least one of the ratio of the low contrast area in the image, the ratio of the repetitive pattern in the image, the ratio of the moving object, and the number of corresponding points calculated in the above. . FIG. 7 shows the positional relationship between the first image 701, the second image 702, and the third image 703 and the positional relationship between the two corresponding point reliability calculation sections 704 and 705 when the swing direction is the 700 direction. It is a conceptual diagram. The first corresponding point reliability calculation section is an area of the first or second alignment frame 704. When there are many low contrast areas in each block in the frame, the number of feature points extracted as a whole is small, so it is determined that the reliability when comparing the number of corresponding points in the next step is low. When each block is occupied by a repetitive pattern such as a tile, for example, a plurality of positions where the difference between two images is small are detected, and the correspondence between feature points is not correctly calculated. Is determined to be low. Furthermore, a block in which a corresponding point whose movement amount is larger than that of another block is detected has a high possibility of detecting a moving object, and is determined to have low reliability. If the calculated number of corresponding points is less than 10% of the maximum value, or if the corresponding points are continuously arranged linearly (only horizontal lines are detected), it is determined that the reliability is low.

  The reliability of the posture information uses the temperature of the gyro sensor (difference from the reference temperature) as an evaluation criterion.

  The reliability evaluation unit 305 classifies the reliability of the corresponding point calculation and the attitude information based on the above-described determination criteria, and passes the information to the tilt correction coefficient setting unit 302b (3007). As shown in FIG. 6C, the tilt correction coefficient setting unit 302b determines whether to set the tilt correction coefficient based on the corresponding point calculation or the posture information based on a combination of the corresponding point calculation and the reliability classification of the gyro sensor. Then, the image correction unit 303b sets the tilt correction coefficient. When the reliability division of the corresponding point calculation and the posture information is different, the tilt correction coefficient of the higher reliability division is adopted (r> pr> ur). For example, when the reliability of the corresponding point calculation in S411 is not reliable (Gur), and when the reliability of the gyro sensor is sufficiently high (Pr) or when the reliability is moderate (Ppr), only the rotation information of the gyro sensor is used. Set the coefficient. On the other hand, if the corresponding point calculation and the reliability section of the posture information are the same in step S409, that is, if the combination is Pr and Gr, and the combination of Ppr and Gpr is the same conclusion (shift correction coefficient) as in step S410, For example, the tilt correction coefficient that has already been set from the corresponding point calculation result is left, and if the conclusion is a different conclusion, it is determined whether the reliability can be improved by adding a third or later sheet described later. In the case of the combination of Pur and Gur, since both are unreliable results, a determination S413 is made as to whether the reliability can be improved by adding a third or later sheet described later.

  If both the reliability of the corresponding point calculation and the reliability of the posture information are low, or if both have different conclusions, the tilt correction coefficient setting unit 302b sends the third and subsequent sheets to the reliability evaluation unit 305. An instruction is issued to determine whether the reliability can be improved by determining whether or not to use the tilt correction (3010, S413). The reliability calculation section 2 (705) of the corresponding point in FIG. 7 is a portion that is a “gap” between the third image and the third image of the second image. It can be determined whether the accuracy of point calculation can be improved. The reliability evaluation unit 305 performs at least one of the low contrast area, the repetitive pattern area, and the moving body area in the reliability calculation section 2 (705) of the second corresponding point. If the reliability evaluation unit 305 determines that the reliability of the corresponding point calculation increases in the third image, the above flow is performed again for the second and third images for the third image. On the other hand, when the reliability evaluation unit 305 receives an input indicating that the reliability division of the corresponding point calculation does not increase in the reliability calculation section 2 (705) of the second corresponding point, the tilt correction coefficient setting unit 302b outputs the tilt correction coefficient. Is set as the coefficient 1, and the process is terminated.

  Next, the processing flow for the third and subsequent sheets (however, when the third sheet is used for the reliability calculation in the above flow, the fourth and subsequent sheets) will be described. As shown in FIG. 2B, for the third and subsequent images, panoramic imaging is performed using the tilt correction coefficients determined for the first and second images. After imaging the third and fourth images 210, development processing is performed S211 and cylinder conversion S2120 and tilt correction S2121 are performed as image pre-processing S212. The tilt correction coefficient used in the tilt correction is determined according to the flow of FIG. 2A, and the input to the image correction unit 303b is used, and the same coefficient is used in the subsequent imaging. This is because, when a tilt correction coefficient that differs between imagings is used, the vertical size of the corrected image and the relative position of the subject in the image with respect to the entire image are different, so that the alignment work becomes difficult. . Thereafter, corresponding point detection S213 is performed on the two images subjected to the tilt correction (the coefficient is not substantially multiplied in the case of coefficient 1) to detect a movement vector indicating how much the corresponding point has moved.

  Next, in order to find out how much the two images should be moved and combined, the fourth image is transformed with respect to the third image to find a conversion method that minimizes the difference (S214). The existing method is applied to the conversion method. Specifically, there are Euclidean transformation, affine transformation, which can rotate and translate, and projective transformation, which can perform trapezoidal correction. However, linear transformation, which cannot translate, cannot be used.

  Movement and rotation to the X-axis and Y-axis can be performed by Euclidean conversion, but camera shake when actually shooting with a camera includes hand movements in the front-rear direction, pan / tilt directions, etc. , Skew, etc., can be corrected. Taking this affine transformation as an example, when the coordinates (x, y) of the corresponding reference point move to the coordinates (x ′, y ′), it is expressed by the following equation 4.

  At this time, the 3 × 3 matrix is called an affine coefficient. An affine coefficient can be calculated from this equation by affine transformation if at least three corresponding point shifts can be detected. However, if the corresponding point is found only at a close distance or becomes linear, the corresponding points that are far from each other and are not linear are selected because the conversion of the point farther than the corresponding point will be inaccurate. Is preferred. On the other hand, when a plurality of corresponding points are detected, the corresponding points close to each other are omitted, and the rest is normalized by the least square method.

  At step S215, the fourth image is deformed with the affine coefficients calculated in this way to perform alignment, and two images are combined at S216. In addition, it is preferable to perform synthesis while changing the synthesis ratio so that boundaries are not conspicuous.

  In the present embodiment, the configuration in which the first and second images used for the tilt correction determination are not used as the final composite image has been described, but the first and second images are stored in the primary storage device 104. In addition, it is preferable to return to the top of the image and perform the synthesis after the synthesis of the third and subsequent sheets is completed. If the image registration is not reliable (Gur) at the time of the additional combination of the first and second images, the position adjustment is performed using the gyro sensor information (obtained at the time of capturing the first and second images and held in the primary storage device 104). Is preferred.

  The above processing provides an image processing apparatus, method, and program that can appropriately perform tilt correction and obtain a high-quality panoramic image even in a scene where the reliability of corresponding point extraction of an image is insufficient. can do.

(Embodiment 2)
In the first embodiment, cylindrical transformation and tilt correction are performed as preprocessing of images on the first two sheets.

  However, when the tilt angle of the image processing apparatus 100 is very large (for example, 45 °), the upper part of the cylinder 806 in FIG. That is, the curvature of the image 808 increases. If the corresponding point of the image is calculated in a state where the curvature is large, the same part of the subject will not be included in the same frame and it will not be possible to evaluate it.Therefore, rotation correction that corrects the curvature of the image enough to enter the frame before tilt correction will be performed. It is desirable to do.

  In a second embodiment, an example will be described in which rotation correction is performed before tilt correction. The flowchart uses FIG. 4 used in the first embodiment.

  In the first embodiment, the tilt correction S403 is performed after the cylindrical conversion S402. In the second embodiment, the image rotation is corrected between the cylindrical conversion and the tilt correction. As the rotation correction amount, a rotation angle calculated from the roll value of the gyro sensor is used, and rotation is performed by a known conversion (for example, Euclidean conversion) that can perform rotation. The method of calculating the rotation angle is the first embodiment, and a method of calculating the angle in the roll direction when calculating the correction coefficient is used.

  As described above, in the second embodiment, by performing the rotation correction before the tilt correction, the corresponding points can be calculated even when the inclination of the image processing apparatus is large.

(Embodiment 3)
In the first embodiment, the value of the acceleration sensor is used for calculating the tilt correction coefficient for the determination image. However, when the tilt angle of the image processing apparatus 100 is very small, the accuracy of the tilt angle obtained from the acceleration sensor may be insufficient.

  In the third embodiment, in addition to the tilt angle calculated from the value of the acceleration sensor, the tilt correction is performed using at least one angle in an angle range of +15 degrees to −15 degrees to the tilt angle. The angle with the largest number of corresponding points is adopted as the correct tilt angle. The reason for the angle range is that the error of the angle calculated from the acceleration sensor when the image processing apparatus 100 is held by hand is about 15 degrees at the maximum.

  Note that the angle range is based on the accuracy of the current acceleration sensor, and the angle range may be narrowed when the accuracy of the acceleration sensor is improved. Conversely, if a lower-precision acceleration sensor is used for cost reduction, it is desirable to widen the angle range.

  As described above, in the third embodiment, appropriate tilt correction can be performed even when the tilt angle of the image processing apparatus 100 is minute.

  As described above, the preferred embodiments of the present invention have been described, but the present invention is not limited to these embodiments.

100 image processing apparatus, 101 optical system, 102 image sensor, 103 CPU,
104 primary storage device, 105 posture information acquisition unit, 106 image processing unit,
107 storage medium, 108 secondary storage device, 109 display unit, 110 operation unit,
111 temperature information acquisition unit, 300 imaging unit, 301 attitude information acquisition unit,
302a tilt correction coefficient setting unit, 302b tilt correction coefficient setting unit,
303a image correction unit, 303b image correction unit, 304 image alignment unit,
305 Reliability Evaluation Unit

Claims (11)

An image processing apparatus for sequentially inputting images captured by an image sensor and connecting the input images to generate a composite image,
An imaging unit that performs imaging with the imaging device;
A posture information acquisition unit for acquiring posture information of the image processing apparatus,
An image correction unit that geometrically deforms an image captured by the imaging unit,
An image alignment unit that calculates the correspondence between the feature points of the reference image and the feature points of the target image, and performs image registration based on the calculation result of the correspondence.
A reliability evaluation unit that evaluates the reliability of the posture information and the correspondence relationship calculation,
A tilt correction coefficient setting unit for setting a tilt correction coefficient,
An image combining unit that combines the aligned images,
At the start of the shooting,
The posture information acquisition unit acquires posture information of the imaging unit,
The correction coefficient setting unit inputs a tilt correction coefficient to the image correction unit based on the posture information or an arbitrary setting value,
The image correction unit creates a temporary image for determination in which the at least two captured images are tilt-corrected,
The image registration unit calculates the correspondence between feature points for each of the temporary image for determination and the image before performing the tilt correction, and calculates the correspondence between the temporary image for determination and the image before performing the tilt correction. Compare the number of correspondences and find the magnitude relationship,
The reliability evaluation unit evaluates the reliability of the posture information and the correspondence calculation and inputs the posture correction coefficient to the tilt correction coefficient setting unit,
The tilt correction coefficient setting unit inputs the tilt correction coefficient to the image correction unit from any of the magnitude information of the attitude information and the corresponding number of the imaging unit based on the evaluation of the reliability,
The image correction unit corrects the image captured based on the tilt correction coefficient,
An image processing apparatus, wherein the image synthesizing unit joins the tilt-corrected images to generate a synthesized image.   An acceleration sensor is provided as the posture information acquisition unit, and when creating the provisional image for determination, in the tilt correction, of the pitch angle and the yaw angle of the acceleration sensor information at the time of imaging, of the direction orthogonal to the swing direction. The image correction unit according to claim 1, wherein the angle is used as a parameter.   At the time of creating the provisional image for determination, in addition to the tilt angle obtained from the acceleration sensor, a tilt correction coefficient is calculated using at least one angle in an angle range of +15 degrees to −15 degrees to the tilt angle. The image correction unit according to claim 2, wherein the calculation is performed.   The gyro sensor is provided as the posture information acquisition unit, and the rotation correction is performed by performing Euclidean transformation using a roll angle of the gyro sensor at the time of creating the temporary image for determination. Image correction unit.   The reliability evaluation unit according to claim 1, wherein the reliability of the posture information of the imaging unit is a gyro sensor or a temperature near the gyro sensor (a difference from a reference temperature) as an evaluation criterion.   The reliability of the corresponding point calculation is at least one of the ratio of the low contrast area in the image, the ratio of the repeating pattern in the image, the ratio of the moving object, the number of corresponding points, and the arrangement of the corresponding points. The reliability evaluation unit according to claim 1, wherein one of the evaluation criteria is used.   The reliability evaluation unit according to claim 1, wherein the reliability evaluation unit divides the reliability of the posture information and the number of correspondences of the imaging unit into three levels of reliability divisions divided by two reliability thresholds. Sex evaluation department.   The tilt correction according to claim 1, further comprising the reliability evaluation unit according to claim 7, wherein when the reliability divisions of the attitude information and the number of correspondences of the imaging unit are different, a result with higher reliability is adopted. Coefficient setting section.   In the case where the sections of the reliability of the attitude information and the corresponding number of the imaging unit are the same but the calculated tilt correction coefficients are different, or if both are not reliable, the part of the reference image or the corresponding image 9. The tilt correction coefficient setting unit according to claim 8, wherein the reliability of the image registration of the image pickup is estimated.   When setting the tilt correction coefficient from the orientation information of the imaging unit, the orientation information of the imaging unit to be used is a roll angle of the gyro sensor at the time of imaging, and is used for determination when the roll angle is a value indicating rotation. The tilt correction coefficient setting unit according to claim 1, wherein the tilt correction coefficient used for creating the temporary image is set in the image correction unit. 2. The method according to claim 1, wherein the corresponding number is a matching number of template matching or a vector number of an image alignment vector, and the setting of the tilt correction coefficient adopts a larger number of the matching number or the vector number. The tilt correction coefficient setting unit described.