JP2014086097A - Image processing apparatus, image processing method, image processing program, and recording medium - Google Patents

Abstract

PROBLEM TO BE SOLVED: To provide an image processing apparatus configured to prevent accumulation of errors to obtain a high-quality panoramic image even if an image captured from a different direction is included when input images are sequentially joined.SOLUTION: An image processing apparatus includes an input unit, a selection unit, a matching unit, an estimation unit, and a synthesis unit. The input unit sequentially inputs images. The selection unit selects a reference image from the input images. The matching unit calculates correspondence relation between a feature point of the reference image and a feature point of a target image. The estimation unit estimates a transformation for associating a coordinate system of the reference image with the coordinate system of the target image, by use of position information of a pair of feature points whose correspondence relation has been calculated by the matching unit, on the assumption that motion between the reference image and the target image is only rotation of an imaging element. The synthesis unit generates a synthetic image by combining the input images with the target image on the basis of the transformation.

Description

  The present invention relates to an image processing apparatus, an image processing method, an image processing program, and a recording medium.

  2. Description of the Related Art Conventionally, as an image processing apparatus, an apparatus that creates a panoramic still image that is a single wide-angle still image by connecting captured images is known (see, for example, Patent Document 1). The image processing apparatus described in Patent Literature 1 combines a plurality of images obtained by imaging different directions from the same point. Specifically, both images are synthesized after unifying the coordinate systems of the two images using a transformation matrix. This transformation matrix is calculated by the least square method.

JP-A-11-73492

  However, in the image processing apparatus described in Patent Document 1, since the components of the conversion matrix are not particularly limited, for example, a conversion matrix that minimizes the position error between two images after conversion using the least square method is used. As a result, a matrix that changes the size of the image in the reduction direction tends to be a solution. Then, in the image processing apparatus described in Patent Document 1, when inputting target images picked up in different orientations sequentially and generating a panoramic image by sequentially synthesizing the input reference image and the target image, The reduced target images are reduced and combined, and the transformation matrix of the next target image is estimated based on the reduced target images. For this reason, errors are accumulated at the time of synthesis, and as a result, a high-quality panoramic image may not be obtained.

  In this technical field, it is desired to suppress accumulation of errors and obtain a high-quality panoramic image even when images having different imaging directions are included when sequentially joining input images. Yes.

  That is, an image processing apparatus according to an aspect of the present invention is an image processing apparatus that sequentially inputs images captured by an image sensor and generates a composite image by joining the images each time they are input. The apparatus includes an input unit, a selection unit, a matching unit, an estimation unit, and a synthesis unit. The input unit sequentially inputs images. The selection unit selects a reference image from input images including one or a plurality of images input by the input unit before the target image that is a processing target image newly input by the input unit. The matching unit calculates a correspondence relationship between the feature points of the reference image and the feature points of the target image. The estimation unit assumes that the movement between the reference image and the target image is only the rotational movement of the image sensor, and uses the positional information of the feature point pair whose correspondence is calculated by the matching unit, A conversion formula that matches the coordinate system of the target image is estimated. The synthesizing unit generates a synthesized image by connecting the reference image and the target image based on the conversion formula.

  In the image processing apparatus according to one aspect of the present invention, a conversion equation that correlates the coordinate systems of both images is estimated on the assumption that the movement between the reference image and the target image is caused only by the rotational movement of the image sensor. For this reason, the conversion formula does not include parameters such as enlargement, reduction, and parallel movement, so that it is possible to avoid the occurrence of an error due to reduction of the input target image. Furthermore, by limiting to only the rotation component, it is possible to avoid the target image that has been reduced or the like as a reference image for the next time or later, and therefore it is possible to avoid accumulation of errors. Therefore, even when the input images are sequentially stitched together, even if images with different imaging orientations are included, accumulation of errors can be suppressed and a high-quality panoramic image can be obtained.

  In one embodiment, the estimation unit may estimate a conversion formula that includes only rotational components of each axis of a three-dimensional coordinate system with the position of the image sensor as the origin.

  In one embodiment, the estimation unit prepares an objective function including a difference in position information of each pair of feature points converted using the conversion formula, and uses the optimization method so that the objective function becomes the minimum value. The conversion formula may be estimated by performing a convergence calculation. By comprising in this way, the conversion type | correspondence which makes the coordinate system between a reference | standard image and an object image correspond can be estimated with a sufficient precision.

  In one embodiment, when there are a plurality of images input by the input unit before the target image, the estimation unit uses an image input by the input unit immediately before the target image as an initial value of the convergence calculation. A conversion formula may be adopted. By configuring in this way, the convergence of calculation is accelerated, so that the calculation cost can be suppressed and the calculation speed can be improved.

  In one embodiment, the estimation unit projects the feature point pair whose correspondence has been calculated by the matching unit onto a spherical surface, and uses the positional information of the projected feature point pair and the reference image coordinate system and the target image. You may estimate the conversion formula which makes a coordinate system correspond. By configuring in this way, the coordinates after using the conversion formula can be made into a form that does not include variable division, so that the calculation cost can be suppressed and the calculation speed can be improved.

  In one embodiment, the estimation unit includes a plurality of images input by the input unit before the target image, and at least one of the images input by the input unit before the target image is a reference image. When the image overlaps the target image, the reference image, the image overlapped with the reference image, and a pair of feature points of the target image are used to correspond the coordinate system of the reference image with the coordinate system of the image overlapped with the reference image. An equation and a conversion equation that associates the coordinate system of the reference image with the coordinate system of the target image may be estimated in association with each other. With this configuration, the positional relationship between these images can be estimated in consideration of not only the positional relationship between the reference image and the target image but also the positional relationship between the reference image and other images. The accuracy of the conversion formula can be improved.

  In one embodiment, when the distance between the reference image and the target image is equal to or greater than a predetermined value, the selection unit selects a target image that is a target image that is newly input from the next time by the input unit. It may be selected as a reference image. By selecting in this way, a reference image having a large overlapping area with the target image can be selected.

  In one embodiment, when the distance between the target image and the past reference image is smaller than the distance between the target image and the reference image, the selection unit newly inputs the past reference image after the next time. You may select as a reference | standard image of the target image which is a target image. By selecting in this way, a reference image having a large overlapping area with the target image can be selected.

  In one embodiment, the matching unit further calculates a correspondence relationship between the feature point of the past reference image and the feature point of the target image when the distance between the target image and the past reference image is a predetermined value or less, The estimation unit further estimates a conversion formula that associates the coordinate system of the past reference image with the coordinate system of the target image using the position information of the feature point pairs of the past reference image and the target image, The reference image and the past reference image are made to correspond to each other by using a conversion formula that associates the coordinate system of the target image with the coordinate system of the target image, and a conversion formula that matches the coordinate system of the past reference image and the coordinate system of the target image. May be. By configuring in this way, reference images that originally have very little overlap can be matched with each other through the target image, so that a higher quality panoramic image can be obtained.

  In one embodiment, the image processing apparatus further includes a guide unit that is connected to a display unit that displays an image and displays a guide display that guides a user's camera operation on the display unit. The image is linked and the relative position is recorded as a determined pair, the combining unit outputs the combined image to the display unit, and the guide unit is connected to the current reference image of the pair whose relative position has been determined. When there is a first image whose number of hops is equal to or greater than a predetermined value and does not overlap with the current reference image, a guidance display for guiding the imaging position from the current imaging position to the image position of the first image is displayed on the display unit. You may let them. By configuring in this way, the user can avoid a situation where the error is accumulated because the positional relationship between the first image and the reference image is relatively determined via a plurality of images. The operation can be prompted.

  An image processing method according to another aspect of the present invention is an image processing method for sequentially inputting images picked up by an image sensor and connecting them each time they are input to generate a composite image. The method includes an input step, a selection step, a matching step, an estimation step, and a synthesis step. In the input step, images are sequentially input. In the selection step, a reference image is selected from input images including one or a plurality of images input in the input step before the target image that is a processing target image newly input in the input step. In the matching step, a correspondence relationship between the feature points of the reference image and the feature points of the target image is calculated. In the estimation step, it is assumed that the movement between the reference image and the target image is only the rotational movement of the image sensor, and the reference image coordinate system A conversion formula that matches the coordinate system of the target image is estimated. In the synthesis step, the reference image and the target image are connected based on the conversion formula to generate a synthesized image.

  According to this image processing method, the same effects as those of the image processing apparatus according to one aspect of the present invention described above can be obtained.

  An image processing program according to another aspect of the present invention is an image processing program for causing a computer to function in such a manner that images captured by an image sensor are sequentially input and stitched together to generate a composite image. The program causes the computer to function as an input unit, a selection unit, a matching unit, an estimation unit, and a synthesis unit. The input unit sequentially inputs images. The selection unit selects a reference image from input images including one or a plurality of images input by the input unit before the target image that is a processing target image newly input by the input unit. The matching unit calculates a correspondence relationship between the feature points of the reference image and the feature points of the target image. The estimation unit assumes that the movement between the reference image and the target image is only the rotational movement of the image sensor, and uses the positional information of the feature point pair whose correspondence is calculated by the matching unit, A conversion formula that matches the coordinate system of the target image is estimated. The synthesizing unit generates a synthesized image by connecting the reference image and the target image based on the conversion formula.

  According to this image processing program, the same effects as those of the image processing apparatus according to one aspect of the present invention described above can be obtained.

  A recording medium according to another aspect of the present invention is a computer-readable recording of an image processing program that causes a computer to sequentially input images picked up by an image pickup device and connect them each time it is input to generate a composite image. It is a possible recording medium. The program causes the computer to function as an input unit, a selection unit, a matching unit, an estimation unit, and a synthesis unit. The input unit sequentially inputs images. The selection unit selects a reference image from input images including one or a plurality of images input by the input unit before the target image that is a processing target image newly input by the input unit. The matching unit calculates a correspondence relationship between the feature points of the reference image and the feature points of the target image. The estimation unit assumes that the movement between the reference image and the target image is only the rotational movement of the image sensor, and uses the positional information of the feature point pair whose correspondence is calculated by the matching unit, A conversion formula that matches the coordinate system of the target image is estimated. The combining unit combines the input image and the target image based on the conversion formula to generate a combined image.

  According to this recording medium, the same effects as those of the image processing apparatus according to one aspect of the present invention described above can be obtained.

  According to various aspects and embodiments of the present invention, when images that have been input are sequentially stitched together, even if images with different imaging orientations are included, error accumulation is suppressed, and a high-quality panorama is achieved. An image processing apparatus, an image processing method, an image processing program, and a recording medium that can obtain an image are provided.

It is a functional block diagram of the portable terminal carrying the image processing apparatus which concerns on embodiment. It is a hardware block diagram of the portable terminal by which the image processing apparatus which concerns on embodiment is mounted. It is a schematic diagram explaining a wide-angle panoramic composite image. It is a schematic diagram explaining the deformation | transformation at the time of the synthesis | combination of a target image. It is a schematic diagram explaining alignment of a reference image and a target image. It is a schematic diagram explaining the deformation | transformation at the time of the synthesis | combination of a target image based on camera rotation. It is a schematic diagram explaining the corresponding point of a reference image and a target image. It is a schematic diagram explaining the detail of alignment of a reference image and a target image. It is a schematic diagram explaining alignment of a some image. It is a schematic diagram explaining the selection conditions of a reference image. It is a schematic diagram explaining the case where the motion of several images is estimated simultaneously. It is a schematic diagram explaining alignment of reference images. It is a schematic diagram explaining a guidance display. It is a schematic diagram explaining a synthetic | combination process. 6 is a flowchart for explaining the operation of the image processing apparatus according to the embodiment. 6 is a flowchart for explaining the operation of the image processing apparatus according to the embodiment.

  Hereinafter, embodiments of the present invention will be described with reference to the accompanying drawings. In addition, in each figure, the same code | symbol is attached | subjected to the same or an equivalent part, and the overlapping description is abbreviate | omitted.

  The image processing apparatus according to the present embodiment is an apparatus that sequentially creates a single image by joining input images every time it is input. For example, a plurality of continuously captured images are joined in real time. This is suitably employed when generating a panoramic image having a wider angle than the captured image. The image processing apparatus according to the present embodiment is preferably mounted on a mobile terminal with limited resources such as a mobile phone, a digital camera, and a PDA (Personal Digital Assistant), but is not limited thereto. For example, it may be mounted on a normal computer system. In the following, an image processing apparatus mounted on a portable terminal having a camera function will be described as an example of the image processing apparatus according to the present invention in consideration of ease of understanding.

  FIG. 1 is a functional block diagram of a mobile terminal 2 including an image processing apparatus 1 according to the present embodiment. A mobile terminal 2 shown in FIG. 1 is a mobile terminal carried by a user, for example, and has a hardware configuration shown in FIG. FIG. 2 is a hardware configuration diagram of the mobile terminal 2. As shown in FIG. 2, the portable terminal 2 physically includes a main storage device such as a CPU (Central Processing Unit) 100, a ROM (Read Only Memory) 101, and a RAM (Random Access Memory) 102, a camera, a keyboard, and the like. The input device 103, the output device 104 such as a display, the auxiliary storage device 105 such as a hard disk, and the like are configured as a normal computer system. Each function of the portable terminal 2 and the image processing apparatus 1 to be described later causes the input device 103 and the output device 104 to be controlled under the control of the CPU 100 by reading predetermined computer software on hardware such as the CPU 100, the ROM 101, and the RAM 102. This is realized by operating and reading and writing data in the main storage device and the auxiliary storage device 105. Although the above description has been given as a hardware configuration of the mobile terminal 2, the image processing apparatus 1 normally includes a CPU 100, a main storage device such as the ROM 101 and the RAM 102, an input device 103, an output device 104, an auxiliary storage device 105, and the like. It may be configured as a computer system. The mobile terminal 2 may include a communication module or the like.

  As shown in FIG. 1, the mobile terminal 2 includes a camera 20, an image processing device 1, and a display unit 21. The camera 20 has a function of capturing an image. For example, an image sensor or the like is used as the camera 20. The camera 20 has a continuous imaging function that repeatedly captures images at a predetermined interval from a timing specified by a user operation or the like, for example. For example, the user can take a continuous image that overlaps at least in the vertical and horizontal directions by sliding the camera 20 or rotating the camera 20 around a predetermined position. For example, the camera 20 has a function of outputting a captured image to the image processing apparatus 1 every time it is captured. The display unit 21 is a display device that can display a composite image and the like.

  The image processing apparatus 1 has a function of sequentially joining captured images to generate a wide-angle panorama composite image. The viewing angle of a normal camera is about 50 to 65 degrees (diagonal angle of view). However, the image processing apparatus 1 has a function to be described later so that input images can be joined to form an image with an angle of view of 65 degrees or more. Generate. For example, as shown in FIG. 3, when the imaging direction of the camera 20 changes as indicated by the arrow K, the sequentially input images are connected and the combined images are sequentially drawn on the combining plane Sp. For example, if the image input this time is Ic, the image Ic is combined with the previous combined image It to generate a single combined image. Note that the image processing apparatus 1 does not simply connect the target image Ic, which is the processing target to be combined, to the composite image It, but also connects it after performing deformation processing. For example, as illustrated in FIG. 4, the image processing apparatus 1 (A) enlarges / reduces the target image Ic, (B) a parallelogram (horizontal direction), (C) a parallelogram (vertical direction), and (D). Rotation, (E) Parallel movement (horizontal direction), (F) Parallel movement (vertical direction), (G) Trapezoid (horizontal direction), (H) Trapezoid (vertical direction). The target image Ic that has undergone these deformation processes or a combination of the deformation processes is drawn on the composite plane Sp. By adopting the eight-degree-of-freedom deformations (A) to (H), it becomes difficult for displacement to occur at the joints of images, and a natural wide-angle panorama can be obtained.

  Below, the detail of the image processing apparatus 1 is demonstrated. The image processing apparatus 1 includes an input unit 10, a selection unit 11, a matching unit 12, an estimation unit 13, a synthesis unit 14, and a guide unit 15.

  The input unit 10 has a function of inputting an image captured by the camera 20. The input unit 10 has a function of inputting, for example, an image captured by the camera 20 every time it is captured. Further, the input unit 10 has a function of saving the first input image in a first temporary storage area (output image buffer) provided in the mobile terminal 2. Further, the input unit 10 has a function of saving images input continuously from the next time in a second temporary storage area (input image buffer) provided in the mobile terminal. When the image is stored in the second temporary storage area, positioning of the image is performed and whether or not the image is a drawing target is determined. When it is determined that the image is a drawing target, The output image stored in the first temporary storage area is updated by synthesis and overwritten. In the following description, the image stored in the first temporary storage area is described as a composite image It, and the image stored in the second temporary storage area is described as a target image Ic (input image).

  The selection unit 11 has a function of selecting a reference image for alignment. The reference image is an image serving as a reference for alignment of the target image Ic. For example, the selection unit 11 is configured to be able to refer to a memory 18 that stores information related to an input image. The input image is an image input by the input unit 10 before the target image Ic, and may be one or a plurality of images. That is, if the n-th target image Ic is Ic (n−1), the input images are Ic (n−2), Ic (n−3),. When there is one input image, the selection unit 11 selects the image Ic1 as the reference image Ir0 of the target image Ic2. Thereafter, unless the predetermined condition is satisfied, the selection unit 11 does not change the reference image. The predetermined condition is when the distance between the reference image and the target image Ic (n) is a predetermined value or more. In this case, the selection unit 11 selects the target image Ic (n) as the reference image Ir (k) of the next new target image Ic (n + 1), and stores information on the target image Ic (n) in the memory 18. save. The information related to the target image Ic (n) may be, for example, only the pixel values and position information of feature points derived by the matching unit 12 described later. As described above, by limiting the information to be recorded in the memory 18, it is possible to reduce the used memory capacity as compared with the case of storing the reference image Ir itself. When the target image Ic (n + 1) is input, the selection unit 11 refers to the memory 18 and selects the target image Ic (n) as the reference image Ir (k). As described above, the selection unit 11 selects one reference image Ir for each target image Ic. In one embodiment, the selection unit 11 may also select a temporary reference image for the target image Ic (n) when a predetermined condition is satisfied. The temporary reference image is an image selected from input images and is a temporary reference image. Details of the selection process of the temporary reference image will be described later. The selection unit 11 outputs image information related to the reference image Ir (information including at least pixel information and position information of feature points) to the matching unit 12.

  The matching unit 12 acquires a correspondence relationship between the reference image Ir and the target image Ic. The matching unit 12 acquires information on the feature points of the reference image Ir and the feature points of the target image Ic. For example, the matching unit 12 acquires the correspondence between the reference image Ir and the target image Ic based on the pixel value of the feature point. As a matching method, a conventional method such as a block matching method can be used. In one embodiment, the matching unit 12 may perform matching after making the reference image Ir and the target image Ic multi-resolution. For example, the matching unit 12 changes the resolution of the reference image Ir and the target image Ic in stages, and generates a plurality of images having different resolutions. And the matching part 12 may acquire the amount of parallel movement of a feature point between images with the smallest resolution, and may perform the matching of the feature point in pixels between images with a higher resolution. In this case, the processing speed can be increased and the calculation cost can be reduced.

  The matching unit 12 acquires position information (coordinate information) of a feature point pair for which the correspondence relationship between the reference image Ir and the target image Ic is calculated. That is, the matching unit 12 acquires a pair of position information of a feature point of the reference image Ir and position information of a feature point of the target image Ic corresponding to the feature point. The matching unit 12 acquires a plurality of pairs of feature points for alignment processing described later. The matching unit 12 outputs the acquired feature point pair to the estimation unit 13. As described above, when the selection unit 11 adds the target image Ic as the reference image Ir for the next and subsequent times, the matching unit 12 sends the pixel values and position information of the feature points of the target image Ic to the selection unit 11. Output.

  The estimation unit 13 has a function of aligning the reference image Ir and the target image Ic based on the correspondence relationship between the reference image Ir and the target image Ic. FIG. 5 is a schematic diagram for explaining an outline of alignment between the reference image and the target image Ic. For example, as shown in FIG. 5A, when only the first image is input, the image is selected as the reference image Ir0. When the second image (target image Ic) is input, the estimation unit 13 aligns the target image Ic with the position of the reference image Ir0 as a reference. In the alignment, for example, as shown in FIG. 5B, a relative position between a predetermined point (here, center C0) of the reference image Ir0 and a predetermined point (here, center C1) of the target image Ic is determined. That is. The estimation unit 13 searches for a position where the feature point pairs acquired by the matching unit 12 overlap each other most. Then, as shown in FIG. 5C, when the estimation unit 13 completes the alignment of the reference image Ir0 and the target image Ic, the fact that the positional relationship is linked to each other (link Re1) is recorded. To do. As described above, when the alignment is completed by the estimation unit 13 and the relative distance between the reference image Ir0 and the target image Ic is equal to or greater than a predetermined value, the selection unit 11 sets the target image Ic to the next and subsequent times. Since it is necessary to add as the reference image Ir1, the matching unit 12 outputs the pixel value and position information of the feature point of the target image Ic to the selection unit 11.

  The estimation unit 13 aligns the reference image Ir and the target image Ic in consideration of the camera movement. FIG. 6 is a schematic diagram for explaining an imaging surface by the rotation of the camera 20. For example, as shown in FIG. 6, if the imaging surface of the camera 20 before rotation is S0 and the imaging surface of the rotated camera 20 is S1, the imaging surface S0 and the imaging surface S1 do not exist in the same plane. Therefore, the position at which the feature point pairs are overlapped by translation is different from the original overlap position. That is, when positioning is performed, the position of the feature point of the reference image Ir and the position of the feature point of the target image Ic are aligned on the same three-dimensional coordinate system in consideration of the movement of the camera. It is necessary to match.

Therefore, the estimation unit 13 estimates a conversion formula that matches the three-dimensional coordinate system of the reference image Ir0 with the three-dimensional coordinate system of the target image Ic. As shown in FIG. 7, the coordinates of the feature points of the reference image Ir0 are (x ₀ , y ₀ , 1), and the coordinates of the feature points of the target image Ic corresponding to the feature points are (x ₁ , y ₁ , 1). Then, the estimation unit 13 estimates a conversion formula in which (x ₀ , y ₀ , 1) matches (x ₁ , y ₁ , 1). Here, the degree of freedom of the camera is 1 degree of freedom for the focal length, 3 degrees of freedom for camera movement (parallel movement in the xy direction, enlargement / reduction by movement in the z direction), and rotation of the camera (x direction, y direction). The distortion of the image (trapezoid) and the rotation of the image in the z-axis are 3 degrees of freedom and 7 degrees of freedom, and considering the rolling shutter distortion (focal plane distortion) to be 8 degrees of freedom, It is expressed by the following formula 1.

The parameters a _{1 to} h ₁ of the conversion matrix (conversion formula) are parameters related to the eight degrees of freedom described above. The estimation unit 13 obtains a parameter of a transformation matrix that allows a plurality of feature point pairs to satisfy the above relationship by convergence calculation using an optimization method. Specifically, the difference between the position (x ₀ , y ₀ , 1) of the feature point of the reference image Ir and the position after converting the position (x ₁ , y ₁ , 1) of the feature point of the target image Ic. Convergence calculation is performed so that the objective function including is the minimum value. As the optimization method, a known method such as Newton method or Gauss Newton method is adopted.

Here, assuming that the movement between the reference image Ir and the target image Ic is only the rotational movement of the camera 20, the estimation unit 13 limits the movement of the camera 20 to three degrees of freedom, and position information of feature point pairs. Has the function of estimating the conversion equation. For example, as shown in FIG. 7, assuming that the imaging position of the camera 20 is the origin, the degree of freedom of the camera is limited to only rotation of the X axis, Y axis, and Z axis. If the parameters of each axis are (β, α, γ), the transformation matrix R can be expressed as follows.

That is, the position of the feature point (x ₀ , y ₀ , 1) of the reference image Ir and the position of the feature point (x ₁ , y ₁ , 1) of the target image Ic are expressed by the following formula 2 using the transformation matrix R: It can be made to correspond as follows.

Here, (c _x , c _y ) is the respective center coordinates when the reference image Ir and the target image Ic have the same image size. F is a focal length. The focal length F may be a value obtained from the specification information of the camera 20.

  When the convergence calculation is performed using the transformation matrix in which the reduction component is considered as in Equation 1 and priority is given to further reducing the error, the error is relatively smaller when the image is reduced. Therefore, a transformation matrix having a large degree of reduction component tends to be a solution. In this case, errors are accumulated every time the images are sequentially combined, and as a result, a high-quality panoramic image is not obtained. On the other hand, as shown in Formula 2, by limiting the transformation matrix R to only the rotation component, the reduction component is not taken into account in the convergence calculation by the optimization method, so that accumulation of errors is avoided. And a high-quality panoramic composite image is generated. Note that the transformation matrix of the previous target image Ic may be adopted as the initial value of the convergence calculation of an optimization method such as the Gauss-Newton method. In this case, since the convergence calculation can be converged more quickly, the calculation speed can be improved.

In one embodiment, the estimation unit 13 projects the two-dimensional coordinates of the feature points on the spherical surface of the three-dimensional space Sp when the transformation matrix is estimated by performing the convergence calculation on Equation 2, and the correspondence relationship of the projected coordinates is expressed. To estimate the transformation matrix. FIG. 8 is a schematic diagram illustrating details of the alignment between the reference image Ir and the target image Ic. As shown in FIGS. 8A and 8B, the estimation unit 13 determines the position (x ₀ , y ₀ , 1) of the feature point of the reference image Ir in the two-dimensional coordinate system and the target image in the two-dimensional coordinate system. The position (x ₁ , y ₁ , 1) of Ic is perspectively projected onto the spherical surface of the three-dimensional space Sp. x _n -c _x the vector _{x n,} the y n -c _x and a vector _{y n,} the coordinates of the projected _{(X n, Y n, Z} n) When, for the coordinates _{(x 1, y 1, F} ) Is projected and projected, for example, according to Equation 3 below.

Further, the coordinate points after conversion by the conversion matrix R can be expressed as follows.

For this reason, the objective function of the convergence calculation includes the following difference r.

Note that the conversion of Expression 3 can be omitted when it is assumed that the target image Ic is close to the reference image Ir. In this case, since the difference r does not include division, it is not necessary to consider the carry-over due to division. For this reason, for example, when the convergence calculation of the objective function is performed by an optimization process using a Gauss-Newton method or the like, the calculation can be facilitated. Therefore, when calculating by projecting onto a spherical surface in a three-dimensional space, the calculation cost can be reduced.

  The estimation unit 13 estimates the transformation matrix R by the above processing, and aligns the reference image Ir and the target image Ic. The estimation unit 13 sequentially aligns the reference image Ir selected by the selection unit 11 and the input target image Ic to generate a link ((C) in FIG. 8). FIG. 9 shows a link in which positioning is performed between eight input images. As shown in FIG. 9, the centers C0 to C7 of the eight input images are linked (links Re1 to Re7). By repeating the operation in which the selection unit 11 selects the reference image and the estimation unit 13 forms a link, the images can be connected while being aligned as shown in FIG.

  Note that the estimation unit 13 estimates the transformation matrix R by projecting onto a spherical surface, and aligns the reference image and the target image. Therefore, when performing coordinate transformation between the two-dimensional plane and the spherical surface, for example, 8 shown in FIG. A degree of freedom of image deformation is considered. In other words, the estimating unit 13 can perform image deformation shown in FIG. 4 when the combining unit 14 described later projects from a spherical surface to a plane by positioning on the spherical surface.

  Below, the modification of the selection part 11 and the estimation part 13 is demonstrated. In one embodiment, the selection unit 11 may use not only the reference image Ir but also the temporary reference image Itr. FIG. 10 is a schematic diagram illustrating the reference image Ir and the temporary reference image Itr. First, as shown in FIG. 10A, when an image is input, the image becomes a reference image Ir for the next and subsequent times. Next, it is assumed that a target image Ic separated from the reference image Ir by a predetermined value or more is input as shown in FIG. In this case, as shown in FIG. 10C, the target image Ic is set as a temporary reference image Itr that is a temporary reference image after the next time. The temporary reference image Itr is a temporary reference image that is not saved as a history. Next, a target image Ic separated from the temporary reference image Itr is input as shown in (D) of FIG. 10, and the target image Ic separated from the temporary reference image Itr by a predetermined value or more as shown in (E) of FIG. Is entered. In this case, as shown in FIGS. 10F and 10G, the temporary reference image Itr thus far is discarded, and the target image Ic is set as a temporary reference image Itr that is a temporary reference image from the next time onward. Next, as shown in (H) of FIG. 10, it is assumed that the target image Ic separated from the reference image Ir as well as the temporary reference image Itr by a predetermined value or more is input. In this case, as shown in (I) of FIG. 10, the temporary reference image Itr up to this time is discarded, and the target image Ic is set as a reference image Ir for the next time and thereafter. Here, the reference image up to this time, that is, the first reference image is Ir0, and the next and subsequent reference images are Ir1. Information about the feature points of the reference images Ir0 and Ir1 is stored for alignment. Next, as shown in (J) of FIG. 10, it is assumed that the target image Ic separated from the reference image Ir1 by a certain value or more is input. In this case, as shown in FIG. 10C, the target image Ic is set as a temporary reference image Itr that is a temporary reference image after the next time, as shown in FIG. Next, it is assumed that the target image Ic closer to the reference image Ir0 than the temporary reference image Itr is input as shown in (L) of FIG. In this case, the temporary reference image Itr up to this time is discarded, and the next and subsequent reference images are set as the reference image Ir0. In this way, by holding the reference image information, the target image can be aligned based on the past reference image even when the camera 20 returns to the original position. Further, by using the temporary reference image Itr and the reference image Ir, it is possible to minimize the data that needs to be recorded.

Moreover, in one embodiment, the estimation part 13 may estimate the motion between several images simultaneously, when several images have overlapped. For example, as shown in FIG. 11, it is assumed that there is an image (past target image Ip1) that overlaps the reference image Ir and the target image Ic. Reference image Ir and past state of being positioned in the target image Ip1, i.e., the conversion matrix R ₁ have already been derived. Then, the coordinates of the feature points of the reference image Ir are (x ₀ , y ₀ , 1), the coordinates of the feature points of the past target image Ip1 are (x ₁ , y ₁ , 1), and the coordinates of the feature points of the target image Ic. Is (x ₂ , y ₂ , 1). Here, the coordinate system of the coordinate system and target image Ic of the reference image Ir in deriving the transformation matrix R ₂ to correspond, set the following conditions.

Expression 7 associates conversion expression R ₁ and conversion expression R ₂ . The estimation unit 13 simultaneously estimates R ₁ and R ₂ that can satisfy the above mathematical expressions 5 to 7 by convergence calculation using an optimization method. In this case, it is possible to avoid wasting information on the feature point pairs of the reference image Ir and the past target image Ip1. Further, by simultaneously estimating between a plurality of images, it is possible to suppress accumulation of errors compared to a case where links are connected in series.

  Further, in one embodiment, when the target image Ic is close to the past reference image Ir, the estimation unit 13 performs alignment with the past reference image Ir as well as the current reference image Ir. For example, as shown in FIG. 13A, the target image Ic13 whose image center is C13 has a link Re13 and a reference image Ir12 whose image center is C12, and the relative position is determined. Here, as shown in FIG. 13B, the estimation unit 13 aligns the reference image Ir1 and the target image Ic13 when the target image Ic13 is close to the past reference image Ir1 at the image center C1. Do. Thereby, the link Re14 is extended between the reference image Ir1 and the target image Ic13. That is, the reference image Ir1 and the reference image Ir12 can be aligned using the target image Ic13. As described above, by achieving the alignment between the reference images Ir using the target image Ic, it is possible to position the reference images Ir that originally have little overlap.

  Moreover, in one embodiment, the estimation part 13 may have the function to adjust the whole position. The adjustment of the overall position is to adjust the overall positional relationship of the drawing target image (the image written in the output image buffer). For example, the entire drawing target image is updated at a timing when a new link is established between the reference images Ir or when a plurality of past transformation matrices are updated by executing simultaneous estimation of movements of a plurality of images. The position is finely adjusted. That is, the conversion matrix R of all drawing target images is recalculated. The overall position does not use the feature points output by the matching unit 12, but randomly extracts corresponding points between images based on the result of the alignment or extracts from a predetermined position. The entire alignment is performed based on the position information. In this case, since it is not necessary to hold a pair of past feature points, the memory usage can be reduced.

  Next, the guide unit 15 will be described. The guide unit 15 has a function of guiding user operations. The guide unit 15 is connected to a display unit 21 that displays an image, and causes the display unit 21 to display a guide display that guides the user's camera operation. For example, the guide unit 15 has a first hop count that is greater than or equal to a predetermined value and does not overlap with the current reference image Ir in the pair of the reference image Ir and the target image Ic whose relative positions have been determined. If an image exists, a guidance display for guiding the imaging position from the current imaging position to the image position of the first image is displayed on the display unit. For example, as illustrated in FIG. 13A, the guide unit 15 determines the number of hops (the number of links Re) between the image at the image center C0 (first image) and the current reference image at the image center C8. Count. Then, the guide unit 15 calculates the distance between the image at the image center C0 and the current reference image at the image center C8. Then, the guide unit 15 determines that the images are long in series when the count number is equal to or greater than the predetermined value and the distance is smaller than the predetermined value (for example, when they do not overlap). If images are long and connected in series, errors are likely to accumulate. Therefore, as shown in FIG. 13B, the guide unit 15 displays the guidance display Ga so that the imaging position of the camera 20 is directed toward the image (first image) at the image center C0. As the guidance display Ga, a frame, an arrow, an icon, or the like is used, and a voice may be given. When the user is guided by the guidance display Ga and changes the imaging direction of the camera 20, a link is formed between the image at the image center C0 and the image at the image center C8, as shown in FIG. The position can be adjusted. Thus, the guide unit 15 avoids accumulation of errors by guiding the user.

  Next, the synthesis unit 14 will be described. The composition unit 14 is connected to the display unit 21 and has a function of drawing a composite image on the display unit 21. The synthesizing unit 14 projects an image group (image to be drawn) aligned on the spherical surface in the three-dimensional space by the transformation matrix estimated by the estimating unit 13 onto a two-dimensional plane. At the time of projection from a spherical surface to a plane, for example, image deformation shown in FIG. 4 is performed. Then, the drawing target image is recorded in the output image buffer as a single composite image. For example, as shown in FIG. 14, it is assumed that the composite plane Sp is divided into a lattice pattern. The synthesizing unit 14 draws only the cells whose four corners are included in the image Id projected onto the synthesis plane Sp. The synthesizing unit 14 adjusts the blend ratio and the like at the boundary between images. In this way, a plurality of drawing target images Id are projected onto the synthesis plane Sp to generate a synthesized image.

  Next, the operation of the image processing apparatus 1 according to this embodiment will be described. 15 and 16 are flowcharts showing the operation of the image processing apparatus 1 according to this embodiment. The control processing shown in FIGS. 15 and 16 is executed, for example, at the timing when the imaging function of the mobile terminal 2 is turned on, and is repeatedly executed at a predetermined cycle. Note that the target image Ic is assumed to be the second and subsequent input images in consideration of ease of understanding.

  As shown in FIG. 15, first, the image processing apparatus 1 executes an image input process (S10: input step). In the process of S <b> 10, the input unit 10 inputs the target image Ic from the camera 20. When the process of S10 ends, the process proceeds to the alignment process (S12).

  In the process of S12, the selection unit 11, the matching unit 12, and the estimation unit 13 perform relative alignment between the reference image Ir and the target image Ic. Details of this processing are shown in FIG. First, the selection unit 11 selects the reference image Ir from the input image (S30: selection step). Next, the matching unit 12 and the estimation unit 13 align the reference image Ir and the target image Ic (S32: matching step and estimation step). Next, the estimation unit 13 determines whether or not the past (or other) reference image Ir and the target image Ic can be compared (S34). In the process of S34, when the estimation unit 13 determines that the comparison is possible, the past reference image Ir and the target image Ic are aligned (S36, for example, FIG. 12B). Then, the estimation unit 13 sets a redraw flag to 1 for adjusting the overall position (S38). Thereafter, the selection unit 11 determines whether or not the distance between the target image Ic and the past reference image Ir is a predetermined value or more (S40). In the process of S40, when the selection unit 11 determines that the distance is greater than or equal to the predetermined value, the target image Ic is recorded in the reference image list to be the next reference image Ir (S42). The reference image list is a list for referring to data in which pixel values and coordinates of feature points of the reference image are recorded. When the process of S42 ends, the alignment process shown in FIG. 16 ends.

  On the other hand, when the estimation unit 13 determines in the process of S34 that the comparison is not possible, the process proceeds to a process of determining the distance between the target image Ic and the past reference image Ir (S40). On the other hand, in the process of S40, when the selection unit 11 determines that the distance is not greater than or equal to the predetermined value, the alignment process illustrated in FIG.

  Returning to FIG. 15, when the process of S12 is completed, the process proceeds to a determination process (S14).

  In the process of S14, the synthesizing unit 14 determines whether or not to add the target image Ic input in the process of S10 as a drawing image. For example, the synthesizing unit 14 is configured to be able to refer to a drawing image list that can refer to image information of an image to be drawn, and the distance from the closest image among the images described in the list Is equal to or greater than the predetermined value, the target image Ic input in the process of S10 is added to the drawing image. If it is determined in the process of S14 that it is added, the process proceeds to a storage process (S16).

  In the process of S16, the synthesizing unit 14 adds the target image Ic to the drawing list and stores the target image Ic. When the process of S16 ends, the process proceeds to a redraw determination process (S18).

  In the process of S18, the estimation unit 13 determines whether or not the redraw flag = 0. In the process of S18, when the redrawing flag = 1, the process proceeds to the drawing position recalculation process (S20).

  In the process of S20, the estimation unit 13 performs an overall alignment process. The estimation unit 13 adjusts the positions of all the drawing images using the updated conversion matrix. When the process of S20 ends, the process proceeds to a preview image drawing process (S22).

  In the process of S22, the synthesizing unit 14 specifies an image to be drawn from, for example, the drawing image list, and generates a preview synthesized image by projecting from the spherical surface of the three-dimensional space onto the two-dimensional plane (S22: synthesizing step). ). Thereafter, a preview image is output and displayed on the display unit 21 or the like (S24). When the processing of S24 is completed, the routine proceeds to image input determination processing (S26).

  On the other hand, in the process of S14, when the synthesizing unit 14 does not determine addition, the process proceeds to a redraw determination process (S18). In the process of S18, when the redrawing flag = 0, the process proceeds to the preview image drawing process (S22).

  In the process of S26, the input unit 10 determines whether or not the input of the target image Ic has been completed. If the input of the target image Ic is not completed in the process of S26, the process proceeds to S10 again. On the other hand, when the input of the target image Ic is completed, the process proceeds to a result image output process (S28).

  In the process of S28, the synthesis unit 14 displays the synthesized image on the display unit 21 or the like. Thus, the control process shown in FIGS. 15 and 16 is completed. By executing the control processing shown in FIGS. 15 and 16, it is possible to avoid accumulation of errors and obtain a high-quality panoramic composite image.

  Next, an image processing program for causing the portable terminal (computer) 2 to function as the image processing apparatus 1 will be described.

  The image processing program includes a main module, an input module, and an arithmetic processing module. The main module is a part that comprehensively controls image processing. The input module operates the mobile terminal 2 so as to acquire an input image. The arithmetic processing module includes a selection module, a matching module, an estimation module, a synthesis module, and a guidance module. Functions realized by executing the main module, the input module, and the arithmetic processing module are the input unit 10, the selection unit 11, the matching unit 12, the estimation unit 13, the synthesis unit 14, and the guide unit 15 of the image processing apparatus 1 described above. These functions are the same.

  The image processing program is provided by a recording medium such as a ROM or a semiconductor memory, for example. The image processing program may be provided as a data signal via a network.

  As described above, according to the image processing device 1, the image processing method, and the image processing program according to the present embodiment, it is assumed that the movement between the reference image Ir and the target image Ic is caused only by the rotational movement of the camera 20. A transformation matrix R to which the system is associated is estimated. For this reason, the transformation matrix R does not include parameters such as enlargement, reduction, and parallel movement, so that it is possible to avoid the occurrence of an error due to the input target image Ic being reduced. Furthermore, by limiting to only the rotation component, it is possible to avoid the target image Ic that has been reduced or the like as the reference image Ir for the next and subsequent times, so that accumulation of errors can be avoided. Therefore, even when the input images are sequentially stitched together, even if images with different imaging orientations are included, accumulation of errors can be suppressed and a high-quality panoramic image can be obtained.

  The embodiment described above shows an example of the image processing apparatus according to the present invention. The image processing apparatus according to the present invention is not limited to the image processing apparatus 1 according to the embodiment, and the image processing apparatus according to the embodiment may be modified or otherwise changed without changing the gist described in each claim. It may be applied to the above.

  For example, in the above-described embodiment, an example in which the camera 20 continuously captures still images has been described, but the camera 20 may capture a moving image. In this case, the input unit 10 may have a function of extracting continuous images from the captured moving image. The image input by the input unit 10 may be an image transmitted from another device via a network.

  In the above-described embodiment, the size of the image captured by the camera 20 is described as being the same. However, the size of the captured image may be different for each imaging.

  In the above-described embodiment, the case where the input unit 10, the selection unit 11, the matching unit 12, the estimation unit 13, the synthesis unit 14, and the guide unit 15 are provided has been described. Also good. For example, the guide unit 15 may not be provided as necessary.

  Furthermore, in the above-described embodiment, the case where the image is deformed with 8 degrees of freedom shown in FIG. 5 has been described. However, the image is not limited to 8 degrees of freedom. For example, as shown in FIGS. It may be 6 degrees of freedom.

  DESCRIPTION OF SYMBOLS 1 ... Image processing apparatus, 10 ... Input part, 11 ... Selection part, 12 ... Matching part, 13 ... Estimation part, 14 ... Composition part, 15 ... Guide part, 20 ... Camera, 21 ... Display part.

Next, the guide unit 15 will be described. The guide unit 15 has a function of guiding user operations. The guide unit 15 is connected to a display unit 21 that displays an image, and causes the display unit 21 to display a guide display that guides the user's camera operation. For example, the guide unit 15 has a first hop count that is greater than or equal to a predetermined value and does not overlap with the current reference image Ir in the pair of the reference image Ir and the target image Ic whose relative positions have been determined. If an image exists, a guidance display for guiding the imaging position from the current imaging position to the image position of the first image is displayed on the display unit. For example, as illustrated in FIG. 13A, the guide unit 15 determines the number of hops (the number of links Re) between the image at the image center C0 (first image) and the current reference image at the image center C8. Count. Then, the guide unit 15 calculates the distance between the image at the image center C0 and the current reference image at the image center C8. Then, when the count number is equal to or greater than the predetermined value and the distance is smaller than the predetermined value ( and does not overlap), the guide unit 15 determines that the images are long and connected in series. If images are long and connected in series, errors are likely to accumulate. Therefore, as shown in FIG. 13B, the guide unit 15 displays the guidance display Ga so that the imaging position of the camera 20 is directed toward the image (first image) at the image center C0. As the guidance display Ga, a frame, an arrow, an icon, or the like is used, and a voice may be given. When the user is guided by the guidance display Ga and changes the imaging direction of the camera 20, a link is formed between the image at the image center C0 and the image at the image center C8, as shown in FIG. The position can be adjusted. Thus, the guide unit 15 avoids accumulation of errors by guiding the user.

Claims (13)

An image processing apparatus that sequentially inputs images picked up by an image pickup device and generates a composite image by stitching each time an input is performed,
An input unit for sequentially inputting the images;
A selection unit that selects a reference image from input images including one or a plurality of images input by the input unit before a target image that is a processing target image newly input by the input unit;
A matching unit that calculates a correspondence relationship between the feature points of the reference image and the feature points of the target image;
Assuming that the movement between the reference image and the target image is only the rotational movement of the imaging device, the position information of the reference image is calculated using the position information of the feature point pair for which the correspondence is calculated by the matching unit. An estimation unit that estimates a conversion formula that associates the coordinate system with the coordinate system of the target image;
Based on the conversion formula, a combining unit that connects the reference image and the target image to generate the combined image;
An image processing apparatus comprising:   The image processing apparatus according to claim 1, wherein the estimation unit estimates a conversion equation including only rotational components of each axis of a three-dimensional coordinate system with the position of the image sensor as an origin.   The estimation unit prepares an objective function including a difference in position information of each pair of feature points converted using the conversion formula, and uses an optimization method so that the objective function becomes a minimum value. The image processing apparatus according to claim 1, wherein the conversion formula is estimated by performing a convergence calculation.   When there are a plurality of images input by the input unit before the target image, the estimation unit sets an initial value of the convergence calculation as an initial value of the image input by the input unit. The image processing apparatus according to claim 3, wherein the conversion formula is adopted.   The estimation unit projects the feature point pair whose correspondence relationship has been calculated by the matching unit onto a spherical surface, and uses the positional information of the feature point pair after the projection, and the coordinate system of the reference image and the target The image processing apparatus according to claim 3, wherein a conversion formula that correlates with an image coordinate system is estimated.   The estimation unit includes a plurality of images input by the input unit before the target image, and at least one of the images input by the input unit before the target image is the reference In the case of overlapping the image and the target image, a pair of feature points of the reference image, the image overlapping the reference image, and the target image are used, and a coordinate system of the reference image and an image overlapping the reference image 6. The conversion formula for associating a coordinate system and the conversion formula for associating the coordinate system of the reference image with the coordinate system of the target image are estimated in association with each other. Image processing device.   When the distance between the reference image and the target image is equal to or greater than a predetermined value, the selection unit selects the target image that is a processing target image newly input from the next time on by the input unit. The image processing apparatus according to claim 1, which is selected as a reference image.   When the distance between the target image and the past reference image is smaller than the distance between the target image and the reference image, the selection unit newly inputs a past reference image after the next time. The image processing apparatus according to claim 7, wherein the image is selected as the reference image of a target image that is a target image. When the distance between the target image and the past reference image is a predetermined value or less, the matching unit further calculates a correspondence relationship between the feature point of the past reference image and the feature point of the target image,
The estimation unit uses a positional information of the pair of feature points of the past reference image and the target image, and uses a conversion formula that associates the coordinate system of the past reference image with the coordinate system of the target image. Further estimate,
Using the conversion equation that associates the coordinate system of the reference image with the coordinate system of the target image, and the conversion equation that associates the coordinate system of the past reference image with the coordinate system of the target image, The image processing apparatus according to claim 8, wherein the reference image is associated with each other. A guidance unit that is connected to a display unit that displays an image and that displays a guidance display for guiding a user's camera operation on the display unit;
The estimation unit links the reference image that has estimated the conversion formula and the target image, and records the relative position as a determined pair,
The combining unit outputs the combined image to the display unit,
In the case where there is a first image in which the number of hops with the current reference image is equal to or greater than a predetermined value and the current reference image does not overlap with the current reference image among the pair whose relative position has been determined, The image processing apparatus according to claim 1, wherein the guidance display that guides the imaging position from the imaging position to the image position of the first image is displayed on the display unit. An image processing method for sequentially inputting images picked up by an image pickup device and connecting them at each input to generate a composite image,
An input step for sequentially inputting the images;
A selection step of selecting a reference image from input images composed of one or a plurality of images input by the input step before a target image that is a processing target image newly input by the input step;
A matching step for calculating a correspondence relationship between the feature points of the reference image and the feature points of the target image;
Assuming that the movement between the reference image and the target image is only the rotational movement of the image sensor, the positional information of the reference image is obtained using the position information of the feature point pair whose correspondence is calculated by the matching step. An estimation step for estimating a conversion formula that associates the coordinate system with the coordinate system of the target image;
Based on the conversion formula, a combining step of connecting the reference image and the target image to generate the combined image;
An image processing method comprising: An image processing program for sequentially inputting images picked up by an image pickup device and causing a computer to function to generate a composite image by stitching each time an input is performed,
The computer,
An input unit for sequentially inputting the images;
A selection unit that selects a reference image from input images composed of one or a plurality of images input by the input unit before a target image that is a processing target image newly input by the input unit;
A matching unit that calculates the correspondence between the feature points of the reference image and the feature points of the target image;
Assuming that the movement between the reference image and the target image is only the rotational movement of the imaging device, the position information of the reference image is calculated using the position information of the feature point pair for which the correspondence is calculated by the matching unit. An estimation unit that estimates a conversion formula that associates the coordinate system with the coordinate system of the target image; and
An image processing program that functions as a combining unit that connects the reference image and the target image to generate the combined image based on the conversion formula. A computer-readable recording medium on which an image processing program for causing a computer to function so as to sequentially input images picked up by an image pickup device and connect them at each input to generate a composite image,
The computer,
An input unit for sequentially inputting the images;
A selection unit that selects a reference image from input images composed of one or a plurality of images input by the input unit before a target image that is a processing target image newly input by the input unit;
A matching unit that calculates the correspondence between the feature points of the reference image and the feature points of the target image;
Assuming that the movement between the reference image and the target image is only the rotational movement of the imaging device, the position information of the reference image is calculated using the position information of the feature point pair for which the correspondence is calculated by the matching unit. An estimation unit that estimates a conversion formula that associates the coordinate system with the coordinate system of the target image; and
A recording medium on which the image processing program for causing the reference image and the target image to be connected to generate a composite image based on the conversion formula is generated.