JP4952363B2 - Image processing apparatus, image conversion method, and program - Google Patents

Description

  The present invention relates to an image processing apparatus, an image conversion method, and a program for obtaining an appropriate projective transformation (homography) based on correspondence of a feature point group from a plane to a plane.

  For example, in an imaging apparatus such as a digital camera, a plurality of continuous images may be overlapped and processed for the purpose of panorama synthesis or camera shake correction. Here, a robust algorithm called RANSAC (random sampling consensus) is known as a method for determining correspondence between feature points between images. This is a calculation method for excluding outliers (outliers) when an error is included in the feature point correspondence, and specifically includes the following procedure.

  That is, first, a temporary transformation parameter (in this case, a projective transformation matrix) is obtained using a small subset of feature points corresponding to randomly extracted features, and the coordinates of the feature points transformed by the transformation parameters are the actual corresponding points. It is determined whether or not the coordinates match, and the number of matching points is counted.

  This random extraction is repeated many times by a loop, and the set of matching points when the number of matching points is the largest is taken as the final inlier, and the others are excluded as outliers. In this case, if accuracy is not required, the conversion parameter when the final inlier is obtained may be used as an answer. If high accuracy is required, the conversion parameter can be recalculated by using a larger nonlinear minimization algorithm or the like using the obtained final inlier set (see Non-Patent Document 1, for example).

  Now, a specific description will be given by taking as an example a case where the image A and the image B are aligned. The processing procedure is as follows.

  FIG. 13 is a flowchart showing a processing procedure for alignment between images according to the conventional method. Note that the processing shown in this flowchart is executed by a general electronic computer.

  First, a plurality of feature points are extracted from image A and image B (step S11), arbitrary four points are selected from these feature points (steps S12 and S13), and image A and image A are selected from the set of these four points. A transformation matrix (projection transformation determinant) for aligning the image B is obtained (step S14). At this point in time, it is a tentative transformation matrix because it is not known whether or not it still fits.

  Here, a transformation matrix is applied to the image A (step S15), and the distance between the coordinates obtained by projective transformation of each feature point on the image A and the corresponding coordinates of each feature point on the image B is calculated (step S15). S16). As a result, feature points whose distance between them is equal to or greater than a predetermined value are excluded as incompatible points, that is, outliers (step S17), and the sum of the remaining compatible points is recorded as the current score (step S18).

In this way, the same process is repeated a predetermined number of times while randomly selecting four points (step S19), the transformation matrix is calculated using the matching points of the best score, and the image is used to A and B are aligned (step S20).
"Robust detection of planar area by random sampling based on feature point distribution", IEICE Transactions D-II Vol. J88-D-II No.2 pp.313-324

  As described above, RANSAC repeats the process of randomly extracting a subset of feature points, obtaining a temporary transformation matrix, and counting the number of matching points. Usually, this repeated loop is several tens of times or more. Even if the simplest linear algorithm is used to obtain a temporary transformation matrix, the number of loops is so large that the amount of calculation becomes very large, processing takes time, or the hardware configuration becomes large. There's a problem.

  The present invention has been made in view of the above points, and image processing capable of obtaining a conversion formula for aligning each image or an inlier set corresponding to a feature point at high speed without requiring complicated calculation. An object is to provide an apparatus, an image conversion method, and a program.

An image processing apparatus according to claim 1 of the present invention includes an image acquisition unit that acquires at least two images, and a feature point selection unit that selects a grid-like feature point from the first image obtained by the image acquisition unit. If a tracking means for tracking the feature points selected by the feature point selecting means on the second image to select any four points from among the feature points tracked by this tracking means, the four points Originally, an evaluation point generating means for adding an evaluation point by repeating an operation for drawing a straight line and an operation for obtaining an intersection point on the second image, and each evaluation point generated by the evaluation point generating means, An evaluation unit that evaluates suitability with each feature point tracked on the second image, and a conversion formula that calculates a conversion formula between images using a set of points having the highest suitability by the evaluation unit. Provided with calculation means And features.

According to a second aspect of the present invention, in the image processing apparatus according to the first aspect, the evaluation point generating unit newly selects four points forming a quadrangle from the points added by the tracking unit, The same processing is repeated based on the points, and the evaluation points are recursively generated.

  According to a third aspect of the present invention, in the image processing apparatus according to the second aspect, the evaluation point generating means includes first additional processing means for obtaining an inner point of the rectangle from four points forming the vertex of the rectangle. A second additional processing means for obtaining a boundary point that is an intersection of the inner point obtained by the first additional processing means and the side of the square; and a boundary point obtained by the second additional processing means; A third additional processing means for obtaining an outer point that is an intersection of an extension line with the vertex of the quadrangle and an extension line of the side of the previous square, an inner point, a boundary point obtained by each of the additional processing means, An outside point is added as an evaluation point.

  According to a fourth aspect of the present invention, in the image processing apparatus according to the first aspect, the first image and the second image are aligned based on the conversion formula obtained by the conversion formula calculation means. And image processing means for generating a predetermined image.

The image conversion method according to claim 5 of the present invention is a first step of acquiring at least two images, and a second step of selecting grid-like feature points from the first image obtained by the first step. Step, a third step of tracking the feature point selected by the second step on the second image, and arbitrary four points among the feature points tracked by the third step. Based on these four points, the fourth step of adding an evaluation point by repeating the operation of drawing a straight line and the operation of obtaining an intersection point on the second image based on these four points, and generated by this fourth step A fifth step of evaluating the relevance of each evaluated point and each feature point tracked on the second image, and using the set of points having the highest relevance by the fifth step Calculate the conversion formula between images The program according to claim 6 of the present invention is a program recorded on a computer-readable recording medium, and acquires at least two images from the computer. A second function for selecting a grid-like feature point from the first image obtained by the first function, and a second feature point selected by the second function. A third function for tracking on the image, and an arbitrary four points selected from each of the feature points tracked by the third function, and an operation for drawing a straight line and an operation for obtaining an intersection point based on these four points. A fourth function for adding evaluation points by repeating on the second image, each evaluation point generated by the fourth function, and each feature point tracked on the second image To evaluate the compatibility with the fifth Function and, characterized in that to realize a sixth function that calculates a conversion formula between the fifth image using a set of points was higher most compatible with features.

  According to the present invention, an inlier set corresponding to a conversion formula or feature point for aligning each image can be obtained at high speed without requiring complicated calculation.

  Hereinafter, embodiments of the present invention will be described with reference to the drawings.

(First embodiment)
FIG. 1 is a diagram showing an external configuration when the image processing apparatus according to the first embodiment of the present invention is applied to a digital camera. FIG. 1 (a) is mainly a front configuration, and FIG. It is a perspective view which mainly shows the structure of a back surface.

  The digital camera 1 has a photographing lens 3, a self-timer lamp 4, an optical finder window 5, a strobe light emitting unit 6, a microphone unit 7 and the like on the front surface of a substantially rectangular thin plate-like body 2 on the upper surface (for the user). On the right end side, a power key 8 and a shutter key 9 are provided.

  The power key 8 is a key operated every time the power is turned on / off, and the shutter key 9 is a key for instructing a photographing timing at the time of photographing.

  Also, on the back of the digital camera 1, a shooting mode (R) key 10, a playback mode (P) key 11, an optical viewfinder 12, a speaker unit 13, a macro key 14, a strobe key 15, a menu (MENU) key 16, a ring A key 17, a set (SET) key 18, a display unit 19, and the like are provided.

  The shooting mode key 10 is operated automatically from the power-off state to automatically turn on the power and shift to the still image shooting mode. On the other hand, by repeatedly operating from the power-on state, the still image mode and the moving image mode are switched. Set cyclically. The still image mode is a mode for photographing a still image. The moving image mode is a mode for shooting a moving image.

  The shutter key 9 is commonly used for these photographing modes. That is, in the still image mode, a still image is taken at the timing when the shutter key 9 is pressed. In the moving image mode, shooting of a moving image is started at a timing when the shutter key 9 is pressed, and shooting of the moving image is ended when the shutter key 9 is pressed again.

  When the playback mode key 11 is operated from the power-off state, the playback mode key 11 is automatically turned on to enter the playback mode.

  The macro key 14 is operated when switching between normal shooting and macro shooting in the still image shooting mode. The strobe key 15 is operated when switching the light emission mode of the strobe light emitting unit 6. The menu key 16 is operated when selecting various menu items including the continuous shooting mode. The ring key 17 is integrally formed with item selection keys in the up, down, left, and right directions, and the set key 18 located in the center of the ring key 17 sets the item selected at that time. To operate.

  The display unit 19 is composed of a color liquid crystal panel with a backlight, and displays a through image on the monitor as an electronic viewfinder in the photographing mode, and reproduces and displays the selected image and the like in the reproduction mode.

  Although not shown, the digital camera 1 has a memory card slot for attaching / detaching a memory card used as a recording medium, a serial interface connector for connecting to an external personal computer, etc., for example, USB (Universal). Serial Bus) connector and the like are provided.

  FIG. 2 is a block diagram showing an electronic circuit configuration of the digital camera 1.

  The digital camera 1 includes an optical lens device 21, an image sensor 22, a memory 23, a display device 24, an image processing device 25, an operation unit 26, a computer interface unit 27, an external storage IO device 28, a program code storage device 29, and a CPU 30. A memory card 31 is provided.

  The optical lens device 21 includes a lens optical system including a focus lens and a zoom lens (not shown) constituting the photographing lens 3 and a driving unit thereof, and collects light from the photographing target on the image sensor 22. Light to form an image.

  The image sensor 22 is for capturing a formed image as digitized image data, and is configured by, for example, a CCD (Charge Coupled Device). If the image sensor 22 is controlled by the CPU 30 and the shutter key 9 is not pressed, digital image data having a low preview resolution is generated, and this image data is periodically stored in the memory 23 at intervals of about 30 sheets per second. To send. Further, when the shutter key 9 is pressed, the image sensor 22 generates image data with high resolution and sends the generated image data to the memory 23. Further, the image sensor 22 can set imaging sensitivity (ISO sensitivity) by the CPU 30.

  The memory 23 temporarily stores a low-resolution preview image from the image sensor 22, high-resolution image data, original image data processed by the image processing device 25, and processed image data. The memory 23 sends the temporarily stored image data to the display device 24 or the image processing device 25.

  The display device 24 is for displaying an image on the display unit 19 which is a liquid crystal monitor. The display device 24 displays a low-resolution preview image or a high-resolution image temporarily stored in the memory 23 on the display unit 19.

  The image processing device 25 is for performing image processing such as image data compression on the image data temporarily stored in the memory 23.

  In addition to the shutter key 9, the operation unit 26 includes a power key 8, a shooting mode key 10, a playback mode key 11, a macro key 14, a strobe key 15, a menu key 16, a ring key 17, a set key 18, and the like. Signals associated with these key operations are sent directly to the CPU 30.

  The computer interface unit 27 operates as a USB storage class driver when the digital camera 1 is connected to the PC 40. Thus, when the PC 40 is connected to the digital camera 1, the PC 40 handles the memory card 31 as an external storage device of the computer.

  The external storage IO device 28 inputs and outputs image data and the like with the memory card 31. The memory card 31 stores image data and the like supplied from the external storage IO device 28.

  The program code storage device 29 is for storing a program executed by the CPU 30, and is configured by a ROM, a flash memory, or the like.

  The CPU 30 controls the entire system according to a program stored in the program code storage device 29. The memory 23 is also used as a work memory for the CPU 30.

  When operation information is transmitted from the operation unit 26 by pressing a switch key of the operation unit 26, the CPU 30 performs image sensor 22, memory 23, display device 24, and image processing device based on the operation information. 25 etc. are controlled.

  Specifically, when operation information indicating that the shooting mode key 10 is pressed is transmitted from the operation unit 26, the CPU 30 sets each unit to the shooting mode. In this state, if the shutter key 9 is not pressed, the image sensor 22 is set to the preview mode, and if the shutter key 9 is pressed, the high-resolution mode for reading a high-resolution image to be captured is set. At this time, if the continuous shooting mode is set by operating the menu key 16, a predetermined number of image reading processes are executed at predetermined time intervals as the shutter key 9 is pressed.

  When the operation information indicating that the playback mode key 11 is pressed is transmitted, the CPU 30 sets each unit to the playback mode.

  Further, the CPU 30 records preview image and high-resolution image data on the memory card 31 via the external storage IO device 28, and reads the recorded image data from the memory card 31. The CPU 30 records image data compressed in, for example, a JPEG (Joint Photographic Experts Group) format in the memory card 31.

  When the image data is temporarily stored in the memory 23, the CPU 30 records the preview image and the high-resolution image data in different storage areas. Further, the CPU 30 records the image data separately in the image file in the memory card 31.

  Further, the CPU 30 records preview image and high-resolution image data on the memory card 31 via the external storage IO device 28, and reads the recorded image data from the memory card 31. The CPU 30 creates an image file for storing image data in the memory card 31.

  FIG. 3 is a block diagram showing a functional configuration of the CPU 30. Functionally, the CPU 30 includes an image acquisition unit 30a, a feature point selection unit 30b, a tracking unit 30c, an evaluation point generation unit 30d, an evaluation unit 30e, a conversion formula calculation unit 30f, and an image processing unit 30g.

  As shown in FIG. 4, the image acquisition unit 30 a acquires at least two images A and B obtained by continuous shooting or the like. In this case, the image A is stored as a reference image in the first buffer 23a of the memory 23, and the image B is stored as a tracked image in the second buffer 23b of the memory 23.

  The feature point selection unit 30b selects feature points arranged on a straight line from the image A obtained by the image acquisition unit 30a under certain restrictions.

  The tracking unit 30c tracks each feature point selected by the feature point selection unit 30b on the image B.

  The evaluation point generation unit 30d selects a predetermined number of points from each feature point tracked by the tracking unit 30c, and recursively generates feature points arranged on a straight line on the image B based on these points as evaluation points. To do. Specifically, the evaluation point generation unit 30d extracts four points that form a rectangular vertex from the feature points obtained by the tracking unit 30c, and arranges them on a straight line on the image B based on the four points. Feature points are generated recursively as evaluation points. The evaluation point generation unit 30d is obtained by the first additional processing unit 30d-1 that obtains the inner point of the rectangle from the four points that form the vertex of the quadrangle, and the first additional processing unit 30d-1. A second additional processing unit 30d-2 that obtains a boundary point that is an intersection of the inner point and the side of the quadrangle, and the boundary point obtained by the second additional processing unit 30-2 and the vertex of the quadrangle A third additional processing unit 30d-3 for obtaining an outer point that is an intersection of the extension line and the extension line of the side of the front square, and the inner points obtained by the additional processing units 30d-1 to 30d-3 , Boundary points, and outer points are added as evaluation points.

  The evaluation unit 30e evaluates the suitability between the evaluation points generated by the evaluation point generation unit 30d and the feature points tracked on the image B.

  The conversion formula calculation unit 30f calculates a conversion formula between images using a collection of points having the highest suitability by the evaluation unit 30e.

  Further, the image processing unit 30g aligns the images A and B based on the conversion formula obtained by the conversion formula calculation unit 30f to generate a predetermined image. The predetermined image is, for example, a panoramic image or an image subjected to camera shake correction.

  Next, before explaining a specific processing procedure, the present method will be described with reference to FIGS. 5 to 8 for easy understanding.

  The entire process is based on the RANSAC algorithm. The difference from the conventional RANSAC that repeatedly calculates the temporary transformation matrix is that the following restrictions are placed on the arrangement of the feature points, and the calculation of the “coordinates transferred by the transformation” used in the matching point determination method is as follows: Rather than performing conversion parameters (projection conversion matrix) as in the prior art, the determination is made directly from the coordinates of each point obtained recursively from four sets of feature points.

  As will be described later, since the coordinates of each point are recursively obtained, there is a concern that calculation errors may accumulate. However, it is sufficient if there is sufficient accuracy to obtain an inlier set by RANSAC, and the final parameters are determined. Since it is only necessary to recalculate after exiting the loop, it is not a big problem.

(1) Restriction of feature point coordinates (Input restriction)
First, the following restrictions are placed when selecting feature points.

  The coordinates of each feature point on the reference plane, which is one plane, are fixedly set to predetermined coordinates in a square lattice pattern. Here, the feature points on the featureless pattern may be removed from the feature point correspondence set.

  The other plane is called the tracked plane, and the coordinates of each feature point on it are determined by the tracking algorithm (block matching or gradient method) to which coordinate the feature pattern in the feature point coordinates on the reference plane has moved. It shall consist of estimated coordinates.

These feature points correspond to n sets, that is,
((X ₀ , y ₀ ) _, (x ″ ₀ , y ″ ₀ )), ((x ₁ , y ₁ ), (x ″ ₁ , y ″ ₁ )),..., ((X _n−1 , y _n-1 ), (x " _n-1 , y" _n-1 ))
In this manner, it is assumed to be configured as an array of pairs.

  Note that (x, y) represents the coordinates of the feature points on the reference plane, and (x ″, y ″) represents the coordinates of the feature points tracked on the tracked plane. Further, (x ′, y ′) described later represents the coordinates of feature points on the tracked plane.

(Subset extraction)
Extraction of a random subset by the conventional RANSAC is performed by randomly selecting four feature points if the algorithm uses four sets of feature points. If feature points overlap or degenerate conditions (aligned in a straight line, etc.), there is no meaning and they are extracted again. However, in this method, restrictions are further strengthened.

That is, four sets of feature points are selected, and the arrangement of the four points on the reference plane (image A) forms each vertex of a rectangle, and the length of one side thereof is 2 ^m (2 ^m ( Where m is an arbitrary positive integer).

  As a specific method, for example, the upper left coordinate and m may be selected at random, or all the combinations may be selected sequentially by a loop because there are few restrictions and the number of combinations is strong. Of course, if the coordinates of feature points that do not exist are included, the coordinates are discarded and the process proceeds to the next combination.

  FIG. 5 shows an example of extracting a subset of feature points on the reference plane. Black dots indicate feature point coordinates. A set of four points constituting each vertex of a square indicated by a dotted frame in the figure is an extraction target.

(2) Derivation method of coordinates to be moved by conversion This is a feature of the present embodiment.

  Usually, the projective transformation is uniquely determined from the correspondence of the four feature points in the square arrangement (except in the case where there is an exceptional corresponding point where the projective transformation is degenerated). In this embodiment, a parameter (transformation matrix) for projective transformation from the reference plane (image A) to the tracked plane (image B) is not obtained at this time, but the name is given as transformation H for convenience. To do.

Other feature points corresponding the surrounding four points, as described below, reference plane of the feature point coordinates (x _i, y _i) is the coordinates (x _'i, which is transferred to the track plane by converting H, y ′ _I ) is obtained. The processing after the determination is similar to general RANSAC, and the compatibility with the tracking coordinates (x ″ _i , y ″ _i ) can be determined by examining the distance.

The reason why the coordinates (x ′ _i , y ′ _i ) can be obtained directly without obtaining the component of the transformation H is that the positional relationship between the feature points is restricted, so This is because “shift to a straight line” can be used. That is, the six straight lines connecting the four feature points extracted from the reference plane are directly transferred to the six straight lines formed by the four feature points extracted from the tracked plane. Furthermore, it can be said that “the intersection of the straight lines moves to the intersection of the straight lines”. Therefore, it is a principle that the coordinates of the intersection point to be moved by conversion are obtained.

  By applying the derivation method shown here recursively to obtain all feature point conversion coordinates, the initial four extraction points are converted from the points a, b, c, and d on the reference plane using the conversion H. The moving points a ′, b ′, c ′, d ′ and the tracked points a ″, b ″, c ″, d ″ have the same coordinates. Other derived points are not necessarily the same coordinates.

  Here, a method for deriving an inner point, a boundary point, and an outer point as other characteristic points around the four points will be described.

(3) Derivation of conversion coordinates of inner point First, a method for deriving an inner point will be described. Here, the inner point refers to the center point of the selected rectangle of the reference image.

  FIG. 6 is a diagram for explaining a method for deriving the transformation coordinates of the inner point. FIG. 6A shows an example of the positional relationship of the reference plane, and FIG. 6B shows an example of the positional relationship of the tracked plane. The figure also shows an example of the coordinate e ″ tracked (the distance between e ′ and e ″ can be determined to determine whether it is a matching point). However, in the subsequent processing, when e ′ is obtained for recursive processing, e ″ does not necessarily exist. Feature points are excluded and tracking is not performed, or derived as an intermediate result of recursive processing. In this case, the width unit may be smaller than the feature point grid width, and the same applies to FIGS.

  Using the transformation H that moves the square vertices a, b, c, d of the reference plane to the points a ′, b ′, c ′, d ′ of the tracked plane, the center point e is converted to the tracked plane. The coordinate e ′ can be obtained as follows.

The internal representation of the plane coordinates (x _p , y _p ) of the point p is represented by an extension vector as (x _p , y _p , 1). Although the point at infinity is also handled later, the point at infinity in the (x _p , y _p ) direction is expressed as (x _p , y _p , 0) when viewed from the plane origin. It may be considered as homogeneous coordinates.

  Here, the point e on the reference plane is an intersection of a straight line passing through the points a and d and a straight line passing through the points b and c. Therefore, the point e ′ on the tracked plane moved by the transformation H may be obtained by calculating the intersection of a straight line passing through the points a ′ and d ′ and a straight line passing through the points b ′ and c ′. This calculation can be easily configured by using “a method for obtaining a straight line passing through two points” and “a method for obtaining an intersection of two straight lines”. In later processing, these two methods are used as subroutines.

"How to find a straight line passing through two points"
A straight line passing through two points p and q on the plane is expressed by the following equation (1).

_{(X 'p -x' q)} (y'-y 'q) = (y' p -y 'q) (x'-x' q)
... (1)
In the above equation (1), when similar terms are put together, a straight line: α ₁ x ′ + α ₂ y ′ + α ₃ = 0.

  Later, one of the points p and q may become an infinite point. At that time, the direction vector is in the direction of the infinite point, and the point on the plane (the one that is not the infinite point) What is necessary is just to obtain a straight line passing through the point.

“How to find the intersection of two straight lines”
Assuming that the intersection of two straight lines is a straight line: α ₁ x ′ + α ₂ y ′ + α ₃ = 0, and a straight line: β ₁ x ′ + β ₂ y ′ + β ₃ = 0, a solution of the following linear equation may be obtained.

However, when the two straight lines are parallel (in the case of numerical calculation, they are almost parallel), an infinite point on the extension of the straight line, for example, (α ₂ , −α ₁ , 0) to (β ₂ , −β ₁ , 0) is determined as the solution.

(4) Derivation of transformation coordinates of boundary point Next, a method of deriving the boundary point will be described. Here, the boundary point refers to the midpoint of each side of the selected rectangle of the reference image.

  FIG. 7 is a diagram for explaining a method for deriving the conversion coordinates of the boundary point. FIG. 7A shows an example of the positional relationship of the reference plane, and FIG. 7B shows an example of the positional relationship of the tracked plane. It should be noted that any of the four sides constituting the square is completely symmetrical, and here, the point f on the right side will be described as an example.

  Using a transformation H that moves the square vertices a, b, c, d of the reference plane to the points a ′, b ′, c ′, d ′ of the tracked plane, the midpoint of any side is Whether it is converted into coordinates can be determined as follows.

  Assume that a point e ′ corresponding to the center point e is obtained. First, an intersection point w ′ between a straight line passing through the points a ′ and b ′ and a straight line passing through the points c ′ and d ′ is obtained by the method described above. As described above, if the straight lines are parallel, w ′ is a point at infinity.

Next, an intersection of a straight line passing through the points b ′ and d ′ and a straight line passing through the e ′ and w ′ is obtained by the above-described method. This intersection is the boundary point f ′ .

  Similarly, when the midpoints of all the sides are obtained, the transformation coordinates of the vertices of four small squares each having a half length are obtained. In addition, if the derivation of the internal point coordinates and the derivation of the boundary point coordinates are applied recursively to these small square transformations, all of the lattice-like feature points inside the original square and the boundary are obtained. Conversion coordinates can be obtained.

(5) Derivation of transformation point of outer point Next, a method for deriving the outer point will be described. Here, the outer point refers to a grid point that is separated from the vertex of the selected rectangle of the reference image by a rectangular side length.

  FIG. 8 is a diagram for explaining a method for deriving the transformation coordinates of the outer points. FIG. 7A shows an example of the positional relationship of the reference plane, and FIG. 7B shows an example of the positional relationship of the tracked plane. It should be noted that any side or direction of the four sides constituting the square is completely symmetrical, and here, the point g with the lower side extended to the right will be described as an example.

  Extending any side outward by the length of the side using transformation H that moves the square vertices a, b, c, d of the reference plane to the points a ′, b ′, c ′, d ′ of the tracked plane It can be determined as follows to which coordinate on the tracked plane the converted point is converted.

Now, it is assumed that a point f ′ corresponding to the midpoint f on the right side is obtained. The point g ′ moved by the transformation H is a straight line passing through the points a ′ and f ′ from the coordinates of the points a ′, b ′, c ′ and d ′ moved by the transformation H, and the points c ′ and d ′. Find the intersection of the straight lines passing through. This intersection is the outer point.

  In this way, all of the outer neighboring points can be determined, and the coordinates of the vertices of four squares of the same size as the original are determined. By applying the above method recursively and sequentially tracing the adjacent points and their interiors / boundaries, the transformation coordinates of all feature points on the screen can be obtained.

A specific processing procedure will be described below.
Assume that two continuous images A and B are aligned. The feature points arranged on the straight line in the image A should be arranged on the straight line even on the image B distorted by the rotation of the camera or the like. Therefore, when the feature points arranged on the straight line in the image A are tracked on the image B and are not on the straight line on the image B, they can be excluded as feature points that are moving abnormally.

  Here, although it is guaranteed that the straight line moves to the straight line, the angle of the straight line itself can change, and the length of each part of the straight line also changes. Therefore, the problem of how to draw an appropriate straight line for evaluation on image B can be solved by repeating the method of “creating an inner point based on four points, creating a boundary point, and creating an outer point”. It is.

  FIG. 9 is a flowchart showing a processing procedure for alignment between images in the first embodiment. FIG. 10 is a flowchart showing the evaluation target addition process included in FIG. Each process shown in these flowcharts is stored in the program code storage device 29 in the form of a program code. CPU30 which is a computer performs the following processes by reading this program.

  As shown in the flowchart of FIG. 9, the CPU 30 acquires two consecutive images A and B (step S100). The images A and B may be acquired directly from the image sensor 22 by continuous shooting or the like, or may be provided from the outside through the memory card 31 or the like. The image A is stored as a reference image in the first buffer 23a of the memory 23 shown in FIG. 3, and the image B is stored as a tracked image in the second buffer 23b.

  When the image A and the image B are obtained, first, the CPU 30 selects a grid-like feature point from the image A that is the reference image as shown in FIG. 5, that is, a collection of feature points arranged on a straight line. Select (step S101). Subsequently, the CPU 30 tracks feature points on the image B corresponding to lattice points (feature points) on the image A using block matching or a gradient method (step S102). Then, arbitrary four points are selected from the feature points on the image B obtained by this tracking process, and these are set as virtual lattice points (step S103). These four points correspond to the points a ′, b ′, c ′, and d ′ shown in FIG. 6B, and form quadrangular vertices.

  The “virtual grid points” referred to here are points that are transferred from the feature points of the image A to the image B by the transformation H, and are used as evaluation points. At this point, there are still only 4 points, so the number is increased by the following method based on these 4 points.

  That is, the CPU 30 adds virtual lattice points by repeating the operation for drawing a straight line and the operation for obtaining an intersection point on the image B based on these four points (step S104).

  Specifically, as shown in the flowchart of FIG. 10, the CPU 30 obtains an inner point based on the four points on the image B selected in step S103 (step S201), and then uses the inner point as a cue to make a boundary. A point is obtained (step S202), and an outer point is obtained using the boundary point as a cue (step S203). Note that how to obtain these points is as described with reference to FIGS.

  Thus, the inner point, the boundary point, and the outer point are obtained in this order based on the four points obtained in step S103. As a result, a total of 13 points, one inner point, four boundary points, and eight outer points, generate virtual lattice points (evaluation points). These points mean the feature point coordinates transferred from the image A to the image B by the transformation H from the four points obtained in step S103.

  Subsequently, the CPU 30 newly selects four points forming a quadrangle from the added points (step S106), and repeats the same processing based on the four points to sequentially add virtual lattice points. (Step S104).

  When the number of virtual lattice points reaches the number of lattice points on the image A, that is, when virtual lattice points are generated in the entire image B (Yes in step S105), the loop here ends. In this case, the addition process may be repeated until the number of grid points on the image A finally, or may be terminated when a predetermined number is reached. If accuracy is desired, it is preferable to repeat additional processing until the number of grid points is reached.

  Next, the CPU 30 calculates the distance between the corresponding virtual lattice point and the tracking point on the image B (step S107). That is, each point transferred from the feature point of the image A to the image B by the transformation H is compared with each point obtained from the feature point of the image A by the tracking process in step S102, and the positional deviation between the two points is compared. Ask for.

  Then, if the distance between the two is equal to or less than the predetermined value, that is, if the positional deviation between the virtual lattice point and the tracking point is within an allowable range, the CPU 30 determines the tracking point at that time as a matching point, and The coordinates are recorded in a predetermined area of the memory 23 (step S108), and the sum of the matching points is obtained as the current score and recorded in the score management table 32 shown in FIG. 11 (step S109).

  For example, if the number of virtual lattice points is 100 and 20 of them are incompatible, 80 points are recorded in the score management table 32 as the score for that time. The score management table 32 is provided in the memory 23.

Subsequently, the CPU 30 again selects another four points as start points from the feature points on the image B (No in step S110 → S103). Then, as described above, the number of virtual grid points is increased by repeating the operation of drawing a straight line based on the four points and the operation of obtaining intersection points, and when the number of virtual grid points reaches the number of grid points, The suitability is determined based on the distance between the points and the tracking points, and the sum of the numbers obtained as the fit points is recorded in the score management table 32 as the current score (steps S104 to S109).
The above process is repeated a predetermined number of times.
When the score for the predetermined number of times is obtained (Yes in step S110), the CPU 30 reads out the matching point of the times with the highest score from the predetermined area of the memory 23, and uses the set of matching points. A transformation matrix (determinant of projective transformation) between image A and image B is calculated (step S111).

  In the example of FIG. 11, scores up to five times are recorded, and the matching point extracted the second time with the highest score among them is selected as the optimum point for conversion. Note that the method for obtaining the transformation matrix is the same as in the prior art, and therefore the description thereof is omitted here.

  After the conversion matrix is obtained in this way, the CPU 30 thereafter aligns the image A and the image B using the conversion matrix and generates, for example, a panoramic image according to a preset shooting mode, Predetermined image processing such as generating an image with corrected camera shake is performed (step S112). Since image processing using this transformation matrix is known, detailed description thereof will be omitted here. An image obtained by this image processing is compressed and stored in, for example, the memory card 31 after being compressed by a predetermined method such as JPEG.

  In this way, feature points arranged on a straight line under certain restrictions are selected, and “calculation for obtaining a straight line passing through two points” and “calculation for obtaining the intersection coordinates of two straight lines” are repeated based on the feature points. Thus, by recursively obtaining the points to be evaluated, it is possible to easily calculate a conversion formula between images using a collection of highly compatible points among them. Therefore, as compared with the conventional method of repeatedly obtaining a temporary conversion formula many times, the calculation amount can be greatly reduced, the hardware configuration can be simplified, and the conversion formula between images can be calculated at high speed.

  In addition, by applying the conversion formula obtained in this way to the image composition of the camera, for example, a panoramic image, an image subjected to camera shake correction, and the like can be easily created.

(Second Embodiment)
Next, a second embodiment of the present invention will be described.

  In the first embodiment, all the feature points on the image A are collectively tracked, and the subsequent processing is performed only on the image B. In the second embodiment, on the image A, While adding feature points, the evaluation points are added in correspondence with the feature points on the image B.

  Since the apparatus configuration is the same as that of the first embodiment, only the processing procedure will be described here.

  FIG. 12 is a flowchart showing a processing procedure for alignment between images in the second embodiment. Each process shown in this flowchart is stored in the program code storage device 29 in the form of a program code. CPU30 which is a computer performs the following processes by reading this program.

  As shown in the flowchart of FIG. 12, the CPU 30 acquires two consecutive images A and B (step S300). The images A and B may be acquired directly from the image sensor 22 by continuous shooting or the like, or may be provided from the outside through the memory card 31 or the like. The image A is stored as a reference image in the first buffer 23a of the memory 23 shown in FIG. 3, and the image B is stored as a tracked image in the second buffer 23b.

  When the images A and B are obtained, first, as described with reference to FIG. 5, the CPU 30 sets two sets of two points arranged on a straight line from the feature points on the image A, which is the reference image, under a certain restriction. In total, four points are selected (step S301). These four points correspond to the points a, b, c, and d shown in FIG. 6A, and form vertices of a quadrangle (rectangle on the image A).

  Subsequently, the CPU 30 tracks four points on the image B corresponding to the four points on the image A using block matching or a gradient method (step S302). In this case, since the image B may be distorted due to the rotation of the camera or the like, the four points on the image B obtained by the tracking process are not necessarily rectangular.

  Here, with respect to the image A, the CPU 30 adds feature points by repeating the operation of drawing a straight line and the operation of obtaining an intersection point in the group of four points selected in step S301 (step S303). This addition process is the same as in FIG. That is, on the image A, the inner point, the boundary point, and the outer point are obtained based on the four points obtained in step S301.

  The same applies to the image B, and the CPU 30 adds evaluation points by repeating an operation of drawing a straight line based on the four points obtained in the tracking process of step S302 and an operation of obtaining an intersection (step S306). . The “evaluation point” has the same meaning as the “virtual grid point” described in the first embodiment, and is a point that is transferred from the feature point of the image A to the image B by the transformation H.

  Further, the CPU 30 newly selects four points forming the vertices of a rectangle (rectangle in the image A) from the points added in the previous process from the image A (step S305), and performs the same process as described above based on the four points. The feature points are sequentially added repeatedly (step S303). The same applies to the image B. Four points that form a rectangular vertex are newly selected from the points added in the previous process from the image B (step S308), and the evaluation values are sequentially added by repeating the same process. (Step S306).

  Note that the processing in step S305 and the processing in step S308 are linked so that when four new points are selected on the image A, it naturally corresponds to the four points on the image A on the image B. You will choose 4 points that are related.

  When the number of points added on the images A and B reaches a predetermined number (Yes in step S304, Yes in step S307), the loop here ends. In this case, the addition process may be repeated until all the feature points on the image A are finally selected, or may be terminated when a predetermined number is reached. If accuracy is desired, it is preferable to repeat the additional process until all the feature points are selected.

  Next, the CPU 30 tracks the feature points on the image B corresponding to the feature points on the image A (step S309). Then, the CPU 30 calculates the distance between the corresponding evaluation point and the tracking point on the image B (step S310). That is, from each point transferred to the image B by transformation H from each feature point of the image A (each point obtained in the second loop) and each feature point of the image A (each point obtained in the first loop). Each point obtained by the tracking process in step S309 is compared, and a positional deviation between them is obtained. Note that the first loop indicates the repetition process of steps S303 to S305, and the second loop indicates the repetition process of steps S306 to S308.

  Then, when the distance between the two is equal to or less than the predetermined value, that is, when the positional deviation between the evaluation point and the tracking point is within an allowable range, the CPU 30 determines that the tracking point at that time is a matching point, and coordinates thereof Is recorded in a predetermined area of the memory 23 (step S311), and the sum of the matching points obtained at that time is obtained as the current score and recorded in the score management table 32 shown in FIG. 11 (step S312). .

  Subsequently, the CPU 30 again selects another four points as the start points from the image A (No in step S313) and tracks the four points on the image B corresponding to the four points (step S302). Then, in the same manner as described above, feature points are increased by repeating an operation of drawing a straight line based on the four points and an operation of obtaining an intersection point on the image A (steps S303 to S305). The same applies to the image B, and the evaluation points are increased while repeating the operation of drawing a straight line based on the four points corresponding to the four points selected in the image A and the operation of obtaining the intersection (steps S306 to S307).

Then, the CPU 30 tracks the feature point on the image B corresponding to the feature point on the image A, and determines the suitability based on the distance between the evaluation value and the tracking point on the image B, as described above. The sum of the numbers obtained as points is recorded in the score management table 32 as the current score (steps S309 to S312).
The above process is repeated a predetermined number of times.

  When the score for the predetermined number of times is obtained (Yes in step S313), the CPU 30 reads out the matching point of the times with the highest score from the predetermined area of the memory 23, and uses the set of the matching points. A transformation matrix (projection transformation determinant) between the images A and B is calculated (step S314).

  When the conversion matrix is obtained in this way, thereafter, the CPU 30 aligns the image A and the image B using the conversion matrix and generates, for example, a panoramic image according to a preset shooting mode, Predetermined image processing such as generation of an image with corrected camera shake is performed (step S315).

  As described above, even when a method for adding an evaluation point corresponding to a feature point on the image B while adding a feature point on the image A, the same effect as that of the first embodiment is obtained. Obtainable.

  The application of the present invention is not limited to RANSAC. For example, when there are four absolutely reliable points by using a special marker or the like, it can also be used when the outlier is removed from corresponding points including other errors at high speed.

  In each of the above-described embodiments, the digital camera has been described as an example. However, in addition to the digital camera, the present invention can be applied to any apparatus that can handle images, such as a personal computer. In this case, as shown in FIG. 2, the digital camera 1 and the PC 40 are connected, and each image continuously taken by the digital camera 1 is transferred to the PC 40, and processing as shown in FIGS. 9 and 10 is performed on the PC 40 side. May be configured to execute.

  In short, the present invention is not limited to the above-described embodiments as they are, and can be embodied by modifying constituent elements without departing from the scope of the invention in the implementation stage. Various inventions can be formed by appropriately combining a plurality of constituent elements disclosed in the respective embodiments. For example, some components may be deleted from all the components shown in the embodiment. Furthermore, constituent elements over different embodiments may be appropriately combined.

  In addition, the methods described in each of the above-described embodiments are, as programs that can be executed by a computer, recording media such as magnetic disks (flexible disks, hard disks, etc.), optical disks (CD-ROM, DVD, etc.), semiconductor memories, etc. Can be applied to various devices, or transmitted by a communication medium and applied to various devices. A computer that implements this apparatus reads the program recorded on the recording medium, and executes the above-described processing by controlling the operation by this program.

FIG. 1 is a diagram showing an external configuration when the image processing apparatus according to the first embodiment of the present invention is applied to a digital camera. FIG. 1 (a) is mainly a front configuration, and FIG. It is a perspective view which mainly shows the structure of a back surface. FIG. 2 is a block diagram showing an electronic circuit configuration of the digital camera in the embodiment. FIG. 3 is a block diagram showing a functional configuration of the CPU of the digital camera in the embodiment. FIG. 4 is a diagram showing an example of two consecutive images to be processed by the digital camera in the embodiment. FIG. 5 is a diagram showing an example of extracting a subset of feature points on the reference plane in the embodiment. FIG. 6 is a diagram for explaining a method for deriving the transformation coordinates of the inner point in the embodiment. FIG. 7 is a diagram for explaining a method for deriving the conversion coordinates of the boundary points in the embodiment. FIG. 8 is a diagram for explaining a method for deriving the transformation coordinates of the outer points in the same embodiment. FIG. 9 is a flowchart showing a processing procedure for alignment between images in the embodiment. FIG. 10 is a flowchart showing an evaluation object addition process in the embodiment. FIG. 11 is a diagram showing an example of a soot score management table in the embodiment. FIG. 12 is a flowchart showing a processing procedure for alignment between images in the second embodiment of the present invention. FIG. 13 is a flowchart showing a processing procedure for registration between images according to the conventional method.

Explanation of symbols

  DESCRIPTION OF SYMBOLS 1 ... Digital camera, 2 ... Body, 3 ... Shooting lens, 4 ... Self-timer lamp, 5 ... Optical finder window, 6 ... Strobe light emission part, 7 ... Microphone part, 8 ... Power key, 9 ... Shutter key, 10 ... Photographing Mode (R) key, 11 ... Playback mode (P) key, 12 ... Optical viewfinder, 13 ... Speaker section, 14 ... Macro key, 15 ... Strobe key, 16 ... Menu (MENU) key, 17 ... Ring key, 18 ... SET (SET) key, 19 ... display unit, 21 ... optical lens device, 22 ... image sensor, 23 ... memory, 24 ... display device, 25 ... image processing device, 26 ... operation unit, 27 ... computer interface unit, 28 ... External storage IO device 29 ... Program code storage device 30 ... CPU 30a ... Image acquisition unit 30b ... Feature point selection unit 30c ... Tracking unit 3 d ... Evaluation point generation unit, 30d-1 ... First addition processing unit, 30d-2 ... Second addition processing unit, 30d-3 ... Third addition processing unit, 30e ... Evaluation unit, 30f ... Conversion formula calculation Part, 30g ... image processing part, 31 ... memory card, 32 ... score management table, 40 ... PC.

Claims (6)

Image acquisition means for acquiring at least two images;
Feature point selection means for selecting grid-like feature points from the first image obtained by the image acquisition means;
Tracking means for tracking the feature point selected by the feature point selection means on the second image;
By selecting arbitrary four points from each feature point tracked by the tracking means and repeating the operation for drawing a straight line and the operation for obtaining the intersection point on the second image based on the four points, the evaluation point is obtained. An evaluation point generating means for adding
Evaluation means for evaluating the suitability between each evaluation point generated by the evaluation point generation means and each feature point tracked on the second image;
An image processing apparatus comprising: a conversion formula calculation unit that calculates a conversion formula between images using a set of points having the highest compatibility by the evaluation unit. The evaluation point generating means newly selects four points forming a quadrangle from the points added by the tracking means, and repeats the same processing based on the four points to generate recursively as evaluation points. The image processing apparatus according to claim 1. The evaluation point generating means includes
First additional processing means for obtaining an inner point of the rectangle from four points forming the vertex of the rectangle;
Second additional processing means for obtaining a boundary point that is an intersection of the inner point obtained by the first additional processing means and the sides of the rectangle;
A third additional processing means for obtaining an outer point that is an intersection of an extension line between the boundary point obtained by the second additional processing means and the vertex of the rectangle and an extension line of the side of the previous square;
The image processing apparatus according to claim 2, wherein an inner point, a boundary point, and an outer point obtained by each of the additional processing means are added as evaluation points. Image processing means for generating a predetermined image by aligning the first image and the second image based on the conversion formula obtained by the conversion formula calculation means. The image processing apparatus according to claim 1. A first step of acquiring at least two images;
A second step of selecting grid-like feature points from the first image obtained by the first step;
A third step of tracking on the second image the feature points selected by this second step;
By selecting arbitrary four points from each feature point tracked in the third step, and repeating the operation of drawing a straight line and the operation of obtaining an intersection point on the second image based on the four points, A fourth step of adding evaluation points ;
A fifth step of evaluating the suitability between each evaluation point generated by the fourth step and each feature point tracked on the second image;
An image conversion method comprising: a sixth step of calculating a conversion formula between images using a set of points having the highest compatibility in the fifth step. A program recorded on a computer-readable recording medium,
In the computer,
A first function for acquiring at least two images;
A second function of selecting grid-like feature points from the first image obtained by the first function;
A third function for tracking the feature point selected by the second function on the second image;
By selecting arbitrary four points from each feature point tracked by the third function, and repeating the operation for drawing a straight line and the operation for obtaining the intersection point on the second image based on the four points, A fourth function for adding evaluation points ,
A fifth function for evaluating suitability between each evaluation point generated by the fourth function and each feature point tracked on the second image;
A program for realizing a sixth function for calculating a conversion formula between images using a set of points having the highest compatibility by the fifth function.

Images