JP2012164188A - Image processing apparatus, image processing method and program - Google Patents

Abstract

PROBLEM TO BE SOLVED: To provide an image processing apparatus, an image processing method and a program which are capable of estimating a position/attitude of a specific area in an image with high accuracy.SOLUTION: The image processing apparatus includes: a feature point extraction part 11 for extracting a feature point of an input image I1; a matching part 12 for determining correspondence of the feature point with a reference image R; a homography calculation part 13 for calculating a relation of projection between the input image I1 and the reference image R based on the correspondence; and an image conversion part 14 for converting at least a part of the input image I1 based on the relation of projection. The homography calculation part 13 calculates a homography matrix H1 based on the correspondence of the feature point with the reference image R for the input image I1, and calculates a homography matrix H2 based on the correspondence of the feature point with the reference image R for the input image I1 converted by the image conversion part 14 on the basis of the homography matrix H1, and calculates again the relation of projection between the input image I1 and the reference image R based on the homography matrixes H1, H2.

Description

  The present invention relates to an image processing apparatus, an image processing method, and a program.

  In recent years, needs for augmented reality (AR) are increasing. Augmented reality is given by, for example, synthesizing an additional image such as a computer image with an input image based on a specific region (or a specific object) in the input image. In this case, the quality of augmented reality greatly depends on the estimation accuracy of the position / posture of the specific region in the input image.

  In such image processing, a reference image corresponding to a marker is used to designate a specific area (marker) in the input image. Then, a feature point of the input image is extracted, a correspondence relationship between the feature point and the reference image is determined, and a projective relationship between the input image and the reference image is calculated based on the correspondence relationship, whereby a marker in the input image is calculated. Is estimated. Then, the additional image is converted based on the projective relationship, and the converted additional image is combined with the input image using the marker as a reference.

  However, there are errors in the marker position / posture estimation results. In particular, when the input image is a moving image, even if the camera and marker are stationary, there is an error in the estimation results for each frame, causing a state where the additional image slightly shakes on the composite image, and the quality of augmented reality Will fall.

  As an error of the estimation result, there is a point that the posture of the image is different between the marker in the input image and the reference image. The feature points of the image are extracted using various filters such as a linear filter and a non-linear filter, and the output of the filter changes according to the posture of the image. For example, a coordinate value of a feature point differs between an image and an image obtained by rotating this image, or the same feature point (corresponding point) corresponding to the image may not be found.

  Here, the error of the coordinate value can be suppressed by averaging if a large number of corresponding points are found, thereby suppressing the influence on the error of the estimation result. However, if the posture of the image differs between the marker and the reference image, many corresponding points cannot be found, the influence on the error of the estimation result cannot be suppressed, and a good estimation result cannot be obtained.

  Accordingly, the present invention is intended to provide an image processing apparatus, an image processing method, and a program capable of accurately estimating the position / posture of a specific region in an image.

  According to an aspect of the present invention, a feature point extraction unit that extracts a feature point of an input image, a correspondence relationship determination unit that determines a correspondence relationship of feature points with a reference image, an input image based on the correspondence relationship, A projection relationship calculation unit that calculates a projection relationship of the reference image; and an image conversion unit that converts at least a part of the input image based on the projection relationship, the projection relationship calculation unit between the input image and the reference image A first projective relationship is calculated based on the feature point correspondence, and an input image converted based on the first projective relationship by the image conversion unit is input to the second reference point based on the feature point correspondence with the reference image. An image processing apparatus is provided that calculates a projective relationship and recalculates a projective relationship between an input image and a reference image based on the first and second projective relationships.

  The image conversion unit may convert at least a part of the input image so that the posture of the specific region in the input image approaches the posture of the reference image based on at least the first projective relationship.

  The projecting relationship calculation unit further calculates a third projecting relationship based on the correspondence relationship of the feature points with the reference image for the input image converted by the image conversion unit based on the first and second projecting relationships. The projection relationship between the input image and the reference image may be calculated again based on the first to third projection relationships.

  The feature point extraction unit may extract feature points using different methods when calculating the first projective relationship and when calculating the second projective relationship.

  The correspondence relationship determination unit may determine the correspondence relationship by using different methods for calculating the first projective relationship and calculating the second projective relationship.

  When the first projective relationship is calculated, a method that is more robust with respect to the posture change of the specific region in the input image may be used than when the second projective relationship is calculated.

  The image conversion unit may convert an image of a specific area corresponding to the reference image in the input image.

  The image processing apparatus further includes an image composition unit that synthesizes an image with the input image, the image conversion unit further converts the additional image based on at least the first and second projection relationships, and the image composition unit converts the added image You may synthesize | combine an image to the specific area | region corresponding to a reference image among input images.

  According to another aspect of the present invention, with respect to an input image, a first projection relationship is calculated based on a correspondence relationship between feature points with a reference image, and at least a part of the input image is calculated based on the first projection relationship. Converting the converted input image, calculating a second projective relationship based on the correspondence between the feature points with the reference image, and determining the projective relationship between the input image and the reference image based on the first and second projective relationships. An image processing method including calculating again is provided.

  According to another aspect of the present invention, there is provided a program for causing a computer to execute the image processing method. Here, the program may be provided using a computer-readable recording medium, or may be provided via a communication unit or the like.

  ADVANTAGE OF THE INVENTION According to this invention, the image processing apparatus, the image processing method, and program which can estimate the position and attitude | position of the specific area | region in an image with high precision can be provided.

It is a figure which shows the concept of the image processing method which concerns on embodiment of this invention. 1 is a block diagram illustrating a configuration of an image processing apparatus according to an embodiment of the present invention. It is a flowchart which shows operation | movement of an image processing apparatus. It is a figure (1/2) which shows an example of an image processing method. It is a figure (2/2) which shows an example of an image processing method. It is a flowchart which shows the general image processing method which synthesize | combines an image to the specific area | region in an image.

  Exemplary embodiments of the present invention will be described below in detail with reference to the accompanying drawings. In addition, in this specification and drawing, about the component which has the substantially same function structure, duplication description is abbreviate | omitted by attaching | subjecting the same code | symbol.

[1. General image processing method]
First, a general image processing method for synthesizing an image with a specific area in the image will be described with reference to FIG. FIG. 5 is a flowchart showing a general image processing method for synthesizing an image with a specific area in the image.

  The image processing method is performed by an image processing apparatus such as a personal computer, a game machine, a PDA, or a mobile phone. In the image processing method, data indicating the feature point P of the reference image R is stored in a memory (not shown) or the like. The reference image R is a planar image for designating a specific region (marker M) in the input image I1, and may be an arbitrary texture, a two-dimensional code, or the like, for example. The feature point P is a pixel point that represents a feature portion in the image, such as a luminance edge component.

  As shown in FIG. 5, in the image processing method, a moving image or a still image is input as an input image I1 from a camera (not shown) or the like (step S51), and a feature point P of the input image I1 is extracted (step S52). . The input image I1 is matched (matched) with the feature point P with the reference image R (step S53). The correspondence relationship between the feature points P is determined by finding the same feature point P corresponding between the input image I1 and the reference image R.

  A projection relationship (homography) is calculated between the input image I1 and the reference image R based on the matching of the feature points P (step S54). The projective relationship between images is a relationship established between images obtained by capturing a specific plane from different points, and is also referred to as homography between images. Homography between images is represented using a 3 × 3 homography matrix. The homography matrix H1 indicating the projection relationship between the input image I1 and the reference image R includes an input image I1 in which the marker M is captured in an arbitrary posture, and a reference image R (an image in which the marker M is captured in a certain posture). Projection relations established between the two are shown.

  If the homography matrix H1 can be calculated appropriately (“Yes” in step S55), the position / posture of the marker M in the input image I1 is estimated. Then, the additional image is converted based on the homography matrix H1 (step S56), and the converted additional image is combined with the input image I1 using the marker M as a reference (step S57) and output as a combined image (step S58). . The additional image data is stored in a memory or the like as data such as a computer image. On the other hand, if the homography matrix H1 cannot be calculated properly (“No” in step S55), the position / posture of the marker M is not properly estimated, and the input image I1 itself is output (step S59).

  However, there are errors in the marker position / posture estimation results. In particular, when the input image is a moving image, even if the camera and marker are stationary, there is an error in the estimation results for each frame, causing a state where the additional image slightly shakes on the composite image, and the quality of augmented reality Will fall.

  As an error of the estimation result, there is a point that the posture of the image is different between the marker in the input image and the reference image. The feature point P of the image is extracted using various filters such as a linear filter and a nonlinear filter, and the output of the filter changes according to the posture of the image. For example, the coordinate value of the feature point P differs between an image and an image obtained by rotating this image, or the same feature point P (corresponding point) corresponding to the image may not be found.

  Here, the error of the coordinate value can be suppressed by averaging if a large number of corresponding points are found, thereby suppressing the influence on the error of the estimation result. However, if the posture of the image differs between the marker and the reference image, many corresponding points cannot be found, the influence on the error of the estimation result cannot be suppressed, and a good estimation result cannot be obtained.

  The homography between images can theoretically be calculated using the coordinate values of four sets of feature points P (corresponding points) corresponding to each other between the images. However, if there is an error in each coordinate value, an error also occurs in the homography. Therefore, in practice, the homography cannot be calculated with high accuracy using only the coordinate values of the four corresponding points.

  For this reason, assuming that the error of the coordinate value is a normal distribution, the number of corresponding points is increased and the homography is calculated by the least square method. In this case, the accuracy of homography calculation increases as the number of corresponding points increases, and more specifically, the number of corresponding points increases as the number of extracted feature points P increases. Therefore, the calculation accuracy of homography depends on the extraction accuracy of the feature point P.

  As an extreme example, if the marker M and the reference image R in the input image I1 are the same, the posture of the image matches between the marker M in the input image I1 and the reference image R. Feature points P are extracted, and the number of corresponding points is maximized. In other words, as the image posture differs between the marker M and the reference image R in the input image I1, the possibility that the feature point P is not extracted increases, the number of corresponding points decreases, and the homography calculation accuracy increases. descend.

[2. Concept of image processing method]
Next, the concept of the image processing method according to the embodiment of the present invention will be described with reference to FIG. FIG. 1 is a diagram showing a concept of an image processing method according to an embodiment of the present invention.

  As shown in FIG. 1, in the image processing method according to the embodiment of the present invention, first, for the input image I1, a first projective relationship (homography line) based on the correspondence relationship of the feature points P with the reference image R. Example H1) is calculated. Therefore, the position / posture of the specific area (marker M) in the input image I1 is roughly estimated. Then, at least a part of the input image I1 is converted based on the homography example H1 so that the posture of the marker M approaches the posture of the reference image R. In the example shown in FIG. 1, the marker M is rotated by about 135 ° clockwise in the input image I1, but slightly rotated by about 15 ° clockwise in the converted image I2.

  Next, for the input image I1 after conversion (converted image I2), a second projective relationship (homographic example H2) is calculated based on the correspondence relationship of the feature points P with the reference image R. As described above, in the converted image I2, the posture of the marker M is closer to the posture of the reference image R than the input image I1. Therefore, in the converted image I2, the position / posture of the marker M is estimated more precisely than when the position / posture of the marker M is estimated in the input image I1. In addition, the homography example H2 acts to cancel the error of the homography example H1. The position / posture of the marker M in the input image I1 can be estimated with high accuracy based on the homography example H obtained by combining the homography examples H1 and H2.

  Thus, the input image I1 is converted so that the posture of the marker M approaches the posture of the reference image R, and the homography between the input image I1 and the converted input image I1 (converted image I2) and the reference image R is calculated. Thus, the position / posture of the marker M in the input image I1 can be estimated with high accuracy.

[3. Configuration of Image Processing Device 10]
Next, the configuration of the image processing apparatus 10 according to the embodiment of the present invention will be described with reference to FIG. FIG. 2 is a block diagram showing a configuration of the image processing apparatus 10 according to the embodiment of the present invention.

  The image processing apparatus 10 according to the embodiment of the present invention is a personal computer, a game machine, a PDA, a mobile phone, or the like. As shown in FIG. 2, the image processing apparatus 10 includes a feature point extraction unit 11, a matching unit 12 (corresponding relationship determination unit), a homography calculation unit 13 (projection relationship calculation unit), an image conversion unit 14, and an image composition unit 15. And a storage unit 16. The image processing device 10 may be configured integrally with an image input device such as a camera, or may be configured integrally with an image display device such as a display.

  The storage unit 16 is configured by a storage device such as a memory, and stores data such as a reference image R, an additional image, and a feature point P of the reference image R, and also intermediate results of image processing such as homography calculation results. Store. The storage unit 16 may be divided into a plurality of sections, such as a storage unit that stores image data and a storage unit that stores feature point data.

  The reference image data is image data specifying the marker M in the input image I1, and the additional image data is image data combined with the input image I1 with the marker M as a reference. The feature point data is pixel point data representing a feature portion in an image, such as a luminance edge component. The feature point data includes at least data of the coordinate value and feature amount (luminance value, luminance value gradient, etc.) of the feature point P in the image.

  A moving image or a still image is input to the feature point extraction unit 11 as the input image I1. The feature point extraction unit 11 generates luminance value data corresponding to the input image I1, extracts the feature point P of the input image I1, and supplies the feature point data indicating the extraction result to the matching unit 12. The feature point extraction unit 11 is supplied with the converted image I2 converted by the image conversion unit 14, and extracts the feature point P of the converted image I2 as in the case of the input image I1.

  The feature point P is extracted using an arbitrary method / parameter such as Harris Corner, SIFT (Scale Invariant Feature Transform), Random Ferns, or the like. SIFT is a technique for describing the feature point P using the gradient direction of pixels around the feature point P, and is robust to the rotation of the marker M. Random Ferns is a method of converting the reference image R into an image and learning in advance, and is robust against changes in the posture of the marker M.

  The matching unit 12 is supplied with the feature point data of the input image I1 and the feature point data of the reference image R. The matching unit 12 matches the feature point P between the input image I1 and the reference image R, and supplies matching data indicating the matching result to the homography calculation unit 13. Further, the matching unit 12 is supplied with the feature data of the converted image I2, and matches the feature point P between the converted image I2 and the reference image R as in the case of the input image I1.

  The feature points P are matched using an arbitrary method / parameter such as a sum of absolute differences (SAD) or a norm of a difference between feature point P vectors as an evaluation value for measuring the similarity between the feature points P. The matching data includes at least coordinate value data in the input image I1 and coordinate value data in the reference image R for the feature point P for which the correspondence between the input image I1 and the reference image R is confirmed. Between the input image I1 and the reference image R, depending on the position / orientation of the marker in the input image I1, the same feature point P corresponding to the image cannot be found, and there is a one-to-one correspondence. May not be confirmed. Note that the description between the converted image I2 and the reference image R is the same as that between the input image I1 and the reference image R.

  The homography calculation unit 13 is supplied with matching data. The homography calculation unit 13 calculates the homography of the input image I1 and the reference image R based on the matching data, and supplies the homography to the image conversion unit 14 and the storage unit 16. Similarly to the case of the input image I1, the homography calculation unit 13 calculates a homography matrix H2 of the converted image I2 and the reference image R, and combines the homography matrices H1 and H2 to form a homography matrix H (= H1 Calculate H2).

  The homography of the input image I1 and the reference image R is a projection established between the input image I1 in which the marker M is captured in an arbitrary posture and the reference image R (an image of the marker M captured in a certain posture). This means a relationship and is expressed using a 3 × 3 homography matrix. The case of the converted image I2 will be described in the same manner as the case of the input image I1. The homography matrix is calculated using the coordinate values of four or more sets of feature points P (corresponding points) constituting the matching data.

  The image conversion unit 14 is supplied with input image data and a homography matrix. Based on the homography matrix H1, the image conversion unit 14 linearly converts the input image I1 (particularly the image of the area of the marker M) so that the posture of the marker M approaches the posture of the reference image R, and the input image I1 after conversion Is temporarily stored in the storage unit 16 as the converted image I2. The image conversion is expressed as Xdst = H · Xsrc, where the homography matrix H, the three-dimensional coordinate vector Xsrc of each pixel of the input image I1, and the conversion result Xdst are used.

  Here, the image conversion target may be only the image of the marker M region or the input image I1 itself including the marker M region. When the image of the marker M area is converted, the coordinates of the marker M area are calculated from the corner coordinates of the reference image R and the homography matrix H, and the image of the marker M area is input based on the coordinates. It is cut out from the image I1. The image conversion unit 14 linearly converts the additional image based on the homography matrix and supplies the converted additional image to the image composition unit 15.

  The image composition unit 15 is supplied with input image data, converted additional image data, and a homography matrix H. Based on the homography matrix H, the image synthesizing unit 15 synthesizes the converted additional image with the input image I1 based on the marker M in the input image I1, generates a synthesized image, and outputs the synthesized image to an image display device or the like.

  Each component of the image processing apparatus 10 is configured as hardware such as circuit logic and / or software such as a program. The component configured as software is realized, for example, by executing a program on a CPU (not shown).

  The image conversion unit 14 may be configured integrally with the image composition unit 15. Further, in order to process the input image I1 and the converted image I2, respectively, at least one or more of the feature amount extraction unit 11, the matching unit 12, the homography calculation unit 13, and the image conversion unit 14 may be provided individually. . Further, the input image I1 may be input to the feature point extraction unit 11 through a buffer (not shown).

[4. Operation of Image Processing Device 10]
Next, the operation of the image processing apparatus 10 according to the embodiment of the present invention will be described with reference to FIGS. 3 and 4A and 4B. FIG. 3 is a flowchart showing the operation of the image processing apparatus 10, and FIGS. 4A and 4B are diagrams showing an example of the image processing method.

  As shown in FIG. 3, the feature point extraction unit 11 receives moving image frame data as data of the input image I1 (step S31), and extracts a feature point P of the input image I1 (step S32). The data of the input image I1 is temporarily stored in the storage unit 16. The matching unit 12 is supplied with the feature point data of the input image I1 and the feature point data of the reference image R, and matches the feature point P between the input image I1 and the reference image R (step S33). Note that the feature point data of the reference image R may be extracted from the reference image R by the feature point extraction unit 11 and supplied to the matching unit 12 instead of being read from the storage unit 16.

  The homography calculation unit 13 is supplied with matching data, and calculates a homography matrix H1 indicating the projection relationship between the input image I1 and the reference image R (step S34). The homography matrix H1 is temporarily stored in the storage unit 16. For example, when four or more sets of feature points P (corresponding points) cannot be found (“No” in step S35), the homography matrix H1 cannot be calculated and the input image I1 is not synthesized. Is output (step S45).

  FIG. 4A shows a matching example of feature points P. In this example, the specific area (marker M) in the input image I1 is designated by a reference image R indicating a map image around Japan. The input image I1 captures a map image (marker M) suspended diagonally from the diagonally front. The reference image R is a map image corresponding to the upright marker M, and nine feature points P1 to P9 such as edge components of luminance values are extracted in advance. In FIG. 4A, for convenience, the feature point P is shown in the map image itself, not the luminance image of the map image.

  Between the reference image R and the input image I1, five feature points out of the nine feature points P1 to P9 are implied by using a line segment that connects the same feature points P (corresponding points) in the correspondence relationship. P1 to P5 are matched. This is because the posture of the image is greatly different between the marker M and the reference image R in the input image I1, and the remaining four feature points P6 to P9 cannot be appropriately extracted from the input image I1.

  The image conversion unit 14 is supplied with the data of the input image I1 and the homography matrix H1, and converts the input image I1 based on the homography matrix H1 (step S36). The converted input image I1 (converted image I2) is obtained by converting the input image I1 so that the posture of the marker M approaches the posture of the reference image R as compared to the input image I1. The data of the converted image I2 is temporarily stored in the storage unit 16.

  The feature point extraction unit 11 is further supplied with the data of the converted image I2, and extracts the feature point P of the converted image I2 (step S37). The matching unit 12 is supplied with the feature point data of the converted image I2 and the feature point data of the reference image R, and matches the feature point P between the converted image I2 and the reference image R (step S38).

  The homography calculation unit 13 is supplied with matching data, and calculates a homography matrix H2 indicating a projection relationship between the converted image I2 and the reference image R (step S39). The homography matrix H2 may be temporarily stored in the storage unit 16. When the homography matrix H2 cannot be calculated (“No” in step S40), the input image I1 is output without performing image synthesis (step S45). The homography calculation unit 13 combines the homography H1 and H2 to calculate a homography matrix H (= H1 · H2) (step S41).

  FIG. 4B shows a matching example of the feature point P related to FIG. 4A. In this example, in the converted image I2, an image including the region of the marker M is cut out from the input image I1 based on the homography matrix H1, and the posture of the marker M is closer to the posture of the reference image R than the input image I1. It is obtained by converting the cutout image. When the area of the marker M in the input image I1 is small, processing required for image conversion can be reduced. In FIG. 4B, for convenience, the feature point P is shown in the map image itself, not the luminance image of the map image.

  Between the reference image R and the converted image I2, all nine feature points P1 to P9 are matched so as to be implied by using a line segment that connects the same feature points P (corresponding points) in a corresponding relationship. Yes. This is because the posture of the image is close between the marker M and the reference image R in the converted image I2, and the feature points P1 to P9 can be appropriately extracted from the converted image I2.

  The image conversion unit 14 is supplied with the data of the additional image and the homography matrix H, and linearly converts the additional image based on the homography matrix H (step S42). The additional image is converted so that the posture of the additional image matches the posture of the marker M. The image synthesis unit 15 is supplied with the converted additional image data, the input image I1 data, and the homography matrix H, and synthesizes the converted additional image with the input image I1 based on the homography matrix H (step S43). Then, it is output as a composite image (step S43).

  The homography matrices H1 and H2 may be calculated using the same method (and parameters), or may be calculated using different methods (and / or parameters). Note that the different methods mean that the feature amount extraction method (including the type of feature amount data of the reference image) and / or the matching method of the feature points P are different.

  The homography matrix H1 is calculated using a technique that is robust against the change in the posture of the marker M because the image position / posture is highly likely to differ greatly between the marker M and the reference image R in the input image I1. It is preferable. On the other hand, the homography matrix H2 is calculated by using a method with high estimation accuracy in most cases because it is unlikely that the position and orientation of the image greatly differ between the marker M and the reference image R in the converted image I2. It is preferred that In general, the robustness with respect to the posture change of the marker M and the estimation accuracy are in a trade-off relationship. Therefore, the homography matrix H1 is calculated using a robust method, and the homography matrix H2 is calculated using a method with high estimation accuracy. By calculating, there is a high possibility that the position / posture of the marker M can be estimated with high accuracy.

  The position / posture of the marker M may be estimated based on the calculation result of three or more homography. For example, when based on the calculation results of the homography matrices H1, H2, and H3, the image conversion unit 14 further converts the converted image I2 into a secondary conversion image based on the homography matrix H2 (of course, the homography matrices H1, H2) Based on this, the input image I1 may be converted into a secondary conversion image).

  In the image processing apparatus 10, the feature point P is extracted with the reference image R, the feature point P is matched, and the homography is calculated for the secondary transformed image, and the projective relationship between the secondary transformed image and the reference image R is calculated. A homography matrix H3 is calculated. Then, a homography matrix H ′ (= H 1, H 2, H 3) indicating the projection relationship between the input image I 1 and the reference image R is calculated, and an additional image is synthesized with the input image I 1 based on the homography matrix H ′.

  Also in this case, the homography matrices H1, H2, and H3 may be calculated using the same method as described above, or may be calculated using different methods. Here, the posture of the marker M in the input image I1 after conversion is expected to approach the posture of the reference image R as the homography calculation stage increases. Therefore, the higher the calculation stage, the higher the possibility that the position / orientation of the marker M can be estimated with high accuracy by using a technique that has a low robustness to the attitude change of the marker M and a high accuracy level.

[5. Summary]
As described above, according to the present invention, while converting the input image I1 so that the posture of the marker M approaches the posture of the reference image R, the input image I1 and the converted input image I1 (converted image I2, etc.) By calculating the homography of the reference image R, the position / posture of the marker M in the input image I1 can be estimated with high accuracy.

  The preferred embodiments of the present invention have been described in detail above with reference to the accompanying drawings, but the present invention is not limited to such examples. It is obvious that a person having ordinary knowledge in the technical field to which the present invention pertains can come up with various changes or modifications within the scope of the technical idea described in the claims. Of course, it is understood that these also belong to the technical scope of the present invention.

  For example, in the above-described embodiment, the case where the image processing apparatus 10 receives the input image I1 from an image input apparatus such as a camera and outputs an output image to an image display apparatus such as a display has been described. However, the input image I1 may be input from an image reproducing device such as a recorder or a player, and the output image may be output to an image recording device such as a recorder or a player.

DESCRIPTION OF SYMBOLS 10 Image processing apparatus 11 Feature point extraction part 12 Matching part (correspondence determination part)
13 Homography calculator (projection relation calculator)
14 Image conversion unit 15 Image composition unit 16 Storage unit M Marker (specific area)
R reference image I1 input image I2 transformed image H, H1, H2 Homography matrix (projection relation)
P Features

Claims (10)

A feature point extraction unit for extracting feature points of the input image;
A correspondence determining unit that determines the correspondence of the feature points with the reference image;
A projection relationship calculating unit that calculates a projection relationship between the input image and the reference image based on the correspondence relationship;
An image conversion unit that converts at least a part of the input image based on the projection relationship,
The projection relationship calculation unit calculates a first projection relationship for the input image based on the correspondence relationship of the feature points with the reference image, and converts the input image based on the first projection relationship. For the input image, a second projection relationship is calculated based on the correspondence relationship of the feature points with the reference image, and the input image and the reference image are calculated based on the first and second projection relationships. An image processing apparatus that calculates a projection relationship again.   The image conversion unit converts at least a part of the input image based on at least the first projective relationship so that the posture of the specific region in the input image approaches the posture of the reference image. An image processing apparatus according to 1.   The projection relationship calculation unit further includes a third based on a correspondence relationship of the feature points with the reference image with respect to the input image converted by the image conversion unit based on the first and second projection relationships. The image processing apparatus according to claim 1, wherein a projection relationship between the input image and the reference image is calculated again based on the first to third projection relationships.   The feature point extraction unit extracts the feature points by using different methods when calculating the first projective relationship and when calculating the second projective relationship. The image processing apparatus according to claim 1.   The correspondence relationship determination unit determines the correspondence relationship by using different methods for calculating the first projective relationship and calculating the second projective relationship. The image processing apparatus according to claim 1.   6. The method according to claim 4, wherein when calculating the first projective relationship, a method that is more robust with respect to a change in posture of a specific region in the input image is used than when calculating the second projective relationship. The image processing apparatus described.   The image processing apparatus according to claim 1, wherein the image conversion unit converts an image in a specific area corresponding to the reference image in the input image. An image composition unit for compositing an image with the input image;
The image conversion unit further converts the additional image based on at least the first and second projective relationships,
The image processing apparatus according to claim 1, wherein the image synthesis unit synthesizes the converted additional image with a specific area corresponding to the reference image in the input image. For the input image, a first projective relationship is calculated based on the feature point correspondence with the reference image,
Converting at least a portion of the input image based on the first projective relationship;
For the converted input image, a second projective relationship is calculated based on the correspondence of feature points with the reference image,
An image processing method comprising: calculating again the projection relationship between the input image and the reference image based on the first and second projection relationships. For the input image, a first projective relationship is calculated based on the feature point correspondence with the reference image,
Converting at least a portion of the input image based on the first projective relationship;
For the converted input image, a second projective relationship is calculated based on the correspondence of feature points with the reference image,
A program for causing a computer to execute an image processing method including calculating again a projection relationship between the input image and the reference image based on the first and second projection relationships.

Images