JP5121673B2 - Image projection apparatus and image projection method - Google Patents

Description

  The present invention relates to an image projection apparatus having a function of automatically correcting distortion of a projected image when an image is projected onto a projection surface.

  In recent years, the use of image projection apparatuses (projectors) that present still images and moving images on a projection surface such as a screen is increasing. At this time, if the projection surface has a non-planar shape, or even if it is a planar shape and does not face the front of the projector, the projection surface has a shape different from the original image input to the projector. An image will be projected.

  In this way, even if an image is projected as it is from a projector, the projected image may be distorted (deformed), and it can be used by projecting after correcting the original image in advance. Several methods have been proposed by those who see the projected image look almost the same as the original image (see, for example, Patent Documents 1 to 6).

  In the method described in Patent Document 1, when a projector and a projection surface are installed, a rectangular pattern image is projected onto the projection surface, and the projected pattern image is captured by a camera. Then, the amount of distortion of the projected image is calculated from the external shape of the pattern image (deformed into a trapezoid) photographed by the camera, and when projecting the image that is actually viewed, the calculated distortion can be projected without distortion. The image is projected after a distortion opposite to the amount is given to the image in advance.

  In the method described in Patent Document 2, when a projector and a projection surface are installed, a test image (pattern image) composed of point images is projected onto the projection surface, and the projected test image is photographed with a camera. Then, by comparing the original test image with the test image taken by the camera, the amount of distortion of the projected image is calculated, so that when projecting an actually viewed image, the image can be projected without distortion. Then, a distortion opposite to the calculated distortion amount is given to the image in advance and corrected before the projection. Further, in the method described in Patent Document 2, a method of projecting a test image in a wavelength region other than the visible light region has also been proposed.

  In the method described in Patent Document 3, when a projector and a projection surface are installed, a pattern image is projected onto the projection surface, and the projected pattern image is captured by a camera. Then, by comparing the original pattern image with the pattern image captured by the camera, the amount of distortion of the projected image is calculated, so that when projecting an actually viewed image, the image can be projected without distortion. Then, a distortion opposite to the calculated distortion amount is given to the image in advance and corrected before the projection. As the pattern image, a method using only one point image and a method using an image including a plurality of points are described.

  In the method described in Patent Document 4, an image projected from a projector onto a projection plane is captured by a camera, feature points are extracted from an original image of the projected image, and correspondence corresponding to the feature points is captured from an image captured by the camera. Extract points. Then, the three-dimensional coordinates of the point on the projection plane corresponding to the feature point are determined using the feature point and the corresponding point, and the parameters of the projector and the camera, and the projected image is determined using the three-dimensional coordinate. Make corrections. Here, when performing correction, the image plane is divided into a plurality of triangular regions having the feature point as a vertex, and the triangle is inside the triangle from the three-dimensional coordinates of the vertex, assuming that the triangle is a plane in a three-dimensional space. A method for interpolating the three-dimensional coordinates of a pixel is shown.

  In the method described in Patent Document 5, the projector holds an approximate expression set in advance in order to correct the distortion of the projection image caused by the shape of the projection surface. Then, when the user inputs a variable value of the approximate expression, a deformation (distortion correction) process corresponding to the shape of the projection surface is performed. Here, as an approximate expression, for example, a parabolic expression is used for a cylindrical or spherical projection surface, and a linear expression that corrects a wall angle for a wall-shaped projection surface having a corner is a sine wave shape. A method using a trigonometric function is shown for the projection plane.

In the method described in Patent Document 6, the projector has an approximate expression (function expression) relating to the shape of the projection surface in advance. Then, the user inputs a variable value (variable parameter) of the approximate expression to generate a mesh model of the projection plane, and distortion correction is performed using the correspondence between the input image (plane) and the mesh model of the projection plane. Do.
Japanese Patent Laid-Open No. 10-200836 JP 2001-83949 A JP 2001-320652 A JP 2004-165944 A JP 2004-320664 A JP 2006-94458 A

  However, the conventional method has the following problems when the shape of the projection surface such as a screen or the positional relationship between the projection surface and the projector changes during image projection.

  In the method described in Patent Document 1, it is necessary to calculate a distortion amount by projecting a pattern image before projecting an actual image (appreciation image) to be appreciated. In other words, if the shape of the projection surface or the positional relationship between the projection surface and the projector changes during the projection of the appreciation image, the appreciation image is temporarily stopped and the pattern image is projected again to distort the image. The amount needs to be recalculated. Therefore, this is not practical when watching a moving image.

  Even the method described in Patent Document 2 has the same problem as the method described in Patent Document 1 when a pattern image composed of point images is used. In other words, when the shape of the projection surface or the positional relationship between the projection surface and the projector changes, it is necessary to stop the projection of the viewing image, project the pattern image again, and recalculate the distortion amount. Therefore, this is not practical particularly when watching a moving image.

  Further, when the pattern image is projected in a wavelength region other than the visible light region by the method described in Patent Document 2, the problem of the method described in Patent Document 1 can be solved. That is, even when the shape of the projection surface or the positional relationship between the projection surface and the projector changes during the projection of the viewing image, the amount of distortion can be obtained without interrupting the viewing of the image. However, since the projector and the camera must handle an image outside the visible light range, an increase in hardware costs is required for the realization.

  The method described in Patent Document 3 also has the same problem as the method described in Patent Document 1 because the distortion amount is calculated using the pattern image. That is, when the shape of the projection surface or the positional relationship between the projection surface and the projector changes, it is necessary to once stop the projection of the viewing image, project the pattern image again, and recalculate the distortion amount. Therefore, this is not practical when watching a moving image.

  In the method described in Patent Document 4, even when the shape of the projection plane or the positional relationship between the projection plane and the projector changes during the projection of the appreciation image, the distortion amount is not interrupted without interrupting the appreciation of the image. And the problem of the method described in Patent Document 1 can be solved. However, the feature points are approximated by a triangular plane in a three-dimensional space. For this reason, when the number of feature points in the image is small, the shape of the projection surface is approximated by a large triangular plane, which causes a problem that distortion correction is insufficient. In addition, since the approximation of the projection plane shape is determined only by the feature points of the image, the approximation of the projection plane shape may vary greatly depending on the image. Therefore, when the processing for calculating the distortion amount of the projection plane from the feature points of the image is performed (updated) at a certain time interval (for example, in units of several tens of frames, units of several seconds, etc.) There is a possibility that the projection surface approximate shape before the update and the projection surface approximate shape after the update are greatly different. As a result, the image correction method changes greatly, and the viewer may feel uncomfortable.

  In the method described in Patent Document 5, the shape of the projection plane can be expressed by an approximate expression, but the detailed shape is made by the user inputting a variable value of the approximate expression. That is, before projecting a real image (appreciation image) to be appreciated, it is necessary to project a pattern image or the like and set the variable value while visually observing by the user. In the case where the shape of the projection surface or the positional relationship between the projection surface and the projector changes during the projection of the appreciation image, the appreciation image is projected once as in the method described in Patent Document 1. It is necessary to stop and project the pattern image again to recalculate the distortion amount. Therefore, this is not practical when watching a moving image.

  In the method described in Patent Document 6, the shape of the projection plane can be expressed by an approximate expression, but the detailed shape is made by the user inputting a variable value of the approximate expression. That is, it is necessary to input a variable value by measuring the shape of the projection surface and the like before projecting a real image (appreciation image) to be appreciated. In the case where the shape of the projection surface or the positional relationship between the projection surface and the projector changes during the projection of the appreciation image, the appreciation image is projected once as in the method described in Patent Document 1. It is necessary to stop and project the pattern image again to recalculate the distortion amount. Therefore, this is not practical when watching a moving image.

  The present invention solves the above-described conventional problems. During projection of an appreciation image, the shape of the projection surface, the positional relationship between the projection surface and the image projection device changes, or the image in the moving image is changed. An object of the present invention is to provide an image projection apparatus capable of performing shape correction stably without interrupting image appreciation even when it greatly changes with time.

  In order to solve the above-described conventional problems, an image projection apparatus according to the present invention is an image projection apparatus that projects an input original image onto a projection plane, and the original image or the original image to be projected is corrected. An imaging unit that captures an image in which a projection target image that is a correction image is projected on the projection plane, a shape model holding unit that holds a shape model indicating a pattern of a predetermined shape, and a point on the projection target image Based on the correspondence between the first feature point that is and the second feature point that is a point on the captured image, the shape pattern of the projection plane is selected from the shape models held by the shape model holding unit. A model selection unit that selects a projection surface shape model that is a shape model indicating the parameters, and a parameter that brings the shape of the projection surface shape model close to the shape of the projection surface so that the captured image approaches the original image And a parameter calculation unit that repeatedly calculates a model parameter, which is a parameter indicating a positional relationship between the projection plane and the image projection device, based on the latest correspondence relationship between the first feature point and the second feature point; A correction unit that corrects the original image into the corrected image based on the projection plane shape model and the calculated model parameter each time the model parameter is repeatedly calculated, and the original image or the corrected image A projection unit that projects the projection target image onto the projection plane.

  According to this, the parameter calculation unit repeatedly calculates the model parameters, so that the shape of the projection plane shape model can be changed to the shape of the projection plane and the projection plane without interrupting the viewing of the image while the appreciation image is being projected. And a shape corresponding to the positional relationship between the projector and the image projection apparatus. In addition, since the original image is corrected based on the projection plane shape model and projected onto the projection plane, even if the image in the moving image changes greatly with time, the approximation of the projection plane shape does not vary greatly. The shape can be corrected stably. Therefore, even when the shape of the projection surface, the positional relationship between the projection surface and the image projection device changes, or when the image in the moving image changes significantly with time, the shape can be stably displayed without interrupting image viewing. Correction can be performed.

  Preferably, the parameter calculation unit is configured such that coordinates obtained by correcting the coordinates of the second feature point by the projection plane shape model and the model parameter are points of the original image corresponding to the second feature point. The model parameters are repeatedly calculated so as to approach the coordinates.

  According to this, the parameter calculation unit can calculate the optimal model parameter of the selected projection plane shape model.

  Preferably, the parameter calculation unit repeatedly calculates the model parameter so that the captured image becomes an image obtained by enlarging or reducing the original image.

  According to this, since the parameter calculation unit calculates the model parameters so that the captured image becomes an image obtained by enlarging or reducing the original image, the user views the image without distortion projected on the projection plane. Can do.

  Preferably, the model selection unit selects one projection plane shape model from among the shape models held by the shape model holding unit, and the parameter calculation unit selects the shape of the projection plane and the projection plane. The model parameter of the selected projection plane shape model is repeatedly calculated according to the change in the positional relationship between the image projection apparatus and the image projection apparatus.

  According to this, when the shape of the projection plane does not change significantly, the model parameters for the selected projection plane shape model are repeatedly calculated, so that the shape of the projection plane and the positional relationship between the projection plane and the image projection apparatus can be determined. Optimal model parameters according to changes can be calculated.

  Preferably, the parameter calculation unit calculates a model parameter corresponding to a shape model when the shape model held by the shape model holding unit is a projection plane shape model, and the model selection unit The difference between the coordinate of the second feature point corrected by the model parameter corresponding to the model and the coordinate of the point on the original image corresponding to the second feature point is less than or equal to a predetermined first threshold value A shape model when the number of certain second feature points exceeds a predetermined second threshold is selected as the projection plane shape model.

  According to this, it is suitable for the shape pattern of the projection plane by calculating the number of second feature points whose difference from the coordinates of the points on the original image is not more than the first threshold value from among a plurality of shape models. A projection plane shape model can be selected.

  The parameter calculation unit calculates model parameters corresponding to each shape model when the shape model held by the shape model holding unit is a projection plane shape model, and the model selection unit A coordinate model in which the coordinates of the second feature point are corrected by the model parameter corresponding to is selected as the projection plane shape model that is closest to the coordinates of the point on the original image corresponding to the second feature point. May be.

  According to this, a projection surface shape model most suitable for the shape pattern of the projection surface can be selected from a plurality of shape models.

  Preferably, the model selection unit selects the projection plane shape model based on an external signal.

  According to this, when the shape of the projection plane is known in advance, the user can specify the projection plane shape model to be selected.

  The present invention can be realized not only as such an image projection apparatus but also as an integrated circuit that includes each processing unit of such an image projection apparatus and controls the image projection apparatus. . In addition, it can be realized as a method in which the processing of each characteristic processing unit included in such an image projection apparatus is a step, or can be realized as a program for causing a computer to execute these steps. Needless to say, such a program can be distributed via a recording medium such as a CD-ROM or a transmission medium such as the Internet.

  As described above, according to the image projecting device of the present invention, the shape of the projection surface and the positional relationship between the projection surface and the projector are changed while the appreciation image is being projected. Even if it changes greatly with time, the shape correction can be performed stably without interrupting the viewing of the image.

  Hereinafter, embodiments of the present invention will be described with reference to FIGS. 1 to 13C.

(Embodiment 1)
FIG. 1 is a block diagram showing a configuration of an image projection apparatus according to Embodiment 1 of the present invention.

  The image projection device 100 is a device that receives an input image (still image or moving image) from the outside, corrects the input image according to the shape of the projection surface 120, and then projects the image onto the projection surface 120. As shown in the figure, the image projection apparatus 100 includes a projection unit 101, an imaging unit 102, image memories 103 and 104, a control unit 105, an image processing unit 106, a correction parameter calculation unit 107, and a correction unit 108.

  The projection unit 101 has a projector such as a liquid crystal transmission type or a micromirror reflection type, and projects the image on the projection plane 120 by projecting the image from the image processing unit 106.

  The imaging unit 102 captures an image projected on the projection plane 120. Specifically, the imaging unit 102 is a camera having a MOS or CCD solid-state imaging device, and captures an image projected on the projection plane 120 by the projection unit 101. The imaging area of the imaging unit 102 is set to include the projection area of the projection unit 101 with a margin within the normal use range of the image projection apparatus 100.

  The image memory 103 is a memory for holding an image captured by the imaging unit 102. Here, the captured image is an image captured by the imaging unit 102.

  The image memory 104 is a memory for holding an input image (original image) from the outside of the image projection apparatus 100 or an input image (corrected image) corrected by the correcting unit 108.

  Here, the original image refers to an image that has not been corrected, that is, an image that has not been input to the correction unit 108. Specifically, an original image is, for example, an image output from a computer, a still image or a moving image obtained by receiving a broadcast wave, or a content recorded on a storage medium such as a DVD, HDD, or semiconductor memory. This refers to still images and moving images that can be obtained. The corrected image refers to an image corrected to suppress image distortion (shape distortion) caused by the relative positional relationship between the image projection apparatus 100 and the projection plane 120 and the shape of the projection plane 120. The original image and the corrected image, which are projection target images, are hereinafter referred to as projection target images.

  The correction parameter calculation unit 107 receives images from the image memory 103 and the image memory 104, and repeatedly calculates correction parameters for correcting shape distortion. Details of the correction parameter calculation unit 107 will be described later.

  The correction unit 108 corrects the original image to a corrected image based on the calculated correction parameter each time the correction parameter is repeatedly calculated. Then, the correction unit 108 outputs the corrected image to the image processing unit 106. Since the correction parameter is obtained for the currently projected image, the actual correction of the image by the obtained correction parameter is delayed by about one frame.

  The image processing unit 106 performs processing for converting the image input from the correction unit 108 into a format suitable for the projection unit 101, and outputs the converted image signal to the projection unit 101. The conversion process includes, for example, image resolution conversion, YUV signal-RGB signal conversion, and the like.

  The control unit 105 controls the entire image projection apparatus 100. Specifically, the control unit 105 controls the timing at which the image memories 103 and 104 output an image to the correction parameter calculation unit 107, the processing of the correction parameter calculation unit 107, and the like.

  2A to 2C are diagrams illustrating an example of the projection surface 120 that is a projection target of the projection unit 101. FIG.

  FIG. 2A shows a case where an image is projected from the non-front direction (oblique direction) toward the projection surface 120a which is a plane from the image projection apparatus 100. In this case, shape distortion (trapezoidal distortion) occurs in the projection area of the projection surface 120 a, and this shape distortion can be corrected by the correction unit 108.

  FIG. 2B shows a case where an image is projected from the image projection apparatus 100 toward a projection surface 120b which is a curtain of a room. Also in this case, shape distortion due to the shape of the curtain occurs in the projection area of the projection surface 120b. This shape distortion can also be corrected by the correction unit 108.

  FIG. 2C shows a case where an image is projected from the image projection apparatus 100 toward a projection surface 120c that is a curved surface. In this case as well, shape distortion (such as irregularities) occurs in the projection area of the projection surface 120 c, and this shape distortion can also be corrected by the correction unit 108.

  Next, details of the correction parameter calculation unit 107 will be described.

  FIG. 3 is a block diagram illustrating a configuration of the correction parameter calculation unit 107.

  As shown in the figure, the correction parameter calculation unit 107 includes a feature point extraction unit 301, a feature point matching unit 302, a shape model holding unit 303, a model parameter estimation unit 304, and a parameter holding unit 305.

  The feature point extraction unit 301 receives the captured image held in the image memory 103 and the original image or the corrected image held in the image memory 104.

  The feature point extraction unit 301 performs feature point extraction on the original image or the corrected image input from the image memory 104 and the captured image input from the image memory 103. Specifically, the feature point extraction unit 301 calculates a first feature point that is a point on the original image and a second feature point that is a point on the captured image until the original image is corrected to a corrected image. Extract. In addition, when the original image is corrected to a corrected image, the feature point extraction unit 301 extracts a first feature point that is a point on the corrected image and a second feature point that is a point on the captured image. That is, the feature point extraction unit 301 extracts a first feature point that is a point on the projection target image and a second feature point that is a point on the captured image.

  The feature point matching unit 302 performs feature point matching between images (corresponding relationship between feature points) from the first feature point and the second feature point extracted by the feature point extraction unit 301.

  The shape model holding unit 303 holds a shape model indicating a pattern having a predetermined shape. That is, the shape model holding unit 303 holds a shape model (approximate expression, function expression) that represents the shape of the projection plane 120. The shape model held here may be one or plural.

  FIG. 4 is a diagram illustrating an example of a shape model held by the shape model holding unit 303.

  As shown in the figure, the shape model holding unit 303 holds the shape model specified by the “formula”. For example, a shape model whose “model name” is a projective deformation model is specified by “formula” (x ′, y ′) = f (x, y). Here, (x, y) is the coordinates of the second feature point, and (x ′, y ′) is the coordinates of the second feature point corrected by the shape model.

  For example, when it can be assumed that the projection surface 120 is a plane, a projective deformation model is used. When the projection surface 120 can be assumed to be a cylinder or a sphere, a cylinder model or a sphere model specified by a functional expression expressing the cylinder or the sphere is used. Further, when the projection surface 120 has a complicated shape, the shape can be modeled using a Thin Plate Spline (TPS) model or the like.

  Here, TPS is a technique for assuming that the image plane is a thin film and performing a deformation that distorts the thin film. At this time, the thin film is deformed so that the first feature point and the second feature point coincide. This deformation is performed by combining affine deformation and local non-linear deformation. In the local non-linear deformation, the deformation is performed by superposing Radial Basis Function.

  Returning to FIG. 3, the model parameter estimation unit 304 includes a model selection unit 306 and a parameter calculation unit 307.

  Based on the correspondence between the first feature point and the second feature point in the feature point matching unit 302, the model selection unit 306 selects the shape of the projection plane 120 from the plurality of shape models held by the shape model holding unit 303. A projection plane shape model which is a shape model indicating a pattern is selected.

  The parameter calculation unit 307 repeatedly calculates model parameters based on the correspondence between the first feature point and the second feature point in the feature point matching unit 302 so that the captured image approaches the original image.

  Here, the model parameter is a parameter that brings the shape of the projection plane shape model close to the shape of the projection plane 120 and is a parameter that indicates the positional relationship between the projection plane 120 and the image projection apparatus 100. Specifically, the functions f and g of the mathematical formula shown in FIG. 4 express model parameters. The projection surface shape model and the model parameters are collectively referred to as a correction parameter.

  The parameter holding unit 305 holds this correction parameter. That is, the parameter holding unit 305 holds the projection plane shape model selected by the model selection unit 306 and the model parameter calculated by the parameter calculation unit 307.

  Next, the operation of the image projection apparatus 100 will be described.

  FIG. 5 is a flowchart showing an example of the operation of the image projection apparatus 100 according to Embodiment 1 of the present invention.

  First, an original image is input to the image projector 100 (S102).

  FIG. 6A is a diagram illustrating an example of an original image input to the image projection apparatus 100. That is, an original image as shown in FIG.

  The input original image is held in the image memory 104. Further, the correction unit 108 outputs the input original image to the image processing unit 106, and the image processing unit 106 performs a process of converting the input original image into a format suitable for the projection unit 101.

  Returning to FIG. 5, the projection unit 101 projects the image output from the image processing unit 106 onto the projection plane 120 (S <b> 104).

  FIG. 6B is a diagram illustrating an example of an image projected on the projection plane 120. That is, when the projection plane 120 is the projection plane 120a as shown in FIG. 2A, an image as shown in the figure is projected.

  Then, returning to FIG. 5, the following processing is repeated until the input of the original image is completed (S106, S118).

  First, the imaging unit 102 captures an image projected on the projection plane 120 (S108). The captured image is stored in the image memory 103. This captured image is the image shown in FIG. 6B.

  Next, the correction parameter calculation unit 107 receives images from the image memory 103 and the image memory 104 and calculates correction parameters (S110). Details of the process in which the correction parameter calculation unit 107 calculates the correction parameter will be described later.

  Then, the original image of the next frame is input (S112).

  The correcting unit 108 corrects the input original image using the correction parameter to a corrected image (S114). Then, the correction unit 108 outputs the corrected image to the image processing unit 106, and the image processing unit 106 performs processing for converting the corrected image into a format suitable for the projection unit 101. The corrected image is held in the image memory 104.

  Then, the projection unit 101 projects the image output from the image processing unit 106 on the projection plane 120 (S116).

  FIG. 7 is a diagram illustrating an example of a corrected image projected on the projection surface 120. That is, the projection unit 101 projects an image as shown in FIG. As described above, once the correction parameters are calculated, a corrected image obtained by correcting the original image is then projected onto the projection plane 120.

  Then, the imaging unit 102 captures the corrected image projected on the projection surface 120 (S118, S106, S108). Therefore, the correction parameter calculation unit 107 calculates the correction parameter in consideration of what correction is performed on the captured image (S110).

  Specifically, the correction parameter calculation unit 107 calculates a correction parameter from the correction image and the captured image input from the correction unit 108. The correction parameter here is a parameter based on the corrected image. That is, the correction parameter calculation unit 107 calculates correction parameters from a correction image that is an image before being projected onto the projection plane 120 and a captured image that is an image after being projected onto the projection plane 120. For this reason, the correction parameter is a parameter that represents the shape of the projection plane 120 and the positional relationship between the projection plane 120 and the image projection apparatus 100.

  In this way, the process of projecting the image (S108 to S116) is repeated, and when the input of the original image is completed, the operation of the image projection apparatus 100 is completed (S106, S118).

  Here, an example of processing performed by the image projection apparatus 100 will be specifically described by taking as an example a case where the projection plane shape model is a projective deformation model.

  8A and 8B are diagrams specifically illustrating an example of processing performed by the image projection apparatus 100 when the projection surface shape model is a projective deformation model.

  As shown in FIG. 8A, first, the original image Io is input to the image projector 100 (S102 in FIG. 5), and the projection unit 101 projects the original image Io onto the projection plane 120 (S104 in FIG. 5).

  Then, the imaging unit 102 captures the original image Io projected on the projection plane 120 (S108 in FIG. 5). Let the image imaged at this time be the captured image Ic.

Next, the correction parameter calculation unit 107 calculates a projective transformation matrix H ₁ that is a correction parameter of the projective deformation model from the original image Io and the captured image Ic (S110 in FIG. 5).

  Then, the original image Io of the next frame is input (S112 in FIG. 5).

The correcting unit 108 corrects the original image Io input using the correction parameter to the corrected image Im (S114 in FIG. 5). Specifically, the corrected image Im is calculated by Im = H ₁ · Io.

  Next, as illustrated in FIG. 8B, the projection unit 101 projects the correction image Im on the projection plane 120 (S116 in FIG. 5), and the imaging unit 102 captures the correction image Im projected on the projection plane 120. (S118, S106, S108 in FIG. 5). Let the image imaged at this time be the captured image Ic.

Then, the correction parameter calculation unit 107 calculates a projective transformation matrix H ₂ from the corrected image Im and the captured image Ic (S110 in FIG. 5).

  Then, the original image Io of the next frame is input (S112 in FIG. 5).

The correcting unit 108 corrects the input original image Io using the correction parameter to a corrected image Im ′ (S114 in FIG. 5). Specifically, the corrected image Im ′ is
Im ′ = H ₂ · Im = H ₂ · (H ₁ · Io) = (H ₂ · H ₁ ) · Io = H ₂ '· Io
Is calculated by

  In this way, the process of projecting the image (S108 to S116 in FIG. 5) is repeated, and when the input of the original image is completed, the operation of the image projection apparatus 100 is completed (S106 and S118 in FIG. 5).

  Next, details of the process in which the correction parameter calculation unit 107 calculates the correction parameter will be described.

  FIG. 9 is a flowchart illustrating an example of processing (S110 in FIG. 5) in which the correction parameter calculation unit 107 calculates correction parameters.

  First, the feature point extraction unit 301 inputs the captured image held in the image memory 103 and the projection target image held in the image memory 104 (S202).

  Here, the captured image input from the image memory 103 is captured by the imaging unit 102 when the projection target image corrected by the correction unit 108 is projected onto the projection plane 120 through the image processing unit 106 and the projection unit 101. It is an image that was made. That is, the image memories 103 and 104 take images into the feature point extraction unit 301 in consideration of the delay time during which the projection target image is processed by the correction unit 108, the image processing unit 106, the projection unit 101, and the imaging unit 102. It is necessary to match the input timing.

  Note that it is only necessary that the captured image from which the feature points are extracted correspond to the projection target image, and therefore the image memories 103 and 104 do not necessarily need to match the timing of outputting the images. For example, picture numbers are assigned to the projection target images, and feature points between the projection target image and the captured image having the same picture number may be extracted. The timing at which images are output from the image memories 103 and 104 is controlled by the control unit 105.

  The feature point extraction unit 301 performs feature point extraction on each image (projection target image and captured image) input from the image memory 103 and the image memory 104.

  That is, the feature point extraction unit 301 extracts a first feature point (S204). Specifically, the feature point extraction unit 301 extracts a point on the projection target image input from the image memory 104 as a first feature point. Here, when the imaging unit 102 captures an original image (S108 in FIG. 5), a point on the original image held in the image memory 104 is set as a first feature point. When the imaging unit 102 captures a corrected image (S108 in FIG. 5), a point on the corrected image held in the image memory 104 is set as a first feature point.

  Further, the feature point extraction unit 301 extracts a second feature point (S206). Specifically, the feature point extraction unit 301 extracts a point on the captured image input from the image memory 103 as a second feature point.

  FIG. 10A is a diagram illustrating an example of a first feature point, and FIG. 10B is a diagram illustrating an example of a second feature point.

  Specifically, FIG. 10A shows first feature points that are feature points on the original image shown in FIG. 6A. FIG. 10B shows second feature points that are feature points on the captured image shown in FIG. 6B.

  Here, the feature point of the image can be extracted using a Harris operator or a SIFT (Scale Invariant Feature Transform) operator. These operators generally extract intersection points of edges in an image as feature points, and obtain feature point parameters that are parameters of the feature points (such as values indicating position and intensity). These feature point parameters are output to the feature point matching unit 302.

  The feature point matching unit 302 performs feature point matching between images using the feature point parameters of the first feature point and the second feature point extracted by the feature point extraction unit 301 (S208).

  For example, when feature points are extracted using SIFT parameters, a vector representation of the pixel gradient at each feature point is the feature point parameter. For each feature point in the first feature point, the most similar feature point is extracted from the feature points in the second feature point. Here, whether or not the feature points are similar can be determined by the similarity between the feature point parameters. By this process, the correspondence between the feature points of the first feature point and the second feature point can be obtained.

  FIG. 11 is a diagram illustrating an example of a correspondence relationship between the first feature point and the second feature point.

  This figure shows the correspondence between the first feature points shown in FIG. 10A and the second feature points shown in FIG. 10B. In the figure, since the diagram becomes complicated when the correspondences of all feature points are shown, only the correspondences of some feature points are shown. This correspondence is output to the model parameter estimation unit 304.

  Returning to FIG. 9, the following processing is repeated for all shape models (S210, S216).

  First, the model selection unit 306 of the model parameter estimation unit 304 acquires a shape model from the shape model holding unit 303 (S212).

  Next, the parameter calculation unit 307 of the model parameter estimation unit 304 calculates model parameters of the shape model from the correspondence between the feature points of the first feature point and the second feature point (S214). Specifically, the parameter calculation unit 307 calculates model parameters so as to approximate the input original image by deforming the captured image under the shape model acquired by the model selection unit 306. That is, the parameter calculation unit 307 repeatedly calculates model parameters so that the captured image becomes an image obtained by enlarging or reducing the original image. More specifically, the parameter calculation unit 307 corrects the coordinates of the second feature point so that the corrected coordinates of the second feature point approach the coordinates of the point on the original image corresponding to the second feature point. Calculate model parameters.

  For example, when the shape model is a projective deformation model, model parameters (eight parameters) of the projective transformation matrix can be calculated by the least square method or the like.

Specifically, when the shape model is a projective deformation model, the shape model formula shown in FIG. 4 is expressed as (x ′, y ′, 1) ^T = aH (x , Y, 1) ^T Here, a is a predetermined constant, and the projective transformation matrix H has eight parameters as follows. Note that a may not be determined in advance, and may be determined in the calculation process according to the shape of the projection plane 120, for example.

  The eight parameters h1 to h8 of the projective transformation matrix H are calculated from the second feature point (x, y), and (x ′, y ′) is an original corresponding to the second feature point (x, y). It is repeatedly calculated by the method of least squares so as to approach the coordinates of the point on the image.

  Specifically, the parameter calculation unit 307 calculates a parameter with the highest approximation accuracy.

  Here, when obtaining the approximation accuracy, the difference between the corrected coordinates (x ′, y ′) of the second feature point and the coordinates of the point on the original image corresponding to the second feature point (x, y). Should be calculated. In addition, a difference for each pixel may be calculated instead of the difference for each feature point. It can be said that the smaller this difference (the sum of squared differences) is, the higher the approximation accuracy is.

  Here, since the correspondence between the feature points of the first feature point and the second feature point is not necessarily correct, the incorrect correspondence relationship can be removed as an outlier. As a method for removing outliers, a RANSAC (RANdom Sample Consensus) method can be used. For example, since the correspondence relationship 601 shown in FIG. 10 is not correct, the correspondence method 601 is removed by the removal method.

  When the processing (S212, S214) for obtaining the shape model and calculating the model parameters for all the shape models is completed (S210, S216), the model selection unit 306 selects the projection plane shape model (S218). . That is, the model selection unit 306 selects a projection plane shape model from a plurality of shape models held by the shape model holding unit 303 based on the correspondence relationship between the first feature points and the second feature points.

  Specifically, when a suitable shape model is automatically selected from a plurality of shape models held in the shape model holding unit 303, the correspondence between the feature points of the first feature point and the second feature point The model parameters for each shape model are calculated using the relationship, and the shape model with the highest approximation accuracy is selected.

  Here, the approximate accuracy is calculated from the difference between the original image and the converted captured image by converting the captured image using the shape model. It can be said that the smaller this difference (the sum of squared differences) is, the higher the approximation accuracy is. That is, the parameter calculation unit 307 calculates model parameters corresponding to each shape model when the shape model held by the shape model holding unit 303 is a projection plane shape model, and the model selection unit 306 The shape model whose coordinates of the second feature point are corrected by the model parameter corresponding to is closest to the coordinates of the point on the original image corresponding to the second feature point is selected as the projection plane shape model.

  It can also be determined that the greater the number of feature point correspondences applicable to the shape model, the higher the approximation accuracy. At this time, the process of selecting the projection surface shape model may be repeated until the number of feature point correspondences exceeds a predetermined value. That is, the parameter calculation unit 307 calculates model parameters corresponding to the shape model when the shape model held by the shape model holding unit 303 is the projection plane shape model, and the model selection unit 306 corresponds to the shape model. The difference between the coordinate of the second feature point corrected by the model parameter and the coordinate of the point on the original image corresponding to the second feature point is equal to or less than a predetermined first threshold value. A shape model when the number exceeds a predetermined second threshold is selected as the projection plane shape model. Here, the first threshold value and the second threshold value may be any numerical value, and are predetermined by the user.

  As a result, even if the projection plane shape model is selected by mistake due to the small number of feature point correspondences at the beginning, the optimum projection plane shape is obtained when the number of feature point correspondences exceeds a predetermined number. You can reselect the model. Thereafter, model parameters can be calculated under the projection plane shape model.

  Also, the user may designate which projection plane shape model to select from among the shape models held in the shape model holding unit 303. That is, in this case, the model selection unit 306 selects a projection plane shape model based on an external signal.

  Then, the model parameter estimation unit 304 outputs the obtained correction parameter, ie, the projection plane shape model and model parameters, to the parameter holding unit 305 and the correction unit 108 (S220).

  In this way, the process (S110 in FIG. 5) in which the correction parameter calculation unit 107 calculates the correction parameter ends.

  Here, the process of selecting the projection plane shape model or calculating the model parameter may be performed for each frame in the moving image, or may be performed every few seconds, every few minutes, or the like.

  FIG. 12 is a flowchart showing a modification of the process (S110 in FIG. 5) in which the correction parameter calculation unit 107 calculates the correction parameter. In this modification, the model selection unit 306 selects one projection surface shape model from among a plurality of shape models, and the parameter calculation unit 307 determines the shape of the projection surface 120 and the projection surface 120 and the image projection apparatus 100. The model parameters of one selected projection plane shape model are repeatedly calculated according to the change in the positional relationship.

  The flowchart shown in FIG. 9 includes a change of the projection plane shape model. That is, when the shape of the projection plane 120 changes with time, the model parameter estimation unit 304 changes the projection plane shape model to be selected. However, when the positional relationship between the image projection apparatus 100 and the projection plane 120 changes with time, but the shape of the projection plane 120 does not change much with time, the model parameter estimation unit 304 uses the same projection plane shape model. Only the model parameters need be corrected.

  First, it is assumed that the model selection unit 306 has selected one shape model as a projection plane shape model from among a plurality of shape models held in the shape model holding unit 303. Note that the model selection unit 306 may select the projection plane shape model according to the flowchart shown in FIG. 9, or may select the projection plane shape model based on an external signal.

  Then, the feature point extraction unit 301 inputs an image (S302), and extracts the first feature point and the second feature point (S304, S306). Then, the feature point matching unit 302 performs feature point matching between images of the first feature point and the second feature point (S308). Since each of these processes (S302 to S308) is the same as each process (S202 to S208 in FIG. 9) shown in FIG. 9, a detailed description thereof will be omitted.

  Next, the parameter calculation unit 307 calculates model parameters of the selected projection plane shape model (S310). Since the calculation of the model parameter is the same as the process shown in FIG. 9 (S214 in FIG. 9), the detailed description is omitted.

  Then, the model parameter estimation unit 304 outputs the model parameter calculated by the parameter calculation unit 307 to the parameter holding unit 305 and the correction unit 108 (S312).

  As described above, according to the image projecting device 100 in the present embodiment, the correspondence relationship between the feature points between the projection target image and the captured image obtained by projecting the projection target image onto the projection plane and obtaining the correspondence is obtained. The shape distortion of the projection plane 120 is obtained from the relationship. Then, the image projecting apparatus 100 can provide an image without distortion as viewed from the user by performing a reverse correction of the shape distortion on the original image to generate a corrected image and projecting the corrected image. . In other words, a highly reproducible image projection can be provided to the user. The shape distortion at this time is obtained by calculating model parameters of the projection surface shape model.

  Therefore, by using the image projection apparatus 100 of the present invention, when projecting an image onto a projection surface 120 such as a screen using a projector, the screen distortion of the projection surface 120 is obtained using the feature points of the appreciation image. For this reason, it is not necessary for the user to measure the shape distortion of the projection surface 120 by projecting a pattern image or measuring the actual size prior to use. In addition, the image projection apparatus 100 interrupts image viewing even when the shape of the projection surface 120 or the positional relationship between the projection surface 120 and the projector changes during projection of the viewing image. The shape correction can be performed by obtaining the distortion amount. In addition, since the image projection apparatus 100 obtains the shape distortion by calculating the model parameters of the projection surface shape model, the shape correction can be performed stably even when the image in the moving image changes greatly with time. It can be performed.

(Embodiment 2)
Furthermore, by recording the program for realizing the image projection apparatus 100 shown in the first embodiment on a recording medium such as a flexible disk, the processing shown in the first embodiment is performed by an independent computer. It can be easily implemented in the system.

  13A to 13C are diagrams for explaining Embodiment 2 of the present invention. Specifically, FIG. 3 is an explanatory diagram when the image projection apparatus 100 according to the first embodiment is implemented by a computer system using a program recorded on a recording medium such as a flexible disk 902.

  FIG. 13A shows an example of a physical format of the flexible disk 902 which is a recording medium body. Further, FIG. 13B shows an appearance of the case F in which the flexible disk 902 is incorporated as viewed from the front, a cross-sectional structure of the case F, and the flexible disk 902.

  As shown in FIG. 13A, a plurality of tracks Tr are concentrically formed on the surface of the flexible disk 902 from the outer periphery to the inner periphery, and each track is divided into 16 sectors Se at equal angles. Therefore, in the flexible disk 902 storing the program, the program is recorded in an area allocated on the flexible disk 902.

  FIG. 13C shows a configuration for recording and reproducing the program on the flexible disk 902. When the program for realizing the image projection apparatus 100 is recorded on the flexible disk 902, the program is written to the flexible disk 902 from the computer system 901 via the flexible disk drive 903. When the image projection apparatus 100 is realized by a program in the flexible disk 902 and the image projection apparatus 100 is constructed in the computer system 901, the program is read from the flexible disk 902 by the flexible disk drive 903 and transferred to the computer system 901. To do.

  In the above description, the flexible disk 902 is used as the recording medium, but the same can be done using an optical disk. Further, the recording medium is not limited to this, and any recording medium such as an IC card or a ROM cassette capable of recording a program can be similarly implemented.

  It should be noted that the present invention is not limited to the first or second embodiment, and various changes and modifications can be made without departing from the scope of the present invention.

  For example, each functional block in the block diagrams shown in FIGS. 1 and 3 may be realized as an LSI which is an integrated circuit. These may be individually made into one chip, or may be made into one chip so as to include a part or all of them.

  The name used here is LSI, but it may also be called IC, system LSI, super LSI, or ultra LSI depending on the degree of integration.

  Further, the method of circuit integration is not limited to LSI's, and implementation using dedicated circuitry or general purpose processors is also possible. An FPGA (Field Programmable Gate Array) that can be programmed after manufacturing the LSI or a reconfigurable processor that can reconfigure the connection and setting of circuit cells inside the LSI may be used.

  Further, if integrated circuit technology comes out to replace LSI's as a result of the advancement of semiconductor technology or a derivative other technology, it is naturally also possible to carry out function block integration using this technology. Biotechnology can be applied.

  The image projection apparatus according to the present invention may be used when the shape of the projection surface, the positional relationship between the projection surface and the projector changes during the projection of the appreciation image, or the image in the moving image changes significantly with time. Even so, there is an effect that the shape correction can be performed stably without interrupting the viewing of the image, and it is useful as an image projection apparatus that projects an image onto a projection surface such as a screen.

It is a block diagram which shows the structure of the image projector in Embodiment 1 of this invention. It is a figure which shows the example of the projection surface used as the projection object of a projection part. It is a figure which shows the example of the projection surface used as the projection object of a projection part. It is a figure which shows the example of the projection surface used as the projection object of a projection part. It is a block diagram which shows the structure of a correction parameter calculation part. It is a figure which shows an example of the shape model which a shape model holding part hold | maintains. It is a flowchart which shows an example of operation | movement of the image projector in Embodiment 1 of this invention. It is a figure which shows an example of the original image input into an image projector. It is a figure which shows an example of the image projected on a projection surface. It is a figure which shows an example of the corrected image projected on a projection surface. It is a figure which illustrates an example of the process which an image projection apparatus performs in case a projection surface shape model is a projection deformation model. It is a figure which illustrates an example of the process which an image projection apparatus performs in case a projection surface shape model is a projection deformation model. It is a flowchart which shows an example of the process in which a correction parameter calculation part calculates a correction parameter. It is a figure which shows an example of a 1st feature point. It is a figure which shows an example of a 2nd feature point. It is a figure which shows an example of the correspondence of a 1st feature point and a 2nd feature point. It is a flowchart which shows the modification of the process in which a correction parameter calculation part calculates a correction parameter. It is a figure explaining Embodiment 2 of this invention. It is a figure explaining Embodiment 2 of this invention. It is a figure explaining Embodiment 2 of this invention.

Explanation of symbols

DESCRIPTION OF SYMBOLS 100 Image projection apparatus 101 Projection part 102 Imaging part 103, 104 Image memory 105 Control part 106 Image processing part 107 Correction parameter calculation part 108 Correction part 120 Projection surface 301 Feature point extraction part 302 Feature point matching part 303 Shape model holding part 304 Model Parameter estimation unit 305 Parameter holding unit 306 Model selection unit 307 Parameter calculation unit 901 Computer system 902 Flexible disk 903 Flexible disk drive

Claims (11)

An image projection apparatus for projecting an input original image onto a projection plane,
An imaging unit that captures an image obtained by projecting the projection target image, which is the original image of the projection target or a corrected image obtained by correcting the original image, onto the projection plane;
A shape model holding unit for holding a shape model indicating a pattern of a predetermined shape;
Based on the correspondence relationship between the first feature point, which is a point on the projection target image, and the second feature point, which is a point on the captured image, from among the shape models held by the shape model holding unit A model selection unit that selects a projection surface shape model that is a shape model indicating the shape pattern of the projection surface;
A parameter that brings the shape of the projection plane shape model closer to the shape of the projection plane so that the captured image approaches the original image, and a parameter that indicates the positional relationship between the projection plane and the image projection apparatus A parameter calculation unit that repeatedly calculates a certain model parameter based on the latest correspondence relationship between the first feature point and the second feature point;
A correction unit that corrects the original image into the corrected image based on the projection plane shape model and the calculated model parameter each time the model parameter is repeatedly calculated;
A projection unit that projects the original image or the corrected image onto the projection plane as the projection target image. The parameter calculation unit is configured such that the coordinates of the second feature point corrected by the projection plane shape model and the model parameter approach the coordinates of the point on the original image corresponding to the second feature point. The image projection apparatus according to claim 1, wherein the model parameter is repeatedly calculated. The image projection apparatus according to claim 1, wherein the parameter calculation unit repeatedly calculates the model parameter so that the captured image becomes an image obtained by enlarging or reducing the original image. The model selection unit selects one projection plane shape model from among the shape models held by the shape model holding unit,
The parameter calculation unit repeatedly calculates a model parameter of the selected one projection plane shape model according to a change in a shape of the projection plane and a positional relationship between the projection plane and the image projection apparatus. The image projection apparatus described. The parameter calculation unit calculates a model parameter corresponding to the shape model when the shape model held by the shape model holding unit is a projection plane shape model,
The model selection unit is configured so that a difference between the coordinates of the second feature point corrected by the model parameter corresponding to the shape model and the coordinate of the point on the original image corresponding to the second feature point is determined in advance. The image projection apparatus according to claim 1, wherein a shape model when the number of the second feature points that are equal to or less than a predetermined first threshold exceeds a predetermined second threshold is selected as the projection plane shape model. The parameter calculation unit calculates a model parameter corresponding to each shape model when the shape model held by the shape model holding unit is a projection plane shape model,
The model selection unit is a shape model in which the coordinates of the second feature point corrected by the model parameter corresponding to the shape model are closest to the coordinates of the point on the original image corresponding to the second feature point. The image projection apparatus according to claim 1, wherein the projection plane shape model is selected. The image projection device according to claim 1, wherein the model selection unit selects the projection plane shape model based on an external signal. The image projection apparatus according to claim 1, wherein the shape model holding unit holds at least one shape model among a projective deformation model, a cylindrical model, a spherical model, and a Thin Plate Spline model. An image projection method for projecting an input original image onto a projection plane,
An imaging step of capturing an image in which the projection target image, which is the original image of the projection target or the corrected image obtained by correcting the original image, is projected on the projection plane;
Based on the correspondence between the first feature point, which is a point on the projection target image, and the second feature point, which is a point on the captured image, a shape model showing a pattern of a predetermined shape From the model selection step of selecting a projection surface shape model that is a shape model indicating the shape pattern of the projection surface,
A parameter that brings the shape of the projection plane shape model closer to the shape of the projection plane so that the captured image approaches the original image, and a parameter that indicates the positional relationship between the projection plane and the image projection apparatus A parameter calculation step of repeatedly calculating a certain model parameter based on the latest correspondence relationship between the first feature point and the second feature point;
A correction step of correcting the original image into the corrected image based on the projection plane shape model and the calculated model parameter each time the model parameter is repeatedly calculated;
A projection step of projecting the original image or the corrected image onto the projection plane as the projection target image. An integrated circuit that controls an image projection apparatus that projects an input original image onto a projection plane,
An imaging unit that captures an image obtained by projecting the projection target image, which is the original image of the projection target or a corrected image obtained by correcting the original image, onto the projection plane;
A shape model holding unit for holding a shape model indicating a pattern of a predetermined shape;
Based on the correspondence relationship between the first feature point, which is a point on the projection target image, and the second feature point, which is a point on the captured image, from among the shape models held by the shape model holding unit A model selection unit that selects a projection surface shape model that is a shape model indicating the shape pattern of the projection surface;
A parameter that brings the shape of the projection plane shape model closer to the shape of the projection plane so that the captured image approaches the original image, and a parameter that indicates the positional relationship between the projection plane and the image projection apparatus A parameter calculation unit that repeatedly calculates a certain model parameter based on the latest correspondence relationship between the first feature point and the second feature point;
A correction unit that corrects the original image into the corrected image based on the projection plane shape model and the calculated model parameter each time the model parameter is repeatedly calculated;
An integrated circuit comprising: a projection unit that projects the original image or the corrected image onto the projection plane as the projection target image. A program for projecting an input original image onto a projection surface,
An imaging step of capturing an image in which the projection target image, which is the original image of the projection target or the corrected image obtained by correcting the original image, is projected on the projection plane;
Based on the correspondence between the first feature point, which is a point on the projection target image, and the second feature point, which is a point on the captured image, a shape model showing a pattern of a predetermined shape From the model selection step of selecting a projection surface shape model that is a shape model indicating the shape pattern of the projection surface,
A parameter that brings the shape of the projection plane shape model closer to the shape of the projection plane so that the captured image approaches the original image, and a parameter that indicates the positional relationship between the projection plane and the image projection apparatus A parameter calculation step of repeatedly calculating a certain model parameter based on the latest correspondence relationship between the first feature point and the second feature point;
A correction step of correcting the original image into the corrected image based on the projection plane shape model and the calculated model parameter each time the model parameter is repeatedly calculated;
A program for causing a computer to execute a projection step of projecting the original image or the corrected image onto the projection plane as the projection target image.

Images