JP2010124061A - System, method and program for specifying pixel position correspondence relation - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To specify correspondence relation of the pixel of a projected image and a pixel in a projection means so as to project the image without distortions, while keeping the resolution of the image to be a projection object, as it is, when projecting the image on an arbitrary projection plane. <P>SOLUTION: A reference pattern image group generation means 73 determines the luminance value as the function value of a continuous function that changes continuously, accompanying the change of a phase and generates a reference pattern image group which is the image group in which the luminance value is determined, by changing the phase for each image and is the image group in which an initial phase value is continuously changed for each coordinate value along one coordinate axis. A phase calculation means 75 calculates the initial phase value in the luminance change of the respective pixels, in the resolution-converted images from the change in the luminance of the pixels in the plurality of resolution-converted-images after the respective images are photographed and then resolution-converted. An association means 76 specifies the correspondence relation of the pixel in the projection object image and the pixel in the projection means 71 by using the initial phase value. <P>COPYRIGHT: (C)2010,JPO&INPIT

Description

  The present invention relates to a pixel position correspondence specifying system, a pixel position correspondence specifying method, a pixel position correspondence specifying program, an image correction system, and an image correction program that define a correspondence between a pixel of one image and a pixel of another image. In particular, it is based on a pixel position correspondence specifying system, a pixel position correspondence specifying method, a pixel position correspondence specifying program, and a corresponding relation for determining a correspondence between a pixel of an image to be projected and a pixel in a projection unit. The present invention relates to an image correction system and an image correction program for correcting an image.

  When projecting an image on a general plane projection surface (screen), if the positional relationship of the projector with respect to the projection surface is not as designed, the projected image is distorted. A standard projector has a function of correcting this distortion. In this distortion correction function, the image to be projected is subjected to a deformation opposite to the distortion (image deformation) caused by projection from an unexpected position or direction, and the corrected image subjected to the deformation is projected to be distorted. Has been countered. In order to realize this function, the amount of deformation of the image determined by the shape of the projection plane and the positional relationship and orientation between the projection plane and the projector, that is, the amount of movement when the pixels in the projected image are actually projected It is necessary to acquire a map representing (deviation).

  Various methods for acquiring this mapping have been proposed. In particular, when the projection surface is a plane, a technique has been proposed that utilizes the fact that the mapping becomes a projective transformation that can be determined with a small number of parameters. For example, you can manually input the positions of the four corner points of an image, or project a test pattern, and automatically detect markers, image edges, screen edges, etc. from the image captured by the camera. There are methods such as obtaining parameters for projective transformation. In Patent Document 1, one distance between the image projection device and the screen is acquired, and the distance at the four corner points of the screen is estimated from the distance and the screen shape, thereby correcting the screen shape and correcting the projected image. An image projecting method is described in which the screen is deformed and projected according to the screen shape.

  Patent Document 2 describes an image projection system that obtains a high-definition projection image by detecting feature points with high accuracy using a laser pointer or the like, or sequentially updating parameters of tilt, rotation, and shift. ing.

  In Patent Document 3, a predetermined pattern image is generated and projected, and a projected image point corresponding to a captured image point is determined from a captured image obtained by capturing the image, and a corresponding point list is provided. Clustering is performed for each plane in the space using a plane projection transformation matrix obtained from the corresponding points in the point list, a projection area is determined from the captured image points clustered for each plane, and an image is obtained using the projection area and the plane projection transformation matrix. Is described.

  Patent Document 4 describes a method of obtaining a parameter of a curved surface assuming a quadratic curved surface as the shape of the projection surface. Patent Document 4 describes a composite video generation method that displays video from a plurality of projectors without distortion on a quadric curved screen. In the method described in Patent Document 4, a mapping function indicating a correspondence between a point projected on a screen viewed from the coordinate system of the projector and a point viewed from an arbitrary hypothetical viewpoint and an inverse function thereof are obtained. Distortion is corrected by an inverse function of each mapping. When obtaining a mapping function and its inverse function, a test pattern projected from a projector onto a quadric surface screen using a transformation matrix M from a reference coordinate system to a viewpoint coordinate system and a transformation matrix S from a camera coordinate to a reference coordinate system A transformation matrix H from the viewpoint coordinate system to the camera coordinate system is obtained for the predetermined points. Then, using the transformation matrix H, the quadric surface parameter Qv in the viewpoint coordinate system with the assumed viewpoint as the origin is obtained from the quadric surface parameter Q obtained in the camera coordinate system. Then, using the quadratic curved surface parameter Qv, a mapping function indicating a correspondence between a point projected on the screen from the projector i coordinate system and a point viewed from an arbitrary assumed viewpoint is obtained, and the mapping function is used. Find its inverse function.

Japanese Patent No. 3951984 Japanese Patent No. 3709395 JP 2007-142495 A JP 2006-221599 A

  However, with the above technique, for example, it is not possible to correct distortion when an image is projected onto an arbitrary projection surface having complicated unevenness. For example, the method described in Patent Document 1 corrects distortion when an image is projected onto a plane, but is not applicable when the shape of the projection surface is arbitrary. The flat projection apparatus described in Patent Document 3 also targets a projection plane that partially includes a plane. Even in the method described in Patent Document 4, the projection surface is limited to a quadratic curve, and distortion when an image is projected onto a surface of an arbitrary shape cannot be corrected.

  In order to correct image distortion, the correspondence between the position in the image to be projected and the position when it is actually projected must be determined with high precision in pixel units. Since the correspondence between pixels cannot be determined when the shape of the projection surface is arbitrary, distortion correction cannot be performed with high accuracy.

  In addition, in order to project the projection target image, which is an image to be projected to show to the observer, with the resolution of the projection target image, the pixels of the projection target image and the pixels in the projection unit are realized so that the resolution can be realized. It is necessary to identify the correspondence relationship. In addition, after obtaining the correspondence, it may be possible to convert the projection target image by performing interpolation or extrapolation on the projection target image to achieve a desired resolution. Will deteriorate. Therefore, it is preferable to specify the correspondence between the pixels of the projection target image and the pixels in the projection unit so that the resolution of the projection target image can be realized. However, in the technique described in Patent Document 1 and the like, The correspondence cannot be determined.

  Therefore, when projecting an image onto a projection surface of an arbitrary shape, the present invention can project the pixels of the image to be projected and the projection so that the image can be projected without distortion while maintaining the resolution of the image to be projected. It is an object of the present invention to provide a pixel position correspondence specifying system, a pixel position correspondence specifying method, and a pixel position correspondence specifying program for specifying a correspondence with a pixel in a means.

  A pixel position correspondence specifying system according to the present invention includes a projecting unit that projects an image on a projection surface, a photographing unit that captures an image projected on the projection surface, and a function of a continuous function that continuously changes as the phase changes. An image group in which a luminance value is defined as a value and the luminance value is determined by changing the phase for each image, and the initial phase value is continuously changed for each coordinate value along one coordinate axis. A reference pattern image group generation unit that generates a certain reference pattern image group, an imaging control unit that causes an image belonging to the reference pattern image group to be projected onto the projection unit, and as a result, causes the imaging unit to capture an image projected on the projection plane; Of the photographed image photographed by the photographing means, a portion corresponding to the projection area of the projection target image on the projection plane is cut out, and the cut-out image is enlarged or reduced to convert the resolution of the image. Resolution conversion means for generating a resolution-converted image, phase calculation means for calculating an initial phase value at a luminance change of each pixel in the resolution-converted image from a luminance change of the pixels in the plurality of resolution-converted images, and a reference pattern image group The generation means performs a reverse calculation for calculating the initial phase value from the coordinate values with respect to the initial phase value calculated by the phase calculation means, associates the coordinate value obtained by the reverse calculation with the pixels of the resolution conversion image, and projects the projection target image. A reference pattern image group related to the first coordinate axis, wherein the reference pattern image generation means changes the initial phase value along the first coordinate axis; Generating a reference pattern image group related to the second coordinate axis in which the initial phase value is changed along the second coordinate axis, and the imaging control means generates the first coordinate The reference pattern image group relating to the second coordinate axis is projected on the projection means, and the image projected on the projection plane as a result is photographed on the imaging means, and the reference pattern image group relating to the second coordinate axis is projected on the projection means, and as a result projected on the projection plane. A plurality of resolution conversions converted from a plurality of photographed images taken by the photographing means when the projection means projects each image of the reference pattern image group relating to the first coordinate axis. The initial phase value at the luminance change of each pixel in the resolution converted image is calculated from the luminance change of the pixel in the image, and the imaging unit takes an image when each image of the reference pattern image group related to the second coordinate axis is projected. The luminance change of each pixel in the resolution converted image is obtained from the luminance change of the pixels in the plurality of resolution converted images converted from the plurality of captured images. The initial phase value in the conversion is calculated, and the associating means is initialized from the coordinates of the first coordinate axis with respect to the initial phase value calculated from the resolution conversion image of the captured image obtained by photographing the reference pattern image group related to the first coordinate axis. Calculate the coordinates of the first coordinate axis by performing the reverse calculation of the calculation for obtaining the phase value, associate the coordinates of the first coordinate axis with the pixels of the resolution converted image, and convert the resolution of the captured image obtained by capturing the reference pattern image group related to the second coordinate axis For the initial phase value calculated from the image, the calculation for obtaining the initial phase value from the coordinates of the second coordinate axis is performed to calculate the coordinates of the second coordinate axis, and the coordinates of the second coordinate axis are calculated in the pixels of the resolution conversion image. By associating, the correspondence between the pixels of the resolution converted image and the pixels of the image projected by the projection unit is specified, and the projection target image is determined using the correspondence between the pixels of the resolution converted image and the coordinates. Features to identify the correspondence between the pixels in the pixel with the projection means in.

  Further, the pixel position correspondence specifying system according to the present invention determines a luminance value as a function value of a projecting unit that projects an image on a projection plane, and a continuous function that continuously changes with a change in phase. A reference pattern image group that generates a reference pattern image group that is an image group in which the luminance value is determined by changing the initial phase value and the initial phase value is continuously changed for each coordinate value along one coordinate axis Generating means, reference pattern projection control means for projecting an image belonging to the reference pattern image group onto the projection means, and a plurality of captured images obtained by photographing each image of the reference pattern image group projected on the projection surface, respectively A portion corresponding to the projection area of the projection target image on the projection plane is cut out, and the cut-out image is enlarged or reduced to generate a resolution-converted image obtained by converting the resolution of the image. Resolution conversion means, phase calculation means for calculating an initial phase value at each pixel brightness change in the resolution conversion image, and reference pattern image group generation means from the coordinate value Back calculation for obtaining the initial phase value is performed on the initial phase value calculated by the phase calculation means, the coordinate value obtained by the reverse calculation is associated with the pixel of the resolution conversion image, and the pixel in the projection target image and the projection means A correlation unit that identifies a correspondence relationship with the pixel, and the reference pattern image generation unit includes a reference pattern image group related to the first coordinate axis in which the initial phase value is changed along the first coordinate axis, and the second coordinate axis. And generating a reference pattern image group related to the second coordinate axis with the initial phase value changed, and the reference pattern projection control means generates a reference pattern related to the first coordinate axis. The pattern image group is projected onto the projecting means, the reference pattern image group related to the second coordinate axis is projected onto the projecting means, and the phase calculation means is a plurality of images obtained by photographing each image of the reference pattern image group related to the first coordinate axis. The initial phase value at the luminance change of each pixel in the resolution-converted image is calculated from the luminance change of the pixels in the plurality of resolution-converted images converted from the captured image, and each image of the reference pattern image group related to the second coordinate axis is captured. The initial phase value at the luminance change of each pixel in the resolution converted image is calculated from the luminance change of the pixels in the plurality of resolution converted images converted from the plurality of captured images obtained in the above, With respect to the initial phase value calculated from the resolution conversion image of the captured image obtained by capturing the reference pattern image group related to one coordinate axis, the first coordinate axis The calculation of obtaining the initial phase value from the coordinates is performed to calculate the coordinates of the first coordinate axis, the coordinates of the first coordinate axis are associated with the pixels of the resolution conversion image, and the reference pattern image group related to the second coordinate axis is photographed. The pixel of the resolution-converted image is calculated by calculating the coordinates of the second coordinate axis by performing a reverse calculation of the initial phase value calculated from the coordinates of the second coordinate axis with respect to the initial phase value calculated from the resolution-converted image of the captured image. By associating the coordinates of the second coordinate axis with the pixels of the resolution converted image and the pixels of the image projected by the projection means, the correspondence between the pixels of the resolution converted image and the coordinates is projected. It is characterized in that the correspondence between the pixel in the target image and the pixel in the projection means is specified.

  Further, the pixel position correspondence specifying method according to the present invention is a group of images in which a luminance value is determined as a function value of a continuous function that continuously changes with a change in phase, and the luminance value is determined by changing the phase for each image. A first reference pattern image group generation step for generating a reference pattern image group related to the first coordinate axis, which is an image group in which the initial phase value is continuously changed for each coordinate value along the first coordinate axis; A first imaging control step of projecting an image belonging to the reference pattern image group related to one coordinate axis onto the projection means, and causing the imaging means to image the image projected on the projection plane, and each image of the reference pattern image group related to the first coordinate axis A portion corresponding to the projection area of the projection target image on the projection plane is cut out from each of a plurality of captured images taken by the photographing unit when the projection unit projects the image, and the cut-out image is enlarged. Alternatively, by reducing the first resolution conversion step of generating a resolution-converted image obtained by converting the resolution of the image, and the luminance change of the pixels in the plurality of resolution-converted images, the luminance change of each pixel in the resolution-converted image The first phase calculation step for calculating the initial phase value and the reverse calculation of the calculation for obtaining the initial phase value from the coordinates of the first coordinate axis in the first reference pattern image group generation step are performed on the initial phase value calculated in the first phase calculation step. A first association step for associating the coordinates of the first coordinate axis obtained by back calculation and the pixels of the resolution-converted image, and determining a luminance value as a function value of a continuous function that continuously changes with a phase change An image group in which the luminance value is determined by changing the phase for each image, and the initial phase value is continuously changed for each coordinate value along the second coordinate axis. A second reference pattern image group generation step for generating a reference pattern image group related to the second coordinate axis, which is an image group, and an image belonging to the reference pattern image group related to the second coordinate axis to be projected onto the projection means, and as a result projected onto the projection plane A second photographing control step for causing the photographing means to photograph the captured image, and projection on the projection plane from a plurality of photographed images photographed by the photographing means when the projection means projects each image of the reference pattern image group related to the second coordinate axis. A second resolution conversion step of cutting out a portion corresponding to the projection area of the target image, enlarging or reducing the cut-out image to generate a resolution-converted image obtained by converting the resolution of the image, and pixels in the plurality of resolution-converted images Second phase for calculating an initial phase value at the luminance change of each pixel in the resolution conversion image from the luminance change of The back calculation of the initial phase value calculated from the coordinates of the second coordinate axis in the phase calculation step and the second reference pattern image group generation step is performed on the initial phase value calculated in the second phase calculation step, and is obtained by back calculation. The second correspondence step for associating the coordinates of the second coordinate axis with the pixels of the resolution conversion image, and the correspondence between the pixels of the resolution conversion image and the coordinates, and the correspondence between the pixels in the projection target image and the pixels in the projection means And a pixel matching step to be specified.

  Further, the pixel position correspondence specifying method according to the present invention is a group of images in which a luminance value is determined as a function value of a continuous function that continuously changes with a change in phase, and the luminance value is determined by changing the phase for each image. A first reference pattern image group generation step for generating a reference pattern image group related to the first coordinate axis, which is an image group in which the initial phase value is continuously changed for each coordinate value along the first coordinate axis; A first reference pattern projection control step for projecting an image belonging to the reference pattern image group related to one coordinate axis to the projection means, and a plurality of images obtained by photographing each image of the reference pattern image group related to the first coordinate axis projected onto the projection plane. A portion corresponding to the projection area of the projection target image on the projection plane is cut out from each captured image, and the resolution of the image is reduced by enlarging or reducing the cut out image. A first resolution converting step for generating a converted resolution converted image, and a first phase calculating step for calculating an initial phase value at a luminance change of each pixel in the resolution converted image from a luminance change of the pixels in the plurality of resolution converted images. The first coordinate axis obtained by performing the reverse calculation of the initial phase value calculated in the first phase calculation step by performing the reverse calculation of calculating the initial phase value from the coordinates of the first coordinate axis in the first reference pattern image group generation step. Luminance values are determined by changing the phase for each image by determining a luminance value as a function value of a continuous function that continuously changes with a change in phase. A reference pattern image related to the second coordinate axis, which is an image group in which the initial phase value is continuously changed for each coordinate value along the second coordinate axis. A second reference pattern image group generation step for generating a group, a second reference pattern projection control step for projecting an image belonging to the reference pattern image group for the second coordinate axis to the projection means, and a second coordinate axis projected on the projection plane A portion corresponding to the projection area of the projection target image on the projection plane is cut out from a plurality of shot images obtained by shooting each image of the reference pattern image group, and the cut-out image is enlarged or reduced to thereby A second resolution conversion step for generating a resolution-converted image obtained by converting the resolution; and a second phase for calculating an initial phase value at a luminance change of each pixel in the resolution-converted image from a luminance change of the pixels in the plurality of resolution-converted images. The initial phase value is calculated from the coordinates of the second coordinate axis in the calculation step and the second reference pattern image group generation step. A second associating step of associating the coordinates of the second coordinate axis obtained by the inverse calculation with the pixels of the resolution converted image, performing reverse calculation of the calculation on the initial phase value calculated in the second phase calculating step; And a pixel association step of specifying a correspondence relationship between the pixel in the projection target image and the pixel in the projection unit using the correspondence relationship between the pixel and the coordinate.

  Also, the pixel position correspondence specifying program according to the present invention is a pixel position correspondence specifying program mounted on a computer that includes a projecting unit that projects an image on a projection surface and a photographing unit that captures an image projected on the projection surface. An image group in which a luminance value is determined as a function value of a continuous function that continuously changes as the phase changes, and the luminance value is determined by changing the phase for each image, and the initial phase A first reference pattern image group generation process for generating a reference pattern image group for the first coordinate axis, which is an image group in which values are continuously changed for each coordinate value along the first coordinate axis, and a reference pattern image group for the first coordinate axis A first patterning control process for projecting an image belonging to the projection unit, and causing the photographing unit to capture an image projected as a result on the projection plane, and a reference pattern image group related to the first coordinate axis A portion corresponding to the projection area of the projection target image on the projection plane is cut out from each of the plurality of photographed images taken by the photographing unit when each image is projected by the projecting unit, and the image is obtained by enlarging or reducing the cut out image. First resolution conversion processing for generating a resolution-converted image obtained by converting the resolution of the first phase, and calculating the initial phase value at the luminance change of each pixel in the resolution-converted image from the luminance change of the pixels in the plurality of resolution-converted images In the calculation process and the first reference pattern image group generation process, the calculation for obtaining the initial phase value from the coordinates of the first coordinate axis is performed on the initial phase value calculated in the first phase calculation process, and the first obtained by the reverse calculation is performed. A first association process for associating the coordinates of the coordinate axes with the pixels of the resolution-converted image, and a function value of a continuous function that continuously changes as the phase changes In the image group in which the luminance value is determined and the luminance value is determined by changing the phase for each image, the initial phase value is continuously changed for each coordinate value along the second coordinate axis. Second reference pattern image group generation processing for generating a reference pattern image group related to a certain second coordinate axis, and an image belonging to the reference pattern image group related to the second coordinate axis is projected onto the projection means, and as a result, the image projected onto the projection plane is photographed A second shooting control process to be shot by the means, and each of the images of the reference pattern image group related to the second coordinate axis is projected from the plurality of shot images shot by the shooting means to the projection area of the projection target image on the projection plane. A second resolution conversion process for generating a resolution-converted image obtained by converting a corresponding portion and cutting out the cut-out image to convert the resolution of the image. From the luminance change of the pixel in the image conversion image, the second phase calculation processing for calculating the initial phase value at the luminance change of each pixel in the resolution conversion image, and from the coordinates of the second coordinate axis in the second reference pattern image group generation processing A second associating process for performing a reverse calculation of the calculation for obtaining the initial phase value on the initial phase value calculated in the second phase calculating process, and associating the coordinates of the second coordinate axis obtained by the reverse calculation with the pixels of the resolution conversion image; And it is characterized by performing the pixel matching process which specifies the correspondence of the pixel in a projection target image, and the pixel in a projection means using the correspondence of the pixel of a resolution conversion image, and a coordinate.

  Further, the pixel position correspondence specifying program according to the present invention is a pixel position correspondence specifying program mounted on a computer having a projection unit for projecting an image onto a projection plane, and is continuously transmitted to the computer as the phase changes. A group of images in which a luminance value is determined as a function value of a continuous function that changes, and the luminance value is determined by changing the phase for each image, and the initial phase value is continuously set for each coordinate value along the first coordinate axis. First reference pattern image group generation processing for generating a reference pattern image group relating to the first coordinate axis, which is a changed image group, and first reference pattern projection control for projecting an image belonging to the reference pattern image group relating to the first coordinate axis to the projection means. Processing, a plurality of captured images obtained by photographing each image of the reference pattern image group related to the first coordinate axis projected on the projection plane are projected on the projection plane. A first resolution conversion process for generating a resolution-converted image in which the resolution of the image is converted by cutting out a portion corresponding to the projection area of the target image and enlarging or reducing the cut-out image. The first phase calculation process for calculating the initial phase value at the luminance change of each pixel in the resolution-converted image from the luminance change, and the calculation for obtaining the initial phase value from the coordinates of the first coordinate axis in the first reference pattern image group generation process Reverse calculation is performed on the initial phase value calculated in the first phase calculation process, and the first correlation process for associating the coordinates of the first coordinate axis obtained by the reverse calculation with the pixels of the resolution conversion image is continuous with the phase change. A group of images in which a luminance value is determined as a function value of a continuous function that changes and a luminance value is determined by changing a phase for each image, and an initial phase value is a second value Second reference pattern image group generation processing for generating a reference pattern image group related to the second coordinate axis, which is an image group continuously changed for each coordinate value along the standard axis, and an image belonging to the reference pattern image group related to the second coordinate axis A second reference pattern projection control process for projecting the image on the projection means, and a plurality of captured images obtained by capturing each image of the reference pattern image group related to the second coordinate axis projected on the projection plane, A second resolution conversion process for generating a resolution-converted image obtained by cutting out a portion corresponding to the projection region and enlarging or reducing the cut-out image to convert the resolution of the image. From the luminance change of the pixels in the plurality of resolution-converted images A second phase calculation process for calculating an initial phase value at a luminance change of each pixel in the resolution-converted image, a second reference pattern Reverse calculation of the initial phase value calculated from the coordinates of the second coordinate axis in the image group generation processing is performed on the initial phase value calculated in the second phase calculation processing, and the coordinates of the second coordinate axis obtained by the reverse calculation and resolution conversion A second associating process for associating the pixels of the image, and a pixel associating process for specifying the correspondence between the pixels in the projection target image and the pixels in the projection unit using the correspondence between the pixels of the resolution-converted image and the coordinates. It is made to perform.

  According to the present invention, when projecting an image on a projection surface of an arbitrary shape, the image pixel to be projected and the projection so that the image can be projected without distortion while maintaining the resolution of the image to be projected. The correspondence relationship with the pixel in the means can be specified.

  Hereinafter, embodiments of the present invention will be described with reference to the drawings. First, after describing a reference embodiment related to the present invention, an embodiment of the present invention will be described.

Reference Embodiment 1
FIG. 1 is a block diagram showing an example of a pixel position correspondence specifying system according to a first reference embodiment related to the present invention. The pixel position correspondence specifying system of the first reference embodiment specifies the correspondence between the position of a pixel of a projection target image to be projected and the position of the pixel in the photographing unit 1. The projection target image is an image projected onto a projection surface (screen) 7 having an arbitrary shape. The pixel position correspondence specifying system according to the present embodiment includes a projection unit 1 that projects an image onto a projection surface 7 having an arbitrary shape, an imaging unit 2 that captures an image projected on the projection surface 7, and a projection. And an information processing apparatus 3 that performs a process of associating a pixel of the target image with a pixel in the projection unit 1. The projection unit 1 is realized by a projector, for example, and the photographing unit 2 is realized by a camera, for example.

  The projection unit 1 projects the reference pattern image group onto the projection plane under the control of the information processing device 3. The reference pattern image group will be described later. Since the projection plane 7 is a plane having an arbitrary shape, the image projected on the projection plane 7 is distorted. The photographing unit 2 photographs a reference pattern image group projected on the projection surface 7 from the same direction as the direction in which the image observer 8 views the projection surface 7. Hereinafter, an image obtained by this photographing is referred to as a photographed image. The information processing device 3 uses the reference pattern image group and each captured image to identify the correspondence between the pixel of the projection target image and the pixel of the captured image obtained by capturing the image actually projected on the projection plane. Furthermore, the correspondence relationship between the pixels of the projection target image and the pixels in the projection unit 1 is specified.

  In this reference embodiment, an example will be described in which the projecting unit 1 projects the reference pattern image group one image at a time on the projection plane, and the image capturing unit 2 captures the projected image one image at a time. However, the projection unit 1 may project a plurality of images simultaneously. The case where the projection unit 1 projects a plurality of images simultaneously will be described later.

  When the correspondence between the pixels of the projection target image and the pixels in the projection unit 1 is determined, the projection target image projected from the projection unit 1 to be shown to the observer 8 is corrected based on the correspondence, and the observer An image that is not distorted into 8 can be observed. In this reference embodiment, the correspondence between pixels of an image used for performing such correction is determined.

  Next, the reference pattern image group will be described. The reference pattern image group is an image group in which a luminance value is determined as a function value of a continuous function that continuously changes with a phase change, and the luminance value is determined by changing the phase for each image, and an initial phase value Is a group of images that are continuously changed for each coordinate value along one coordinate axis. The function that determines the luminance value only needs to be a function that continuously changes as the phase changes. In this case, a function (such as a sine function) in which the luminance value changes as the phase changes becomes a sine wave is used. Take as an example.

  FIG. 2 shows an example of a function that represents a luminance value that continuously changes as the phase changes. Let us denote such a function as f. In the function illustrated in FIG. 2, initial phase values such as a, b, and c are determined according to the coordinate values. Here, the x-axis representing the pixel position of the image will be described as an example. The initial phase value corresponding to a certain x coordinate is a, the initial phase value corresponding to another x coordinate is b, the initial phase value corresponding to another x coordinate is c,. An initial phase value is determined according to each value. For example, the luminance value of each pixel of the x coordinate corresponding to the initial phase value a is f (a), the luminance value of each coordinate of the x coordinate corresponding to the initial phase value b is f (b), and the initial phase value c Let f (c) be the luminance value of each pixel of the x coordinate corresponding to. By determining the initial phase value corresponding to each x coordinate in this way, the first (0th) image in the reference pattern image group is obtained. The initial phase value is determined according to the x coordinate, and the luminance value is determined by the function f of the sine wave as illustrated in FIG. 2 according to the initial phase value. A changing image is obtained.

  In addition, an image in which the luminance is determined by changing the phase by the same phase amount from the initial phase value corresponding to each x coordinate is another image in the reference pattern image group. FIG. 3 is an explanatory diagram illustrating a luminance value when the phase is shifted by d from the initial phase value illustrated in FIG. For example, the luminance value of the x coordinate corresponding to the initial phase value a is f (a + d), the luminance value of the x coordinate corresponding to the initial phase value b is f (b + d), and the x coordinate corresponding to the initial phase value c is Let the luminance value be f (c + d). Similarly, a single image can be obtained by determining a luminance value corresponding to each x coordinate. A plurality of types of similar images can be obtained by changing the phase value (d above) that is changed from the initial phase value.

  In any of these images belonging to the reference pattern image group, the luminance continuously changes as the coordinate value along one coordinate axis (the x coordinate value in the above example) changes. When the images belonging to the reference pattern image group are compared, it is observed that the entire image moves along the coordinate axis as the phase amount d changed from the initial phase value increases. For example, when each image is compared in a reference pattern image group generated by setting an initial phase value for each x coordinate, the entire image is observed as if it moved in the x-axis direction. Further, for example, when the images are compared in the reference pattern image group generated by determining the initial phase value for each y coordinate, it is observed that the entire image has moved in the y-axis direction.

  Further, in each image belonging to the reference pattern image group, when the change in luminance is observed focusing on the individual coordinates along the coordinate axis, the brightness changes with the change in the phase amount d changed from the initial phase value. To do.

  The information processing device 3 generates a reference pattern image group with respect to two coordinate axes. Hereinafter, a case where the information processing apparatus 3 generates a reference pattern image group related to the x axis and a reference pattern image group related to the y axis will be described as an example. However, the two coordinate axes are not limited to the x axis and the y axis. The two coordinate axes may be two axes that are not parallel. The direction of the axis is not limited to the horizontal direction or the vertical direction, and may be any direction.

  The projecting unit 1 projects each image belonging to such a reference pattern image group onto the projection plane 7, and the image capturing unit 2 captures each image. At this time, the positional relationship among the projection unit 1, the photographing unit 2, and the projection plane 7 is kept constant. When attention is paid to a pixel at the same position in each captured image, the luminance of the pixel changes for each image, and the initial phase value in the luminance change of the pixel can be obtained from the change of the image. The information processing device 3 obtains an initial phase value in the luminance change of each pixel from the luminance change of the pixel between the captured images. Then, the information processing device 3 specifies which pixel of the coordinate value in the projected reference pattern image group corresponds to the pixel in the captured image captured by the capturing unit 2, and further, the pixel of the projection target image and the projecting unit The correspondence relationship with the pixel in 1 is specified.

  The information processing apparatus 3 creates a reference pattern image group focusing on the x-axis direction, projects each image on the projection unit 1, and causes the photographing unit 2 to photograph the image. Then, the information processing apparatus 3 identifies which x coordinate pixel in the reference pattern image group corresponds to the pixel in the captured image. Similarly, the information processing apparatus 3 creates a reference pattern image group focusing on the y-axis direction, causes each image to be projected onto the projecting unit 1, and causes the image capturing unit 2 to capture the image. Then, the information processing device 3 identifies which y coordinate pixel in the reference pattern image group the pixel in the captured image corresponds to. Therefore, the correspondence between the pixel in the image captured by the imaging unit 2 from the same direction as the direction of the observer 8 and the coordinate (x, y) of the pixel in the projected image is determined, and the pixel of the projection target image and the projection unit The correspondence with the pixel in 1 is specified.

  FIG. 4 is a block diagram illustrating a configuration example of the information processing apparatus 3. The information processing apparatus 3 includes a pattern phase shift unit 4, an imaging control unit 5, a phase calculation unit 6, and an association unit 9.

  The pattern phase shift means 4 creates a reference pattern image group for each of the x coordinate and the y coordinate. When creating a reference pattern image group for the x coordinate, the pattern phase shift means 4 obtains an initial phase value corresponding to each value of the x coordinate, and determines a luminance value at each x coordinate from the initial phase value, thereby generating a reference value. The first (0th) image of the pattern image group is generated. Further, the pattern phase shift means 4 obtains a phase value corresponding to each x coordinate by changing the phase amount by the same phase amount d from the initial phase value corresponding to each x coordinate, and each x value is calculated from the phase value. By determining the luminance value at the coordinates, the next image to which the reference pattern image group belongs is generated. The pattern phase shift means 4 increases the amount of phase change from the initial phase value, and similarly generates images belonging to the reference pattern image group.

  The pattern phase shift means 4 similarly generates the reference pattern image group for the y coordinate.

  The imaging control means 5 projects the individual images belonging to the respective reference pattern image groups of the x coordinate and the y coordinate onto the projection means 1 (see FIG. 1), and causes the imaging means 2 to project the image projected on the projection plane 7. Let them shoot. In the present embodiment, the imaging control unit 5 projects images belonging to the reference pattern image group onto the projection unit 1 one image at a time, and causes the imaging unit 2 to capture the projected images one image at a time.

  The phase calculating means 6 obtains an initial phase value in the luminance change between the captured images based on the luminance change between the captured images obtained by capturing the images belonging to the reference pattern image group of the x coordinate. Similarly, the phase calculation means 6 obtains an initial phase value for a luminance change between the captured images of each captured image obtained by capturing each image belonging to the reference pattern image group of the y coordinate.

  The associating means 9 specifies which x coordinate in the reference pattern image group the pixel in the captured image corresponds to from the initial phase value obtained from the captured image of the reference pattern image group in the x coordinate. Similarly, the y-coordinate in the reference pattern image group corresponding to the pixel in the captured image is specified from the initial phase value obtained from the captured image of the reference pattern image group in the y coordinate. The association unit 9 further specifies the correspondence between the pixels of the projection target image and the pixels in the projection unit 1.

  The pattern phase shift unit 4, the imaging control unit 5, the phase calculation unit 6, and the association unit 9 are realized by, for example, a CPU that operates according to a program (pixel position correspondence relationship specifying program). The program is stored, for example, in a program storage device (not shown) included in the information processing apparatus 3, and the CPU reads the program, and according to the program, the pattern phase shift means 4, the imaging control means 5, the phase calculation means 6, And it may operate as the association means 9. In addition, the pattern phase shift unit 4, the imaging control unit 5, the phase calculation unit 6, and the association unit 9 may be separate hardware.

Next, the operation will be described.
FIG. 5 is a flowchart illustrating an example of processing progress of the first reference embodiment. The information processing apparatus 3 determines which pixel in the captured image obtained by capturing the image projected on the projection plane 7 corresponds to which x coordinate in the projected image (reference pattern image group) (step S1). Subsequently, the information processing apparatus 3 determines which pixel in the captured image obtained by capturing the image projected on the projection plane 7 corresponds to which y coordinate in the projected image (step S2). By executing steps S1 and S2, it is determined which coordinate value the pixel of the captured image is in the projected image. The information processing apparatus 3 determines the correspondence between the position of the pixel of the projection target image and the position of the pixel in the projection unit 1 based on the correspondence determined in steps S1 and S2 (step S3).

  The processes in steps S1 and S2 are the same processes except that the object for determining the correspondence relationship with the pixels in the captured image is the x coordinate or the y coordinate. The details of the process for determining the correspondence between the pixels of the captured image and the coordinates will be described below using step S1 for determining the correspondence for the x coordinate as an example. FIG. 6 is a flowchart showing an example of the process of the association (step S1) regarding the x coordinate.

In step S1, the pattern phase shift unit 4 first generates a reference pattern image group (step S11). When generating a reference pattern image group for the x coordinate, the pattern phase shift means 4 obtains an initial phase value corresponding to the x coordinate. The initial phase value is a function having the coordinate value as a variable, and may be obtained as a function value of a function that continuously changes as the coordinate value is converted. Hereinafter, a function for obtaining the initial phase value corresponding to the coordinate value is represented by s ₀ . For example, the pattern phase shift means 4 may obtain an initial phase value corresponding to each coordinate value by a function represented by the following expression (1).

s ₀ (x) = T / (w−1 + 2 · m) · (x−m) Equation (1)

  In Expression (1), x is a value of an x coordinate of a pixel of the reference pattern image group. w is the width (number of pixels) of the image that can be projected by the projection means 1. T is a period in a function f for obtaining a luminance value using the phase as a variable. In this example, the case where the luminance change by the function f becomes a sine wave is taken as an example. Therefore, in this example, T = 2π. In consideration of phase calculation errors, m is a constant that is determined so that no wraparound occurs between a pixel with x = 0 and a pixel with x = w−1, and is called a margin constant. The margin constant may be determined as a value of about 1% of w, for example.

  Expression (1) is an example of a function for obtaining an initial phase value corresponding to the x coordinate, and the initial phase value may be calculated by a function other than Expression (1).

For each x coordinate, the pattern phase shift means 4 obtains a luminance value corresponding to the initial phase value s ₀ (x) obtained from the x coordinate value by the function f. That is, for each x coordinate, the luminance value is determined using the function of the following formula (2).

I (x) = f (s ₀ (x)) Equation (2)

  As a result, an image defining a luminance value for each x coordinate is obtained. This image is the first (0th) image in the reference pattern image group. I (x) on the left side in Equation (2) means the luminance of the pixel according to the value of the x coordinate. As described above, the function f is a function that continuously changes with a change in phase. In this example, the function f is a function in which the luminance change is a sine wave.

Further, the pattern phase shift means 4 obtains a phase obtained by adding a common phase value to each initial phase value obtained for each x coordinate. That is, the phase corresponding to each x-coordinate value is calculated for each x-coordinate with the phase amount to be added being d (i). Then, the pattern phase shift unit 4 calculates the luminance value of each x coordinate by substituting the obtained phase into a function f for obtaining the luminance value. As a result, an image defining a luminance value for each x coordinate is obtained. The pattern phase shift unit 4 changes the phase value to be added and similarly generates an image. Assuming that the luminance obtained by substituting the phase value obtained by adding d (i) to the initial phase value into the function f is I _i (x), I _i (x) is expressed by the following equation (3). Can be expressed as:

I _i (x) = f (s ₀ (x) + d (i)) Equation (3)

  The value of d (i) may be determined in advance, for example, as in the following expression (4).

d (i) = (T / N _s ) × i Equation (4)

N _s is the number of images to be obtained as the reference pattern image group. i is 1, 2,..., N _s −1. For example, when N _s = 4, d (i) = T / 4, T / 2, and 3T / 4. In this example, since T = 2π, d (i) = π / 2, π, and 3π / T. That is, in addition to the first image whose luminance is obtained as f (s ₀ (x)), an image whose luminance is obtained as f (s ₀ (x) + π / 2), and f (s ₀ (x) + π). An image for which luminance is obtained and an image for which luminance is obtained as f (s ₀ (x) + 3π / 2) are generated. As a result, a reference pattern image group including four images from the 0th to the third is obtained. i in d (i) represents this order.

  However, Expression (4) is an example of a calculation formula for obtaining the phase amount to be added to the initial phase value, and the phase amount to be added may be determined by another method. In the above example, the case of generating a reference pattern image group including four images is illustrated, but the number of images to be obtained is not limited to four, and may be five or more. Three may be sufficient. When a function whose luminance change is a sine wave is used as the function f, at least three images may be generated.

  After the generation of the reference pattern image group, the imaging control unit 5 projects each image belonging to the reference pattern image group toward the projection plane 7 on the projection unit 1 one image at a time. Then, the photographing control unit 5 causes the photographing unit 2 to photograph each image projected on the projection surface 7 one by one, and receives the photographed image from the photographing unit 2 (step S12).

  After step S12, the phase calculation means 6 obtains an initial phase value for the change on a pixel-by-pixel basis based on the change in luminance between the captured images (step S13). Even if the projected image is distorted due to the projection surface 7 having an arbitrary shape, the position at which a pixel having a certain coordinate value is projected does not change even when the projected image is switched. Since the luminance is changed for each individual x coordinate value, the luminance of the pixel at the same location in each captured image also changes in the same way as the luminance of the image at the same location in each image belonging to the reference pattern image group. FIG. 7 is an explanatory diagram illustrating an example of a luminance change of the same pixel in a plurality of captured images. In step S12, the phase calculation means 6 obtains an initial phase in a change in luminance as exemplified in FIG.

The phase calculation means 6 should just obtain | require the initial phase value in the luminance change of the pixel of a picked-up image as shown below. The luminance I _j of the pixel corresponding to the common position in each captured image and the pixel in the j-th captured image is expressed as the following Expression (5).

I _j = A · f (φ + d (j)) + B Formula (5)

  However, d (j) represents the phase amount added to the initial phase value in the j-th captured image. For the 0th captured image (that is, the captured image of the first image in the reference pattern image group), d (0) = 0. For j = 1 and later, let d (j) be the same value as d (i) used when generating the reference pattern image group. A in the equation (5) is an amplitude in a change in luminance value that changes for each captured image. Further, B in the equation (5) is the luminance of the background.

For pixels corresponding to the same position, the values of I _j and d (j) for each captured image are known. The phase calculation means 6 substitutes I _j , d (j) for each photographed image in equation (5), and calculates φ, which is an unknown value. There are A, B, and φ as unknown values, and φ may be calculated by solving simultaneous equations in which I _j and d (j) are substituted into Equation (5) for each captured image.

If more than the number necessary for deriving the simultaneous equations is obtained by substituting I _j , d (j) into Equation (5) for each captured image, the minimum two Φ may be calculated by multiplication. For example, if there are N captured images and the following N equations are obtained, the value of φ may be obtained from the N equations by the least square method.

  The phase calculation means 6 may perform this process for each pixel in the captured image.

  Further, as exemplified in step S11, the function f for obtaining the luminance value is a function in which the luminance change is a sine wave, the period thereof is 2π, and d (i) is expressed by Equation (4). Suppose you were looking for. In this case, the phase calculation means 6 may calculate the initial phase value φ in the pixel of interest by performing the calculation of the following equation (6).

φ = π-arctan ((I _c (0) -I _c (2)) / (I _c (3) -I _c (1)))
Formula (6)

  In this case, A and B can be obtained by the following equations (7) and (8), respectively.

A = 0.5 × √ ((I _c (0) −I _c (2)) ² + (I _c (1) −I _c (3)) ² )
Formula (7)

B = (I _c (0) + I _c (1) + I _c (2) + I _c (3)) / 2.0 Formula (8)

Here, I _c (0), I _c (1), I _c (2), and I _c (3) are pixel values of pixels having a common position, respectively, and I _c (0) is the 0th Pixels in the captured image, I _c (1) is a pixel in the first captured image, I _c (2) is a pixel in the second captured image, and I _c (3) is a pixel in the third captured image. It is.

  The phase calculation means 6 may perform the process for obtaining the initial phase value by Expression (6) for each pixel in the captured image.

  A portion (denoted as a target image region) corresponding to the projected image of the captured image is determined in advance. The area may be configured to be designated by an operator. For example, the information processing apparatus 3 may be configured to include a display device that displays a captured image and a pointing device that receives designation of a target image region in the captured image. Alternatively, the phase calculation means 6 may determine that a rectangular area including an area having a luminance value equal to or higher than a predetermined value in the captured image is the target image area.

  After step S13, the association unit 9 determines which x coordinate value in the projected image corresponds to the pixel of the captured image from the initial phase value obtained for the pixel in the captured image (step S14).

Even if the image is distorted on the projection surface 7, the luminance change of the corresponding pixels of the projected image and the captured image is the same. Therefore, the initial phase value of the luminance change is equal between the corresponding pixels. Accordingly, in step S14, the associating unit 9 performs the back calculation of the x coordinate from the initial phase value calculated in step S13, so that the corresponding x coordinate value (that is, the projection unit 1 projects the pixel) in the captured image. X coordinate value in the image to be calculated). For example, when the pattern phase shift means 4 calculates the initial phase value by calculating the expression (1), the associating means 9 uses the initial phase value calculated in step S13 as s ₀ (x) and the expression (1 ) And the x coordinate is calculated backward. By performing this process for each pixel, it is determined which x coordinate in the projected image corresponds to the pixel in the captured image.

  In step S14, when the association unit 9 calculates the x-coordinate corresponding to the pixel in the photographed image, a table indicating the correspondence between the pixel in the photographed image and the pixel of the projected image (hereinafter referred to as an intermediate table). The x and y coordinates of the photographed image are associated with the corresponding x coordinates. For example, it is assumed that the x coordinate obtained by calculating backward from the initial phase value of the (50, 150) pixel in the captured image is “100”. In this case, associating means 9 associates the pixel (50, 150) of the captured image with the x coordinate “100” in the corresponding pixel, as shown in FIG. At this time, the y-coordinate of the pixel corresponding to the pixel of the photographed image has not been specified, and thus is undecided. Although FIG. 8A illustrates the pixels (50, 150) of the captured image, the other pixels are processed similarly.

  Step S1 (refer FIG. 5) is complete | finished by the process of the above steps S11-S14. Similarly, in the process of determining which y coordinate the pixel of the captured image corresponds to (step S2), the process of steps S11 to S14 may be performed.

  However, in step S11, the pattern phase shift means 4 generates a reference pattern image group for the y coordinate. At this time, the pattern phase shift means 4 obtains an initial phase value corresponding to the y coordinate. For example, what is necessary is just to obtain | require the initial phase value according to each coordinate value with the function of the following formula | equation (9).

s ₀ (y) = T / (h−1 + 2 · m) · (ym−) ′ (9)

  In Expression (9), y is the value of the y coordinate of the pixel of the reference pattern image group. h is the height (number of pixels) of the image that can be projected by the projection means 1. T is the same as that of Formula (1). m ′ is a margin constant similar to m in Equation (1), and may be determined as a value of about 1% of h, for example. Expression (9) is an example, and the initial phase value may be determined by a function other than Expression (9).

  When the initial phase value corresponding to the y coordinate is obtained, the first (0th) image of the reference pattern image group is generated by determining the luminance value corresponding to the initial phase value. Further, by determining a luminance value corresponding to the phase value obtained by adding the phase d (i) to the initial phase value for each pixel, another image belonging to the reference pattern image group is generated.

  This process is the same as step S11 already described except that the initial phase value corresponding to the y coordinate value is determined.

  In addition, steps S12 and S13 are performed after step S11. This process is the same as steps S12 and S13 already described.

  In step S14, the association unit 9 determines which y coordinate in the projected image corresponds to the pixel of the captured image from the initial phase value obtained in step S13. At this time, a calculation for obtaining an initial phase value from the y coordinate (for example, Equation (9)) may be performed backward to obtain a y coordinate value from the initial phase value. When the association unit 9 calculates the y coordinate corresponding to the pixel in the captured image, the association unit 9 associates the y coordinate with the pixel in the captured image and adds it to the intermediate table. For example, it is assumed that the y coordinate obtained by calculating backward from the initial phase value of the (50, 150) pixel in the captured image is “50”. In this case, the association unit 9 adds the y coordinate “50” in association with the pixel (50, 150) of the captured image, as shown in FIG. In this way, the y coordinate is added to the state where the y coordinate is undetermined (FIG. 8A) (see FIG. 8B). The association means 9 performs this process for each pixel, and completes the intermediate table. This intermediate table represents the correspondence between the pixels in the captured image and the pixels of the projected image.

After step S2 (see FIG. 5), in step S3, the association unit 9 associates the coordinates of the pixel in the projection target image to be projected onto the pixel in the captured image with the coordinate corresponding to the pixel in the captured image. . FIG. 9 is an explanatory diagram illustrating an example of association in step S3. In order for the image to be observed without distortion, the pixel at the coordinates (X _a , Y _a ) in the projection target image must be projected so as to be the pixel at the coordinates (X _p , Y _p ) in the captured image. To do. A pixel in a captured image observed without distortion is referred to as a target pixel. In the intermediate table, the coordinates associated with the pixel (X _p , Y _p ) in the captured image are (X _b , Y _b ). The associating unit 9 searches the intermediate table for a pixel (X _b , Y _b ) corresponding to the target pixel (X _p , Y _p ), and the pixel (X _a , Y in the projection target image to be projected onto the target pixel. _a ) and the coordinates (X _b , Y _b ) associated with the target pixel in the intermediate table are associated with each other. This association is performed for each pixel in the projection target image to be projected onto each target pixel. Here, the luminance value of the pixel at the coordinate (X _b , Y _b ) in the projection unit 1 is set to a luminance value equal to the pixel at the coordinate (X _a , Y _a ) in the projection target image, and the other values in the projection unit 1 are set. If the luminance value of the pixel is set in the same manner, the pixel of (X _a , Y _a ) in the projection target image has been changed to the position of (X _b , Y _b ), and in this state, the projection is performed on the projection plane 7. Then, it is projected to the position of the desired target pixel (X _p , Y _p ). As described above, the association unit 9 associates the pixel in the projection target image to be projected onto the pixel in the photographed image and the coordinate corresponding to the pixel in the photographed image, thereby obtaining the position of the pixel in the projection target image, The position of the pixel in the projection means 1 is matched.

  When the association unit 9 associates the pixel position of the projection target image with the pixel position in the projection unit 1, the association unit 9 stores or outputs the correspondence relationship as a table (hereinafter referred to as a pixel correspondence table).

  According to the present embodiment, as described above, by associating the pixel of the projection target image to be projected at the target position with the coordinates associated with the position in the intermediate table, the projection target image The correspondence between the pixel position and the pixel position in the projection unit 1 can be specified.

  This correspondence relationship is maintained unless the positional relationship between the projection unit 1, the photographing unit 2, and the projection plane is changed. Therefore, when the image presented to the observer 8 (see FIG. 1) is projected on the projection plane 7 without changing the positional relationship between the projection unit 1, the photographing unit 2, and the projection plane, correction is performed using this correspondence relationship. By doing so, an image with reduced distortion can be shown to the observer.

  Further, in the present embodiment, an initial phase value is determined for each pixel, and the luminance value is obtained from the initial phase value. Further, the phase is shifted from the initial phase value, and the luminance value is obtained from the phase value. Are generated as a reference pattern image group, and each image is projected. Further, each image is taken. At this time, the luminance of each pixel changes between the captured images obtained by the imaging in the same manner as the luminance change between the images of the reference pattern image group. The corresponding initial pixels have the same initial phase value. This property is established regardless of the shape of the projection plane 7. In the present embodiment, this is used to define the correspondence between the pixels in the captured image and the pixels of the projected image. Therefore, even if the shape of the projection surface 7 is an arbitrary shape, it is possible to specify the correspondence between the pixel position of the projection target image and the pixel position in the projection unit.

  Further, if a reference pattern image group is generated and projected onto a projection plane, and a captured image obtained by capturing each image is obtained, the correspondence can be obtained. Therefore, since it is not necessary to use a three-dimensional shape measuring device for measuring the shape of the projection surface, the correspondence can be obtained at low cost.

  In the above-described reference embodiment, the case where a function in which the change in luminance value is a sine wave is used as the function f for obtaining the luminance value has been described as an example. However, the function f changes continuously with a change in phase. Any continuous function may be used. For example, the function f may be a function in which the change in luminance value is a triangular wave instead of a function in which the function value changes smoothly like a sine function.

  The reason for using a continuous function as the function f is as follows. If the function f is not a continuous function and there is a discontinuous portion such as a sawtooth wave, the luminance value of adjacent pixels changes rapidly. Then, when the resolution of the photographing unit 2 is insufficient, a portion where the luminance value changes suddenly is blurred and an intermediate value of the suddenly changing value may be recognized as the luminance. Then, the pixel may be associated with another pixel that should not be associated with the original pixel.

  On the other hand, in the present embodiment, since the continuous function that continuously changes with the phase change is used as the function f for obtaining the luminance value, it is possible to prevent the erroneous association as described above. This means that the correction accuracy can be improved when correcting the projected image based on the correspondence between the pixels.

  Further, in the present embodiment, it is sufficient that the phase calculation unit 6 generates an image that can calculate the initial phase value as the reference pattern image group, and it is not necessary to generate a large number of images as the reference pattern image group. Therefore, the processing can be speeded up. As an example, if a function whose luminance value is changed to a sine wave is used as the function f for obtaining the luminance value, a reference pattern image group including at least three images per coordinate axis may be generated. In this case, since the number of images used as the reference pattern image group can be reduced, the processing can be speeded up.

Reference Embodiment 2.
Similar to the first embodiment, the pixel position correspondence specifying system according to the second embodiment related to the present invention includes a projecting unit 1, a photographing unit 2, and an information processing device 3 (FIG. 1). reference). Here, a second reference embodiment will be described with reference to FIG. Similarly to the first reference embodiment, the pixel position correspondence specifying system of the second reference embodiment creates a pixel correspondence table indicating the correspondence between the pixel position of the projection target image and the pixel position in the projection unit 1. To do. In the second reference embodiment, the information processing device 3 further corrects an image projected on the projection surface 7 for presentation to the observer 8 using the pixel correspondence table, and the corrected image is projected on the projection unit 1. To project.

  FIG. 10 is a block diagram illustrating a configuration example of the information processing device 3 in the second reference embodiment. Constituent elements similar to those in the first reference embodiment are denoted by the same reference numerals as those in FIG. 4, and detailed description thereof is omitted. The information processing apparatus 3 in the second reference embodiment includes a pattern phase shift unit 4, an imaging control unit 5, a phase calculation unit 6, an association unit 9, a correction unit 10, and a projection control unit 11. . The operations of the pattern phase shift unit 4, the imaging control unit 5, the phase calculation unit 6, and the association unit 9 are the same as those in the first reference embodiment.

  The correction unit 10 performs a process of correcting the projection target image projected on the projection plane 7 for presentation to the observer 8 using the pixel correspondence table. The projection control unit 11 projects the corrected image on the projection unit 1.

Next, the operation of this reference embodiment will be described.
First, the information processing device 3 generates a pixel correspondence table that indicates the correspondence between the pixel position of the projection target image and the pixel position in the projection unit 1. This operation is the same as in the first reference embodiment, and a description thereof will be omitted.

After the pixel correspondence table is generated, the correction unit 10 corrects the projection target image presented to the observer 8 (see FIG. 1). The correcting unit 10 may generate an image in which the luminance value of the pixel in the projecting unit 1 is set to the luminance value of the pixel of the projection target image corresponding to the pixel as the corrected image. When the coordinates (X _a , Y _a ) and the coordinates (X _b , Y _b ) are associated with each other as in the example illustrated in FIG. 9, the luminance value of the pixel at (X _b , Y _b ) The luminance value of the pixel (X _a , Y _a ) in the image is set, and an image in which the luminance values of other pixels are similarly set may be generated as a corrected image. As a result, the pixel of (X _a , Y _a ) in the projection target image has been changed to the position of (X _b , Y _b ), and when projected onto the projection plane 7 in that state, the pixel is at the desired position. Projected. As a result, image distortion when projected onto the projection surface 7 can be eliminated.

  The projection control unit 11 causes the projection unit 1 to project the image corrected by the correction unit 10 toward the projection plane 7. It is assumed that the positional relationship between the projection unit 1 and the projection plane 7 is not changed from the time of pixel correspondence table creation at the time of image projection. In addition, the observer 8 views the projection plane 7 from the same direction as the photographing unit 2 when creating the pixel correspondence table.

  When the image is projected without correction, pixels in the projection target image are projected at a position different from the desired position, and the image is distorted. However, in this reference embodiment, the projection target image is corrected using the pixel correspondence table generated in the same manner as in the first reference embodiment, and the corrected image is projected. Therefore, it is possible to project a desired luminance value at each position and project an image on the projection surface 7 having an arbitrary shape without causing the observer to recognize the distortion.

  According to the present embodiment, since the image is corrected using the pixel correspondence table and the image is projected, it is possible to show the observer an image in which distortion caused by the shape of the projection surface is suppressed.

  Similarly to the first reference embodiment, even if the shape of the projection surface 7 is an arbitrary shape, the correspondence relationship between the pixels of the projection target image and the pixels in the projection unit 1 is specified. Regardless of this, an image with reduced distortion can be shown to the observer.

  Further, as in the first reference embodiment, the intermediate table can be created at low cost, at high speed, and with high accuracy.

  Further, the information processing apparatus 3 may be an image correction system that receives a pixel correspondence table from outside without creating a pixel correspondence table and corrects a projection target image using the pixel correspondence table. In this case, the image correction system may not include the projection unit 1 and the photographing unit 2. The information processing apparatus 3 only needs to include the correction unit 10. For example, a pixel position correspondence specifying system similar to that in the first reference embodiment may create a pixel correspondence table and input the pixel correspondence table to the image correction system. Furthermore, the image correction system may not only correct the projection target image but also project the corrected image onto the projection plane. In this case, the image correction system only needs to include a projection unit used when creating the pixel correspondence table and a projection control unit that projects the corrected image onto the projection unit.

  In each of the above-described reference embodiments and the present invention, the correspondence relationship between the pixel position of the projection target image and the pixel position of the projection unit 1 (using the phase of the reference pattern image group in which the luminance continuously changes) ( Pixel correspondence table). Therefore, it is possible to obtain a correspondence relationship for each pixel (that is, at a high density), and it is possible to obtain a highly accurate correspondence relationship in which the corresponding pixel is represented by, for example, a floating point. In the embodiment of the present invention described below, in particular, before the calculation of the initial phase value, the resolution of an area for which a correspondence relationship is to be obtained in the captured image (an area corresponding to an area where the projection target image is to be projected) is the projection target image. The resolution is converted so as to be the same, and then the initial phase value is calculated.

Embodiment 1. FIG.
The embodiment of the present invention includes a projecting unit 1, a photographing unit 2, and an information processing device 3 (see FIG. 1), as in the first and second reference embodiments. In the pixel position correspondence specifying system according to the embodiment of the present invention, if the captured image and the image to be presented to the observer (projection target image) have different resolutions, the captured image is enlarged or reduced, The resolution of the captured image is adjusted to the resolution of the image to be presented to the observer. Then, processing for obtaining an initial phase value for the captured image after resolution conversion (that is, after enlargement or reduction) (see step S13, FIG. 6) is performed, and the coordinates of the reference pattern image group of the pixels of the captured image after resolution conversion are performed. Is processed (step S14, see FIG. 6), and then a pixel correspondence table is generated. As a result of this processing, even when the resolution of the captured image and the projection target image does not match, a pixel correspondence table that matches the desired resolution can be generated.

  In particular, when a captured image is reduced, a highly accurate pixel correspondence table can be obtained. For example, if the resolution of the captured image is higher than that of the projection target image, if the captured image is reduced before the initial phase value is calculated, the image has less noise and higher contrast than the performance of the imaging means 2 (camera). Is obtained. If the initial phase value is calculated using the image in the same manner as in the first embodiment and a pixel correspondence table is obtained, a highly accurate pixel correspondence table can be obtained.

  FIG. 11 is a block diagram illustrating a configuration example of the information processing apparatus 3 in the present embodiment. Constituent elements similar to those in the first and second reference embodiments are denoted by the same reference numerals as those in FIGS. 4 and 10, and detailed description thereof is omitted. The information processing apparatus 3 according to this embodiment includes a pattern phase shift unit 4, an imaging control unit 5, a resolution conversion unit 12, a phase calculation unit 6, an association unit 9, a correction unit 10, and a projection control unit 11. With.

  The resolution conversion unit 12 enlarges or reduces the target image area in the captured image so that the resolution of the target image area in the captured image becomes a desired resolution to be presented to the observer. The resolution conversion means 12 may cut out the target image area from the captured image and enlarge or reduce the cut-out image. In the captured image, a portion of the projection plane corresponding to a region desired by the operator who wants to project the image to be captured may be designated as the target image region by the operator, for example.

  The pattern phase shift unit 4, the imaging control unit 5, the phase calculation unit 6, the association unit 9, the correction unit 10, the projection control unit 11, and the resolution conversion unit 12 are a CPU that operates according to a program (pixel position correspondence specifying program). It is realized by. The program is stored, for example, in a program storage device (not shown) included in the information processing apparatus 3, and the CPU reads the program, and according to the program, the pattern phase shift means 4, the imaging control means 5, the phase calculation means 6, The association unit 9, the correction unit 10, the projection control unit 11, and the resolution conversion unit 12 may be operated. Further, each means may be separate hardware.

Next, the operation of this embodiment will be described.
First, the information processing device 3 performs association (step S1, see FIG. 5) regarding the x coordinate and association (step S2, see FIG. 5) regarding the y coordinate. In steps S1 and S2, basic pattern image generation processing and photographing processing are performed, respectively. The basic pattern image generation process and the photographing process are the same as the processes in steps S11 and S12 in the first reference embodiment, and a description thereof will be omitted.

  After the basic pattern image generation process and the photographing process, the resolution conversion unit 12 sets the target image area in the photographed image so that the resolution of the target image area in the photographed image is the same as that of the desired resolution presented to the observer. Zoom in or out. An image after resolution conversion by enlargement or reduction can be called a resolution conversion image.

  In the case of enlargement, the resolution conversion means 12 determines the luminance value at the coordinate after resolution conversion as the corresponding luminance value at the coordinate before resolution conversion. In addition, since the number of pixels increases in the case of enlargement, among the pixels of the enlarged image, the pixels having no corresponding coordinates before resolution conversion are interpolated from the predetermined luminance value by linear interpolation or bilinear interpolation. The luminance value thus determined may be determined.

In the case of reduction, the resolution conversion means 12 determines the luminance value at the coordinate after resolution conversion based on the luminance value at the corresponding coordinate before resolution conversion. In the case of reduction, one pixel in the image after conversion corresponds to a plurality of pixels in the image before conversion. The resolution conversion unit 12 weights the luminance values of a plurality of pre-conversion pixels associated with the post-conversion pixels with an area corresponding to the post-conversion pixels, and obtains a linear sum, thereby obtaining the luminance of the post-conversion pixels. Find the value. The pixel after conversion is a pixel in the image after conversion, and the pixel before conversion is a pixel in the image before conversion. FIG. 12 is an explanatory diagram illustrating an example of determining the luminance of the pixel after the reduction conversion. It is assumed that the pixel 65 in the image after reduction corresponds to the four pixels 61 to 64 in the image before reduction. Then, the luminance values of the pixels 61 to 64, respectively, and a _P 1 to P _4. In addition, the pixel 61 does not correspond only to the converted pixel 65 but also corresponds to another pixel (not shown) in contact with the pixel 65, and the area corresponding to the converted pixel 65 in the pixel 61 is as follows. , 1/4. The same applies to the other pixels 62, 63, 64 before conversion. Therefore, in the example shown in FIG. 12, when the luminance values of the pixels 61 to 64 are weighted by the ratio of the area and the linear sum is obtained, P ₁ × (1/4) + P ₂ × (1/4) + P ₃ × (1/4) + P ₄ × (1/4) = (P ₁ + P ₂ + P ₃ + P ₄ ) / 4. The resolution conversion means 12 may perform this calculation to determine the luminance value of the converted pixel 65 as (P ₁ + P ₂ + P ₃ + P ₄ ) / 4. The resolution conversion unit 12 similarly determines a luminance value for each pixel in the converted image.

  The resolution conversion unit 12 obtains individual luminance values as described above, and generates an enlarged or reduced image. In addition, the resolution conversion unit 12 enlarges or reduces each captured image in the same manner.

  After each photographic image has undergone resolution conversion (ie, enlargement or reduction), the phase calculation means 6 is based on a change in luminance between each photographic image after resolution conversion (ie, each resolution conversion image) for each pixel. The initial phase value in the change is obtained. This process is the same as step S13 in the first reference embodiment. Subsequently, the associating unit 9 associates the coordinates of the pixels in the resolution-converted image with the coordinates of the pixels in the projected image. This process is the same as step S14 in the first reference embodiment.

  The information processing device 3 performs the resolution conversion on the captured image as described above after performing the basic pattern image group generation processing and the imaging processing in the association processing (step S1, see FIG. 5) regarding the x coordinate, and performs the resolution conversion. Using the image, the initial phase value is calculated, and the pixel coordinate of the resolution-converted image is associated with the x coordinate of the pixel of the image to be captured. Similarly, in the association processing related to the y-coordinate (see step S2, FIG. 5), after performing the basic pattern image group generation processing and shooting processing, resolution conversion is performed on the shooting image as described above, and the resolution conversion image is used. Thus, the initial phase value is calculated, and the coordinates of the pixels of the resolution-converted image are associated with the y-coordinates of the pixels of the captured image. The association means 9 generates a pixel correspondence table using this result (step S3 shown in FIG. 5).

  The correction means 10 corrects the image projected on the projection surface 7 for presentation to the observer 8 using this pixel correspondence table. The operation of the correcting means 10 is the same as that in the second reference embodiment. In addition, the projection control unit 11 projects the image corrected by the correction unit 10 onto the projection plane 7 on the projection unit 1. It is assumed that the positional relationship between the projection unit 1 and the projection plane 7 is not changed from the time of pixel correspondence table creation at the time of image projection. In addition, the observer 8 views the projection plane 7 from the same direction as the photographing unit 2 when creating the pixel correspondence table. This is the same as in the second reference embodiment.

  According to the present embodiment, even if the projection target image that is an image that the observer wants to show to the observer and the captured image do not have the same resolution, a pixel correspondence table corresponding to the resolution of the desired projection target image is created and The projection target image can be projected at a resolution of. In addition, an image with suppressed distortion can be shown to the observer regardless of the shape of the projection surface.

  Further, when the resolution conversion unit 12 reduces the captured image, a highly accurate pixel correspondence table can be obtained. The determination accuracy of the correspondence relationship (pixel correspondence table) using the initial phase value depends on the measurement accuracy of the initial phase value, and the measurement accuracy of the initial phase value mainly occurs in the captured image with respect to the contrast of the captured pattern. Depends on the noise level. If the contrast of the photographed pattern is large or the noise is small, the measurement accuracy of the initial phase value is improved. When a high-resolution camera is used, if the resolution conversion unit 12 reduces the captured image of the reference pattern image group, the captured image can be converted into an image with a large contrast: noise ratio. This is because, since the luminance values of a plurality of pixels are added in the reduction process, noises that are random for each pixel cancel each other and become smaller, and the contrast becomes higher due to the addition. The accuracy of the pixel correspondence table can be improved by calculating the initial phase value using such an image after conversion and generating the pixel correspondence table.

Next, modifications of the above-described reference embodiments and embodiments of the present invention will be described.
In each of the above reference embodiments and embodiments of the present invention, the pattern phase shift means 4 may generate a plurality of types of reference pattern image groups for each of the x-axis and y-axis coordinate axes. Hereinafter, a case where the pattern phase shift means 4 generates two types of reference pattern image groups for each coordinate axis will be described.

  In this modification, the pattern phase shift means 4 generates a first reference pattern image group and a second reference pattern image group. The luminance change period in the coordinate axis direction in the second reference pattern image group is 1 / a times the luminance change period in the coordinate direction in the first reference pattern image group. In other words, the second reference pattern image group is an image group obtained by compressing the first reference pattern image group by 1 / a times in the coordinate axis direction. However, each image belonging to the second reference pattern image group has the same size as each image belonging to the first reference pattern image group, and the same pattern appears repeatedly in the images belonging to the second reference pattern image group. It is an image. That is, the image belonging to the second reference pattern image group is an image in which a pattern obtained by compressing the image belonging to the first reference pattern image group by 1 / a times is repeated a times. a is a value larger than 1, for example. Here, a case where a = 2 will be described as an example.

  FIG. 13 is an explanatory diagram schematically showing two types of reference pattern image groups. FIG. 13A shows a first reference pattern image group, and FIG. 13B shows a second reference pattern image group. Further, FIG. 13 shows a case where the luminance is changed in the x-axis direction, where the solid line represents the portion where the luminance is minimum (darkest place), and the wavy line represents the portion where the luminance is highest (brightest place). ing. When each reference pattern image group is obtained as described above with a = 2, as shown in FIG. 13, the luminance change period in the second reference pattern image group is the luminance change period in the first reference pattern image group. It is 1/2 the period. The image belonging to the second reference pattern image group is an image obtained by repeating twice a pattern obtained by compressing an image belonging to the first reference pattern image group by a factor of 1/2.

  The pattern phase shift means 4 may generate the first reference pattern image group similarly to the reference pattern image group described in the first reference embodiment. Further, the pattern phase shift means 4 can obtain the second reference pattern image group by using a function obtained by multiplying a function for obtaining an initial phase value corresponding to the coordinate value (for example, Expressions (1) and (9)). That's fine. For example, when generating the first reference pattern image group, when Expression (1) or Expression (9) is used, when generating the second reference pattern image group, the following expression (1 ′ ) Or formula (9 ′) may be used.

_{s 0 (x) = T /} (w-1 + 2 · m) · (x-m) · a formula (1 ')
s ₀ (y) = T / (h−1 + 2 · m) · (y−m ′) · a Formula (9 ′)

  The generation process of the first reference pattern image group and the generation process of the second reference pattern image group are the same processes except for the function for obtaining the initial phase value according to the coordinate value.

  The imaging control unit 5 projects each image belonging to the first reference pattern image group and each image belonging to the second reference pattern image group onto the projection plane 7 on the projection unit 1 one by one. The photographing control unit 5 causes the photographing unit 2 to photograph each image projected on the projection surface 7 one image at a time and receives the photographed image from the photographing unit 2.

  Next, the phase calculation means 6 obtains, for each pixel, an initial phase value in the luminance change of each pixel based on the photographed image corresponding to the first reference pattern image group. This process is the same as step S13 in the first reference embodiment. Similarly, the phase calculation means 6 obtains an initial phase value for each pixel for the luminance change of each pixel based on the captured image corresponding to the second reference pattern image group.

  Next, the associating unit 9 calculates an initial phase value (referred to as a first initial phase value group) calculated based on a captured image corresponding to the first reference pattern image group, and a second reference pattern image group. The coordinate value obtained from the initial phase value is associated with the pixel in the captured image using the initial phase value (referred to as a second initial phase value group) calculated based on the captured image according to the above. Specifically, the associating unit 9 reversely calculates coordinates from the initial phase values for the second initial phase value group. If the equations used to calculate the initial phase value at the time of generating the second reference pattern image group are the equations (1 ') and (9'), the coordinate values may be calculated by performing a reverse calculation of these equations.

  However, as shown in FIG. 13B, each image belonging to the second reference pattern image group is an image in which the same pattern is repeated. For this reason, even if the coordinate values are different, the initial phase values match and there is an indefiniteness that the coordinate values cannot be uniquely determined from the initial phase values. In order to eliminate ambiguity, the associating unit 9 adds a coordinate value obtained by adding an integer multiple of the period of luminance change in the second reference pattern image group to the coordinate value obtained by back calculation from the second initial phase value group. Is calculated as a candidate. For example, assuming that the coordinate value obtained by back calculation is t and the luminance change period in the second reference pattern image group is Q, t + Q, t + 2Q,.

  In addition, the associating unit 9 includes a captured image corresponding to the first reference pattern image group located at the same position as the pixel of the captured image corresponding to the second reference pattern image group for which the corresponding coordinate value is to be obtained. Also for the pixel, the coordinate value is calculated backward from the initial phase value. Then, the associating means 9 is the coordinate closest to the coordinate calculated backward from the initial phase value of the pixel of the photographed image corresponding to the first reference pattern image group among the coordinate candidates obtained from the second initial phase value group. Are determined as coordinate values corresponding to the pixels. This process may be performed for each pixel.

  In this modification, the accuracy of the association process of the association unit 9 can be improved. The measurement error of the initial phase value is the same in both reference pattern image groups, but in the second reference pattern image group, the initial phase value with respect to the corresponding coordinates changes by a times. That is, the error when the corresponding coordinates are obtained from the initial phase value is 1 / a. For this reason, the precision of a matching process can be improved.

  In each of the above-described reference embodiments and the embodiments of the present invention and modifications thereof, the correction unit 10 updates the luminance of the image projected by the projection unit 1 using the reflectance, and then applies the correction to the image. Correction may be performed. In this case, the information processing apparatus 3 includes a reflectance calculation unit that calculates the reflectance on the projection surface. The reflectance calculation unit includes a unit (for example, the correction unit 10 or the phase calculation unit 6) included in the information processing apparatus 3. ). Here, a case where the correcting unit 10 operates also as a reflectance calculating unit will be described as an example.

The correction unit 10 _{calculates the} luminance amplitude (Ap) in the reference pattern image group generated by the pattern phase shift unit 4 and the luminance amplitude ( _Ac ) in the captured image captured by the imaging unit 2. The reflectance A _cp is calculated. The correcting unit 10 may obtain the reflectance A _cp by calculating A _cp = A _c / A _p .

Further, the correcting means 10 does not obtain the luminance amplitude _Ac in the photographed image from the photographed image of the reference pattern image group, but projects a maximum brightness image and a minimum brightness image on the projection plane separately from the reference pattern image group. Then, it may be obtained from the difference between the luminance values of the captured images obtained by capturing the images. In this case, the imaging control means 5 sets the brightness value of all pixels to the maximum value and the brightness value of all pixels to the minimum value separately from the reference pattern image group. The dark image (the dark image as a whole) may be projected onto the projection unit 1 and the projected bright image and dark image may be photographed onto the photographing unit 2, respectively. Correcting means 10 may be determined amplitude A _c from the difference of the luminance values of the two photographed images.

Further, when the luminance of the image projected by the projection unit 1 to be presented to the observer is K, the correction unit 10 divides the luminance by the reflectance A _cp (that is, K / A _cp ). Update. Then, after the luminance is updated, the image may be corrected using the pixel correspondence table.

  In each of the above-described reference embodiments and the embodiments of the present invention, the case where there is one projection unit 1 has been described as an example. However, the pixel position correspondence relationship specifying system may include a plurality of projection units 1. In that case, the information processing apparatus 3 may create a pixel correspondence table for each projection unit 1. When an image is projected on the plurality of projection means 1, the image presented to the user may be corrected using a pixel correspondence table created for each projection means 1.

  The plurality of projection means 1 may project an image on the same area on the projection plane 7. In this case, the image projected from the plurality of projection means 1 on the same region is presented to the user, so that the image projected on the projection plane 7 can be brightened.

  Alternatively, the projection of individual images obtained by dividing a large image may be assigned to each projection unit 1 so that each projection unit 1 projects the projection position on the projection plane while shifting. For example, the first projection unit 1 may share the right half projection in the image presented to the user, and the second projection unit 1 may share the left half projection. And the 2nd projection means 1 may project an image next to the area | region where the 1st projection means 1 projects an image. In this case, a larger image can be presented to the user than when an image is projected using only one projection unit 1.

  In the above description, the case where the photographing unit 1 projects images belonging to the reference pattern image group one image at a time and the photographing unit 2 photographs the images one image at a time has been described. May project a plurality of images onto the projection surface at the same time. At this time, the imaging unit 2 may shoot the projected image and separate it for each color having a different wavelength of light.

  It is assumed that only one projection means 1 is provided. At this time, the projecting unit 1 may simultaneously project the images belonging to the reference pattern image group as images of colors having different light wavelengths. For example, it is assumed that there are three images belonging to the reference pattern image group, and the projection unit 1 can project in three colors of red, green, and blue. In this case, the projection unit 1 may project the first to third images at the same time in three colors of red, green, and blue. In this case, the imaging means 2 may shoot images simultaneously projected in red, green, and blue and separate them in red, green, and blue colors. The information processing apparatus 3 may perform processing on the separated image.

  A plurality of projection means may be provided, and each projection means may simultaneously project an image with a color having a different wavelength of light. For example, three projection means 1 are provided, a red image is projected by the first projection means, a green image is projected by the second projection means, and a blue image is projected by the third projection means. May be performed simultaneously. In this case as well, the photographing means 2 may shoot images simultaneously projected in red, green, and blue, and separate them in red, green, and blue colors.

  Alternatively, a plurality of projection units may be provided, and each projection unit may project an image with a plurality of colors at the same time. For example, five projection means capable of projecting light at 15 types of wavelengths are provided. In this case, each of the projecting means simultaneously projects three types of images with colors of different wavelengths. Each projection means also projects simultaneously. The photographing means 2 may shoot each image projected with colors of 15 types of wavelengths and separate them into 15 types of colors.

  Further, in each of the reference embodiments and the embodiments of the present invention, the case where the pixel position correspondence specifying system includes the photographing unit 2 (see FIG. 1) has been described. However, the pixel position correspondence specifying system is the photographing unit 2. The projection means for projecting the image projected on the projection plane 7 (see FIG. 1) may be provided separately from the pixel position correspondence specifying system. As such a projecting means, for example, a mobile terminal with a camera can be used. In this case, the information processing apparatus 3 only needs to include a reference pattern projection control unit that causes the projection unit 1 to project an image belonging to the reference pattern image group, instead of the imaging control unit 5. The camera-equipped mobile terminal shoots each image of the reference pattern image group projected on the projection surface, for example, by the operation of the observer, and outputs the image to the information processing device 3. The information processing apparatus 3 may perform the same operation as the above-described reference embodiments and embodiments of the present invention using a plurality of captured images input from the outside.

  Next, the outline of the present invention will be described. FIG. 14 is a block diagram showing an outline of the present invention. The pixel position correspondence specifying system according to the present invention includes a projection unit 71 (for example, the projection unit 1 in the embodiment), an imaging unit 72 (for example, the imaging unit 2 in the embodiment), and a reference pattern image group generation unit 73 (for example, , Pattern phase shift means 4), imaging control means 74 (for example, imaging control means 5 in the embodiment), resolution conversion means 77 (for example, resolution conversion means 12 in the embodiment), and phase calculation means 75 (for example, Phase calculation means 6) in the embodiment and association means 76 (for example, association means 9 in the embodiment) are provided.

  The projection unit 71 projects an image on the projection plane. The imaging unit 72 captures an image projected on the projection plane.

  The reference pattern image group generation unit 73 is an image group in which a luminance value is determined as a function value of a continuous function that continuously changes with a change in phase, and the luminance value is determined by changing the phase for each image, A reference pattern image group which is an image group in which the initial phase value is continuously changed for each coordinate value along one coordinate axis is generated.

  The resolution conversion unit 77 cuts out a portion corresponding to the projection area of the projection target image on the projection plane from the photographed image taken by the photographing unit 2, and converts the resolution of the image by enlarging or reducing the cut-out image. A resolution-converted image is generated.

  The imaging control unit 74 causes the projection unit 71 to project an image belonging to the reference pattern image group, and as a result, causes the imaging unit 72 to capture an image projected on the projection plane.

  The phase calculation means 75 calculates the initial phase value at the luminance change of each pixel in the resolution conversion image from the luminance change of the pixels in the plurality of resolution conversion images.

  The associating means 76 performs a reverse calculation of the calculation (for example, the calculation of the equation (1)) by which the reference pattern image group generating means obtains the initial phase value from the coordinate values with respect to the initial phase value calculated by the phase calculating means 75, The coordinate value obtained by the back calculation is associated with the pixel of the resolution conversion image, and the correspondence between the pixel in the projection target image and the pixel in the projection unit 71 is specified.

  Specifically, the reference pattern image group generation unit 73 applies the reference pattern image group related to the first coordinate axis whose initial phase value is changed along the first coordinate axis (for example, the x axis) and the second coordinate axis (for example, the y axis). And a reference pattern image group related to the second coordinate axis in which the initial phase value is changed along with the reference phase image group.

  Then, the photographing control unit 74 causes the projection unit 71 to project the reference pattern image group related to the first coordinate axis, and causes the photographing unit 72 to capture the image projected on the projection plane as a result. The image is projected on the projection unit 71, and as a result, the image projected on the projection plane is caused to be photographed by the photographing unit 72.

  The phase calculation means 75 is based on pixel brightness changes in a plurality of resolution-converted images converted from a plurality of photographed images photographed by the photographing means when the projection means 71 projects each image of the reference pattern image group related to the first coordinate axis. The initial phase value at the luminance change of each pixel in the resolution-converted image is calculated, and a plurality of captured images captured by the imaging unit 72 when the projection unit 71 projects each image of the reference pattern image group related to the second coordinate axis. From the luminance changes of the pixels in the plurality of resolution-converted images converted from the above, the initial phase value at the luminance change of each pixel in the resolution-converted image is calculated.

  The associating unit 76 obtains an initial phase value from the coordinates of the first coordinate axis (for example, the x coordinate) with respect to the initial phase value calculated from the resolution conversion image of the captured image obtained by capturing the reference pattern image group related to the first coordinate axis. Calculate the coordinates of the first coordinate axis by performing the reverse calculation of the calculation (for example, the calculation of Expression (1)), associate the coordinates of the first coordinate axis with the pixels of the resolution conversion image, and photograph the reference pattern image group related to the second coordinate axis For the initial phase value calculated from the resolution-converted image of the captured image, a reverse calculation of the initial phase value from the coordinates of the second coordinate axis (for example, the y coordinate) (for example, the calculation of equation (9)) is performed. By calculating the coordinates of the second coordinate axis and associating the coordinates of the second coordinate axis with the pixels of the resolution conversion image, the correspondence relationship between the pixels of the resolution conversion image and the pixels of the image projected by the projection unit 71 Identified, to identify the correspondence between the pixels in the pixel with the projection means in the projection target image using the correspondence between the pixel and the coordinates of the resolution-converted image.

  Further, the above embodiment discloses a configuration in which the resolution conversion unit 77 generates a resolution-converted image of the image by reducing the image cut out from the captured image.

  In the above-described embodiment, a correction unit (for example, the correction unit 10 in the embodiment) that corrects the projection target image based on the correspondence between the pixel in the projection target image and the pixel in the projection unit 71, and a correction unit A configuration including a projection control unit (for example, the projection control unit 11 in the embodiment) that causes the projection unit 71 to project the image corrected by the above is disclosed.

  Further, in the above embodiment, the reference pattern image group generation unit 73 is configured to multiply the first reference pattern image group and the luminance change cycle in the first reference pattern image group by a predetermined multiple (for example, 1 / a times). The second reference pattern image group is generated, and the imaging control unit 74 causes the projection unit 71 to project the image belonging to the first reference pattern image group and the image belonging to the second reference pattern image group. The image projected on the projection plane is photographed by the photographing means 72, and the phase calculating means 75 calculates the initial phase value in the luminance change of each pixel based on the resolution conversion image of the photographed image corresponding to the first reference pattern image group. And calculating an initial phase value in the luminance change of each pixel based on the resolution-converted image of the captured image corresponding to the second reference pattern image group. Using the initial phase value calculated based on the resolution-converted image of the captured image corresponding to the initial phase value calculated based on the resolution-converted image of the captured image corresponding to the second reference pattern image group, A configuration is disclosed in which the coordinate values obtained from the image and the pixels of the resolution-converted image are associated with each other and the correspondence between the pixels in the projection target image and the pixels in the projection unit 71 is specified using the correspondence. According to such a configuration, it is possible to improve the accuracy of association.

  Further, the above embodiment includes a reflectance calculation unit (for example, the correction unit 10) that calculates the reflectance on the projection surface, and the correction unit reflects the luminance value of the pixel of the projection target image and reflects the luminance value. A configuration is disclosed in which a value obtained by dividing by a rate is updated and correction is performed on the updated image.

  In addition, a configuration is disclosed in which a continuous function (for example, f) that continuously changes with a change in phase is a function in which a change in function value becomes a sine wave.

  Further, a configuration including a plurality of projection means is disclosed.

  In the above-described embodiment, the projecting unit 1 simultaneously projects images belonging to the reference pattern image group as images of colors having different light wavelengths, and the image capturing unit 2 captures the image and the light wavelength is reduced. The structure which isolate | separates for every different color is disclosed.

  The present invention relates to a pixel position that defines a correspondence relationship between a pixel of an image to be projected and a pixel of an image that is actually projected onto a projection surface in order to correct image distortion when the image is projected onto the projection surface. The present invention is suitably applied to a correspondence relationship identification system and an image projection system that corrects and projects an image presented to an observer using the correspondence relationship.

It is a block diagram which shows the example of the pixel position corresponding | compatible relationship specific system of 1st Embodiment relevant to this invention. It is explanatory drawing which shows the example of the function showing the luminance value which changes continuously with the change of a phase. FIG. 3 is an explanatory diagram illustrating a luminance value when the phase is shifted by d from the initial phase value illustrated in FIG. 2. It is a block diagram which shows the structural example of information processing apparatus. It is a flowchart which shows the example of the process progress of 1st reference embodiment. It is a flowchart which shows the example of the process of the matching regarding x coordinate. It is explanatory drawing which shows the example of the brightness | luminance change of the same pixel in a some picked-up image. It is explanatory drawing which shows the example of the intermediate table which matched the pixel of the image and the pixel of a picked-up image which are projected. It is explanatory drawing which shows the example of matching in step S3. It is a block diagram which shows the structural example of the information processing apparatus in 2nd reference embodiment. It is a block diagram which shows the structural example of the information processing apparatus in embodiment of this invention. It is explanatory drawing which shows the example of the brightness | luminance determination of the pixel after reduction conversion. It is explanatory drawing which shows typically two types of reference pattern image groups. It is a block diagram which shows the outline | summary of this invention.

Explanation of symbols

DESCRIPTION OF SYMBOLS 1 Projection means 2 Imaging means 3 Information processing apparatus 4 Pattern phase shift means 5 Imaging control means 6 Phase calculation means 7 Projection surface 8 Observer 9 Corresponding means 10 Correction means 11 Projection control means 12 Resolution conversion means

Claims (17)

Projection means for projecting an image onto a projection plane;
Photographing means for photographing an image projected on the projection surface;
A group of images in which the luminance value is determined as a function value of a continuous function that continuously changes with the change of the phase, and the luminance value is determined by changing the phase for each image, and the initial phase value along one coordinate axis A reference pattern image group generation means for generating a reference pattern image group which is an image group continuously changed for each coordinate value;
An imaging control unit that causes the projection unit to project an image belonging to a reference pattern image group, and as a result, causes the imaging unit to capture an image projected on the projection plane;
A portion corresponding to the projection area of the projection target image on the projection plane is cut out from the photographed image photographed by the photographing means, and a resolution-converted image in which the resolution of the image is converted is generated by enlarging or reducing the cropped image. Resolution conversion means for
A phase calculation means for calculating an initial phase value at a luminance change of each pixel in the resolution conversion image from a luminance change of the pixels in the plurality of resolution conversion images;
The reference pattern image group generation means performs reverse calculation of calculation for obtaining the initial phase value from the coordinate value with respect to the initial phase value calculated by the phase calculation means, and obtains the coordinate value obtained by the reverse calculation and the pixel of the resolution conversion image. An association means for specifying correspondence between a pixel in the projection target image and a pixel in the projection means;
The reference pattern image generation means includes
Generating a reference pattern image group related to the first coordinate axis in which the initial phase value is changed along the first coordinate axis, and a reference pattern image group related to the second coordinate axis in which the initial phase value is changed along the second coordinate axis;
The photographing control means includes
Projecting a reference pattern image group related to the first coordinate axis on the projecting unit, causing the image capturing unit to capture an image projected on the projection plane, and projecting a reference pattern image group related to the second coordinate axis to the projecting unit; As a result, the image projected on the projection plane is photographed by the photographing means,
The phase calculation means includes
From the luminance change of the pixels in the plurality of resolution conversion images converted from the plurality of photographed images captured by the photographing unit when the projection unit projects each image of the reference pattern image group related to the first coordinate axis, A plurality of images converted from a plurality of photographed images photographed by the photographing means when the projection means projects each image of the reference pattern image group related to the second coordinate axis by calculating an initial phase value at a luminance change of each pixel. From the luminance change of the pixels in the resolution-converted image, the initial phase value at the luminance change of each pixel in the resolution-converted image is calculated,
The association means includes
The initial phase value calculated from the resolution conversion image of the captured image obtained by capturing the reference pattern image group related to the first coordinate axis is subjected to the reverse calculation of the calculation for obtaining the initial phase value from the coordinates of the first coordinate axis, and the coordinates of the first coordinate axis And the coordinates of the first coordinate axis are associated with the pixels of the resolution conversion image, and the second phase is calculated with respect to the initial phase value calculated from the resolution conversion image of the captured image obtained by capturing the reference pattern image group related to the second coordinate axis. Calculating the initial phase value from the coordinates of the coordinate axes, calculating the coordinates of the second coordinate axes by calculating the coordinates of the second coordinate axes, and associating the coordinates of the second coordinate axes with the pixels of the resolution converted image; The correspondence between the pixels of the image projected by the means is specified, and the correspondence between the pixels of the resolution converted image and the coordinates is used for the pixels in the projection target image and the projection means. Kicking pixel position correspondence specifying system, wherein the identifying the correspondence between the pixels.   The pixel position correspondence specifying system according to claim 1, wherein the resolution conversion unit generates a resolution conversion image of the image by reducing an image cut out from the captured image. Correction means for correcting the projection target image based on the correspondence between the pixels in the projection target image and the pixels in the projection means;
The pixel position correspondence specifying system according to claim 1, further comprising: a projection control unit that causes the projection unit to project the image corrected by the correction unit. The reference pattern image group generation means generates a first reference pattern image group and a second reference pattern image group in which the period of luminance change in the first reference pattern image group is a predetermined multiple,
The imaging control means causes the projection means to project an image belonging to the first reference pattern image group and an image belonging to the second reference pattern image group, and as a result, the image projected onto the projection plane is the imaging means. Let me shoot,
The phase calculation means calculates an initial phase value in a luminance change of each pixel based on a resolution conversion image of the captured image corresponding to the first reference pattern image group, and performs imaging corresponding to the second reference pattern image group. Calculate the initial phase value in the luminance change of each pixel based on the resolution conversion image of the image,
The associating means applies an initial phase value calculated based on a resolution-converted image of the captured image corresponding to the first reference pattern image group and a resolution-converted image of the captured image corresponding to the second reference pattern image group. Using the initial phase value calculated based on this, the coordinate value obtained from the initial phase value is associated with the pixel of the resolution conversion image, and the correspondence between the pixel in the projection target image and the pixel in the projection unit The correspondence relationship identifying system according to any one of claims 1 to 3, wherein the correspondence relationship is identified. A reflectance calculating means for calculating the reflectance at the projection surface;
The pixel position according to claim 3 or 4, wherein the correction means updates the luminance value of the pixel of the projection target image to a value obtained by dividing the luminance value by the reflectance, and corrects the updated image. Correspondence relationship identification system.   The pixel position correspondence specifying system according to any one of claims 1 to 5, wherein the continuous function that continuously changes with a change in phase is a function in which a change in function value becomes a sine wave. . The pixel position correspondence specifying system according to claim 1, comprising a plurality of projection means. The projecting means simultaneously projects images belonging to the reference pattern image group as images of colors having different wavelengths of light,
The pixel position correspondence specifying system according to any one of claims 1 to 7, wherein the photographing unit photographs the image and separates the images for each color having a different wavelength of light. Projection means for projecting an image onto a projection plane;
A group of images in which the luminance value is determined as a function value of a continuous function that continuously changes with the change of the phase, and the luminance value is determined by changing the phase for each image, and the initial phase value along one coordinate axis A reference pattern image group generation means for generating a reference pattern image group which is an image group continuously changed for each coordinate value;
Reference pattern projection control means for causing the projection means to project an image belonging to a reference pattern image group;
A portion corresponding to the projection area of the projection target image on the projection plane is cut out from a plurality of shot images obtained by shooting each image of the reference pattern image group projected on the projection plane, and the cut-out image is enlarged or reduced. Resolution conversion means for generating a resolution-converted image obtained by converting the resolution of the image,
A phase calculation means for calculating an initial phase value at a luminance change of each pixel in the resolution conversion image from a luminance change of the pixels in the plurality of resolution conversion images;
The reference pattern image group generation means performs reverse calculation of calculation for obtaining the initial phase value from the coordinate value with respect to the initial phase value calculated by the phase calculation means, and obtains the coordinate value obtained by the reverse calculation and the pixel of the resolution conversion image. An association means for specifying correspondence between a pixel in the projection target image and a pixel in the projection means;
The reference pattern image generation means includes
Generating a reference pattern image group related to the first coordinate axis in which the initial phase value is changed along the first coordinate axis, and a reference pattern image group related to the second coordinate axis in which the initial phase value is changed along the second coordinate axis;
The reference pattern projection control means includes
Projecting a reference pattern image group related to a first coordinate axis on the projection means, projecting a reference pattern image group related to a second coordinate axis to the projection means,
The phase calculation means includes
From the luminance change of the pixels in the plurality of resolution converted images converted from the plurality of captured images obtained by imaging each image of the reference pattern image group related to the first coordinate axis, the luminance change of each pixel in the resolution converted image From the luminance change of the pixels in the plurality of resolution conversion images converted from the plurality of photographed images obtained by calculating the initial phase value and photographing each image of the reference pattern image group related to the second coordinate axis, each in the resolution conversion image Calculate the initial phase value at the luminance change of
The association means includes
A reverse calculation of the initial phase value calculated from the coordinates of the first coordinate axis is performed on the initial phase value calculated from the resolution-converted image of the captured image obtained by shooting the reference pattern image group related to the first coordinate axis. The initial coordinate calculated from the resolution-converted image of the captured image obtained by calculating the coordinates of one coordinate axis, associating the coordinates of the first coordinate axis with the pixels of the resolution-converted image, and capturing the reference pattern image group related to the second coordinate axis. By performing a reverse calculation of the calculation for obtaining the initial phase value from the coordinates of the second coordinate axis with respect to the phase value, the coordinates of the second coordinate axis are calculated, and the coordinates of the second coordinate axis are associated with the pixels of the resolution conversion image. The correspondence relationship between the pixel of the converted image and the pixel of the image projected by the projection unit is specified, and the pixel in the projection target image is determined using the correspondence relationship between the pixel of the resolution converted image and the coordinates Pixel position correspondence specifying system, wherein the identifying the correspondence between the pixels in the projection means. A group of images in which a luminance value is determined as a function value of a continuous function that continuously changes with a change in phase, and the luminance value is determined by changing the phase for each image, and the initial phase value along the first coordinate axis A first reference pattern image group generation step for generating a reference pattern image group related to the first coordinate axis, which is an image group continuously changed for each coordinate value;
A first imaging control step of causing the projection unit to project an image belonging to the reference pattern image group related to the first coordinate axis, and causing the imaging unit to image the image projected on the projection plane;
A portion corresponding to the projection area of the projection target image on the projection plane is cut out from a plurality of photographed images taken by the photographing means when the projection means projects each image of the reference pattern image group related to the first coordinate axis. A first resolution conversion step of generating a resolution-converted image obtained by converting the resolution of the image by enlarging or reducing the image;
A first phase calculation step of calculating an initial phase value at a luminance change of each pixel in the resolution conversion image from a luminance change of the pixels in the plurality of resolution conversion images;
In the first reference pattern image group generation step, the reverse calculation of the initial phase value calculated from the coordinates of the first coordinate axis is performed on the initial phase value calculated in the first phase calculation step, and the first coordinate axis obtained by the reverse calculation is calculated. A first associating step for associating coordinates and pixels of a resolution-converted image;
A group of images in which a luminance value is determined as a function value of a continuous function that continuously changes with a change in phase, and the luminance value is determined by changing the phase for each image, and the initial phase value along the second coordinate axis A second reference pattern image group generation step for generating a reference pattern image group related to the second coordinate axis, which is an image group continuously changed for each coordinate value;
A second imaging control step of causing the projection unit to project an image belonging to a reference pattern image group related to the second coordinate axis, and causing the imaging unit to capture an image projected on the projection plane;
A portion corresponding to the projection area of the projection target image on the projection plane was cut out from a plurality of photographed images taken by the photographing means when the projection means projected each image of the reference pattern image group related to the second coordinate axis. A second resolution conversion step of generating a resolution-converted image obtained by converting the resolution of the image by enlarging or reducing the image;
A second phase calculating step of calculating an initial phase value at a luminance change of each pixel in the resolution converted image from a luminance change of the pixels in the plurality of resolution converted images;
In the second reference pattern image group generation step, reverse calculation of the initial phase value calculated from the coordinates of the second coordinate axis is performed on the initial phase value calculated in the second phase calculation step, and the second coordinate axis obtained by the reverse calculation is calculated. A second associating step for associating coordinates with pixels of a resolution-converted image;
A pixel position correspondence specifying method comprising: a pixel association step for specifying a correspondence between a pixel in a projection target image and a pixel in the projection unit using a correspondence between a pixel and a coordinate of a resolution-converted image. . A group of images in which a luminance value is determined as a function value of a continuous function that continuously changes with a change in phase, and the luminance value is determined by changing the phase for each image, and the initial phase value along the first coordinate axis A first reference pattern image group generation step for generating a reference pattern image group related to the first coordinate axis, which is an image group continuously changed for each coordinate value;
A first reference pattern projection control step of causing the projection means to project an image belonging to the reference pattern image group related to the first coordinate axis;
A portion corresponding to the projection area of the projection target image on the projection plane is cut out from a plurality of shot images obtained by shooting each image of the reference pattern image group related to the first coordinate axis projected on the projection plane, and the cut out image is obtained. A first resolution conversion step of generating a resolution-converted image obtained by converting the resolution of the image by enlarging or reducing;
A first phase calculation step of calculating an initial phase value at a luminance change of each pixel in the resolution conversion image from a luminance change of the pixels in the plurality of resolution conversion images;
In the first reference pattern image group generation step, the reverse calculation of the initial phase value calculated from the coordinates of the first coordinate axis is performed on the initial phase value calculated in the first phase calculation step, and the first coordinate axis obtained by the reverse calculation is calculated. A first associating step for associating coordinates and pixels of a resolution-converted image;
A group of images in which a luminance value is determined as a function value of a continuous function that continuously changes with a change in phase, and the luminance value is determined by changing the phase for each image, and the initial phase value along the second coordinate axis A second reference pattern image group generation step for generating a reference pattern image group related to the second coordinate axis, which is an image group continuously changed for each coordinate value;
A second reference pattern projection control step for causing the projection means to project an image belonging to a reference pattern image group related to a second coordinate axis;
A portion corresponding to the projection area of the projection target image on the projection plane is cut out from a plurality of captured images obtained by shooting each image of the reference pattern image group related to the second coordinate axis projected on the projection plane, and the cut out image is obtained. A second resolution conversion step of generating a resolution-converted image obtained by converting the resolution of the image by enlarging or reducing;
A second phase calculating step of calculating an initial phase value at a luminance change of each pixel in the resolution converted image from a luminance change of the pixels in the plurality of resolution converted images;
In the second reference pattern image group generation step, reverse calculation of the initial phase value calculated from the coordinates of the second coordinate axis is performed on the initial phase value calculated in the second phase calculation step, and the second coordinate axis obtained by the reverse calculation is calculated. A second associating step for associating coordinates with pixels of a resolution-converted image;
A pixel position correspondence specifying method comprising: a pixel association step for specifying a correspondence between a pixel in a projection target image and a pixel in the projection unit using a correspondence between a pixel and a coordinate of a resolution-converted image. . Generating a resolution-converted image of the image by reducing the image cut out from the captured image in the first resolution conversion step;
The pixel position correspondence specifying method according to claim 10 or 11, wherein, in the second resolution conversion step, a resolution conversion image of the image is generated by reducing an image cut out from the captured image. A correction step of correcting the projection target image based on the correspondence between the pixel in the projection target image and the pixel in the projection unit;
The pixel position correspondence specifying method according to claim 10, further comprising: a projection control step of projecting the image corrected in the correction step onto the projection unit. A pixel position correspondence specifying program installed in a computer including a projecting unit that projects an image on a projection surface, and a photographing unit that captures an image projected on the projection surface,
In the computer,
A group of images in which a luminance value is determined as a function value of a continuous function that continuously changes with a change in phase, and the luminance value is determined by changing the phase for each image, and the initial phase value along the first coordinate axis First reference pattern image group generation processing for generating a reference pattern image group related to the first coordinate axis, which is an image group continuously changed for each coordinate value;
A first imaging control process in which an image belonging to a reference pattern image group relating to a first coordinate axis is projected onto the projection unit, and as a result, an image projected on the projection plane is captured by the imaging unit;
A portion corresponding to the projection area of the projection target image on the projection plane is cut out from a plurality of photographed images taken by the photographing unit when the projection unit projects each image of the reference pattern image group related to the first coordinate axis. A first resolution conversion process for generating a resolution-converted image obtained by converting the resolution of the image by enlarging or reducing the image;
A first phase calculation process for calculating an initial phase value at a luminance change of each pixel in the resolution-converted image from a luminance change of the pixels in the plurality of resolution-converted images;
In the first reference pattern image group generation process, a reverse calculation for calculating the initial phase value from the coordinates of the first coordinate axis is performed on the initial phase value calculated in the first phase calculation process, and the first coordinate axis obtained by the reverse calculation is calculated. A first association process for associating coordinates with pixels of a resolution-converted image;
A group of images in which a luminance value is determined as a function value of a continuous function that continuously changes with a change in phase, and the luminance value is determined by changing the phase for each image, and the initial phase value along the second coordinate axis Second reference pattern image group generation processing for generating a reference pattern image group related to the second coordinate axis, which is an image group continuously changed for each coordinate value;
A second imaging control process in which an image belonging to a reference pattern image group relating to a second coordinate axis is projected onto the projection unit, and as a result, an image projected on the projection plane is captured by the imaging unit;
A portion corresponding to the projection area of the projection target image on the projection plane was cut out from a plurality of photographed images taken by the photographing means when the projection means projected each image of the reference pattern image group related to the second coordinate axis. A second resolution conversion process for generating a resolution-converted image obtained by converting the resolution of the image by enlarging or reducing the image;
A second phase calculation process for calculating an initial phase value at a luminance change of each pixel in the resolution-converted image from a luminance change of pixels in the plurality of resolution-converted images;
In the second reference pattern image group generation process, the calculation for obtaining the initial phase value from the coordinates of the second coordinate axis is performed on the initial phase value calculated in the second phase calculation process, and the second coordinate axis obtained by the reverse calculation is calculated. A second associating process for associating coordinates with pixels of a resolution-converted image; and
A pixel position correspondence specifying program for executing a pixel matching process for specifying a correspondence between a pixel in a projection target image and a pixel in the projection unit using a correspondence between a pixel and a coordinate of a resolution-converted image. A pixel position correspondence specifying program installed in a computer including a projection unit that projects an image onto a projection plane,
In the computer,
A group of images in which a luminance value is determined as a function value of a continuous function that continuously changes with a change in phase, and the luminance value is determined by changing the phase for each image, and the initial phase value along the first coordinate axis First reference pattern image group generation processing for generating a reference pattern image group related to the first coordinate axis, which is an image group continuously changed for each coordinate value;
A first reference pattern projection control process for causing the projection means to project an image belonging to a reference pattern image group related to a first coordinate axis;
A portion corresponding to the projection area of the projection target image on the projection plane is cut out from a plurality of shot images obtained by shooting each image of the reference pattern image group related to the first coordinate axis projected on the projection plane, and the cut out image is obtained. A first resolution conversion process for generating a resolution-converted image obtained by converting the resolution of the image by enlarging or reducing;
A first phase calculation process for calculating an initial phase value at a luminance change of each pixel in the resolution-converted image from a luminance change of the pixels in the plurality of resolution-converted images;
In the first reference pattern image group generation process, a reverse calculation for calculating the initial phase value from the coordinates of the first coordinate axis is performed on the initial phase value calculated in the first phase calculation process, and the first coordinate axis obtained by the reverse calculation is calculated. A first association process for associating coordinates with pixels of a resolution-converted image;
A group of images in which a luminance value is determined as a function value of a continuous function that continuously changes with a change in phase, and the luminance value is determined by changing the phase for each image, and the initial phase value along the second coordinate axis Second reference pattern image group generation processing for generating a reference pattern image group related to the second coordinate axis, which is an image group continuously changed for each coordinate value;
A second reference pattern projection control process for causing the projection means to project an image belonging to the reference pattern image group related to the second coordinate axis;
A portion corresponding to the projection area of the projection target image on the projection plane is cut out from a plurality of shot images obtained by shooting each image of the reference pattern image group related to the second coordinate axis projected on the projection plane, and the cut-out image is enlarged. Or a second resolution conversion process for generating a resolution-converted image obtained by converting the resolution of the image by reducing,
A second phase calculation process for calculating an initial phase value at a luminance change of each pixel in the resolution-converted image from a luminance change of pixels in the plurality of resolution-converted images;
In the second reference pattern image group generation process, the calculation for obtaining the initial phase value from the coordinates of the second coordinate axis is performed on the initial phase value calculated in the second phase calculation process, and the second coordinate axis obtained by the reverse calculation is calculated. A second associating process for associating coordinates with pixels of a resolution-converted image; and
A pixel position correspondence specifying program for executing a pixel matching process for specifying a correspondence between a pixel in a projection target image and a pixel in the projection unit using a correspondence between a pixel and a coordinate of a resolution-converted image. On the computer,
In the first resolution conversion process, a resolution-converted image of the image is generated by reducing the image cut out from the captured image,
The pixel position correspondence specifying program according to claim 14 or 15, wherein, in the second resolution conversion process, a resolution conversion image of the image is generated by reducing an image cut out from the captured image. On the computer,
Correction processing for correcting the projection target image based on the correspondence between the pixels in the projection target image and the pixels in the projection unit; and
The pixel position correspondence specifying program according to any one of claims 14 to 16, wherein a projection control process for projecting the image corrected by the correction process onto the projection unit is executed.

Images