JP2010283674A - Projection system and projection method - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To easily and accurately correct image distortion even if a projection plane has recesses and protrusions, or a projection apparatus and the projection plane do not face each other. <P>SOLUTION: A projection control device 3 creates first conversion information by associating an image captured by a camera 2 with a projection image of a projector 1 using a phase shift method, sets an image projection region on the image captured by the camera 2 provided at a predetermined viewpoint position, and converts the image projection region on the captured image into an image projection region on the projection image using the first conversion information. A content image region indicating the shape of a content image is set to create second conversion information based on a relationship between the image projection region on the converted projection image and the content image region. In content image projection, the content image data are converted using the second conversion information. <P>COPYRIGHT: (C)2011,JPO&INPIT

Description

  The present invention relates to a projection system and a projection method for projecting light.

  2. Description of the Related Art Conventionally, there have been increasing opportunities to perform presentations by projecting a personal computer screen using a projector as a video projection device.

  In addition to the widespread use of projectors and personal computers, in addition to projecting a dedicated screen as a projection surface, there is an increasing use of using a conference room wall as a projection surface. Thus, when projecting an image on a screen other than a dedicated screen, the projection surface may be uneven. If the projection surface is uneven, the projected image may be distorted and the image may be difficult to see. Further, when projecting an image from a projector in a small room, it is difficult to install the projector so as to be orthogonal (facing) to the projection plane. In this case, it is generally necessary to perform trapezoidal correction or the like on the projected image.

  Furthermore, there are cases where the projection plane cannot be set in front of the person who sees the projection surface and the person who sees the projection image must see the projection plane from an oblique direction. In addition, when a plurality of people view the same projection plane, there is a possibility that image distortion may be significant depending on the viewer.

  Even in such a situation, in order to provide an appropriate projection image to the viewer, the three-dimensional shape of the projection plane is measured, the image is corrected based on the measured projection plane shape, and the image distortion is reduced. There has been no projection.

  For example, Patent Document 1 describes that a three-dimensional shape of a projection surface is obtained by the principle of triangulation using the light projection method, and a corrected image is obtained by back projection simulation.

  In Patent Document 2, a test image is projected and imaged on a projection surface, and the image is corrected by obtaining a distortion amount based on a deviation between the captured image and the projected image. With the technique described in Patent Document 2, a good correction amount can be obtained in a portion where a lattice point of a test image exists.

JP 2004-140845 A JP 2001-83949 A

  However, the technique described in Patent Document 1 requires a large amount of calculation in order to correct an image. In addition, in the technique described in Patent Document 1, in order to accurately measure a three-dimensional shape, a calibration process for obtaining internal parameters and external parameters of a camera or a projector is separately required. It is difficult to set.

  In addition, the technique described in Patent Document 2 has a problem in that an accurate correction amount cannot be obtained in a portion where a grid point of a test image cannot be projected. In addition, when trying to set lattice points at all points on which an image is projected, it is necessary to repeat the projection of the test image, imaging, and calculation of the correction amount many times, which requires a long work time.

  Therefore, the present invention has been proposed in view of the above-described circumstances, and it is easy to correct image distortion even when the projection surface is uneven or the projection device and the projection surface are not facing each other. It is another object of the present invention to provide a projection system and a projection method that can be performed with high accuracy.

  A projection system according to the present invention includes a projection device that projects a projection image using content image data, and an imaging device that captures a projection image projected by the projection device and acquires a captured image. In order to solve the above-described problem, the projection system projects and images a fringe pattern whose brightness periodically changes in a first direction orthogonal to the light projecting direction of the projection device, and phase shifts in the first direction. The captured image and the projected image are associated with each other using a method, and a fringe pattern whose brightness changes periodically in a second direction orthogonal to the first direction is projected and imaged to perform a phase shift method in the second direction. A first conversion information creating means for creating first conversion information defining a correspondence relationship between pixels of the captured image and the projected image by using the captured image and the projected image in association with each other; A projection area setting unit that sets an image projection area on a captured image captured by the imaging apparatus provided at a viewpoint position, and an image projection area on the captured image that is set by the projection area setting unit are converted into the first change. Using the information, an area conversion means for converting to an image projection area on the projection image, a content image area setting means for setting a content image area indicating the shape of the content image, and setting on the captured image by the projection area setting means Based on the relationship between the image projection area thus set and the content image area set by the content image area setting means, the content image data is converted so as to project the content image area onto the image projection area on the captured image. Second conversion information creating means for creating second conversion information, and image conversion means for converting content image data using the second conversion information and projecting a projection image from the projection device.

  In the projection system according to the present invention, the imaging device is provided at a viewpoint position of an observer who views the projection image or an arbitrary position around the projection device.

  In the projection system according to the present invention, the projection area setting means detects the image projection area from the captured image by predetermined image processing.

  In the projection system according to the present invention, the projection area setting means detects the position of a mark attached to the image projection area of the projection device and obtains the vertex of the image projection area.

  In the projection system according to the present invention, the first conversion information creating unit corrects the association between the captured image and the projection image performed using the phase shift method using a spatial encoding method, and First conversion information is created.

  In order to solve the above-described problem, the projection method according to the present invention projects and captures a fringe pattern whose brightness changes periodically in a first direction orthogonal to the light projecting direction of the projection apparatus, and performs the imaging in the first direction. A phase shift method is used to associate a captured image with a projected image, and a fringe pattern whose brightness changes periodically in a second direction orthogonal to the first direction is projected and imaged to shift the phase in the second direction. The first conversion information defining the correspondence between the captured image and the projected image is created in advance by associating the captured image with the projected image using a method, and is provided at a predetermined viewpoint position An image projection area is set on the captured image captured by the imaging device, and the image projection area on the set captured image is converted into an image projection area on the projection image using the first conversion information. , Content image shape The content image area to be shown is set, and the content image area is projected onto the image projection area on the captured image based on the relationship between the image projection area on the converted captured image and the set content image area Thus, the second conversion information for converting the content image data is created. In this projection method, when the content image is actually projected, the content image data is converted using the second conversion information, and the projection image is projected from the projection device.

  According to the projection system and the projection method of the present invention, based on the first conversion information that defines the correspondence between the captured image and the projected image using the phase shift method, and the relationship between the captured image and the content image. The second conversion information is created, the content image data is converted by the second conversion information, and the content image can be projected from the projection device onto the projection plane. Even when the projection apparatus and the projection plane are not directly facing each other, it is possible to easily and accurately correct image distortion.

  Further, according to the projection system of the present invention, the content image data is obtained using the first conversion information and the second conversion information regardless of whether or not the viewpoint position of the observer who views the projection image faces the projection plane. By converting, it is possible to project a content image without distortion when viewed from the observer's position.

  Furthermore, according to the projection system of the present invention, the image projection area is detected from the captured image by predetermined image processing, and the vertex of the image projection area is obtained from the detected image projection area. You can set the area.

  Further, in the projection system according to the present invention, the position of the mark attached to the image projection area of the projection device is detected and the vertex of the image projection area is obtained, so that a desired image projection area can be automatically set.

  Further, in the projection system according to the present invention, the first conversion information is generated by correcting the association between the captured image and the projection image performed using the phase shift method using the spatial coding method. Even when the unevenness of the surface is large, a content image without distortion can be projected.

It is a block diagram which shows the structure of the projection system shown as 1st Embodiment of this invention. It is a block diagram which shows the structure of the projection control apparatus in the projection system shown as 1st Embodiment of this invention. In the projection system shown as 1st Embodiment of this invention, it is a figure which shows the positional relationship of a viewpoint position, a projector, a camera, and a projection surface. It is a figure which shows the relationship between a captured image, a projection image, and a content image in the projection system shown as 1st Embodiment of this invention. It is a figure which shows the relationship between a captured image and a projection image in the projection system shown as 1st Embodiment of this invention. It is a figure which shows the conversion table Tcb in the projection system shown as 1st Embodiment of this invention. It is a figure which shows the correspondence of a captured image and a content image in the projection system shown as 1st Embodiment of this invention. It is a flowchart which shows the projection operation | movement in the projection system shown as 1st Embodiment of this invention. It is a flowchart which shows the operation | movement which produces the conversion table Tcb in the projection system shown as 1st Embodiment of this invention. 5 is a flowchart showing an operation of taking a correspondence between a captured image and a projected image in a first direction in the projection system shown as the first embodiment of the present invention. 5 is a flowchart showing an operation of taking a correspondence between a captured image and a projected image in a second direction in the projection system shown as the first embodiment of the present invention. It is a flowchart which shows the detailed operation | movement which produces the conversion table Tcb in the projection system shown as 1st Embodiment of this invention. 5 is a flowchart showing a distortion transformation matrix creation operation in the projection system shown as the first embodiment of the present invention. In the projection system shown as 1st Embodiment of this invention, it is a block diagram which shows the structure which takes a response | compatibility with a captured image and a projection image in a 1st direction. It is a block diagram which shows the structure which takes a response | compatibility with a captured image and a projection image in a 2nd direction in the projection system shown as 1st Embodiment of this invention. It is a block diagram explaining the principle of a phase shift method in the projection system shown as 1st Embodiment of this invention. It is a figure which shows the fringe pattern projected by the phase shift method in the projection system shown as 1st Embodiment of this invention. It is a figure which shows the projection image projected by the spatial coding method in the projection system shown as 1st Embodiment of this invention. It is a figure explaining the process which calculates | requires an absolute phase by unwrapping a phase in the projection system shown as 1st Embodiment of this invention. It is a figure which shows the positional relationship of a viewpoint position, a projector, a camera, and a projection surface in the projection system shown as 2nd Embodiment of this invention. It is a figure which shows the relationship between a captured image, a projection image, and a content image in the projection system shown as 2nd Embodiment of this invention. It is a flowchart which shows the projection operation | movement in the projection system shown as 2nd Embodiment of this invention. It is a flowchart which shows the production | generation operation | movement of a distortion transformation matrix in the projection system shown as 2nd Embodiment of this invention. It is a figure which shows the positional relationship of a viewpoint position, a projector, a camera, and a projection surface in the projection system shown as 3rd Embodiment of this invention. It is a flowchart which shows the production | generation operation | movement of a distortion transformation matrix in the projection system shown as 3rd Embodiment of this invention. In the projection system shown as 4th Embodiment of this invention, it is a figure which shows the positional relationship of a viewpoint position, a projector, a camera, and a projection surface. It is a flowchart which shows the production | generation operation | movement of a distortion transformation matrix in the projection system shown as 4th Embodiment of this invention.

  Hereinafter, embodiments of the present invention will be described with reference to the drawings.

[First Embodiment]
[Schematic configuration of lighting system]
The projection system shown as the first embodiment of the present invention includes a projector 1 as a projection device 1, a camera 2 as an imaging device, and a projection control device 3, as shown in FIG. In this projection system, the projection area of the projection image of the projector 1 and the imaging area of the camera 2 overlap, and the projection plane 10 as an irradiation target is included in the overlapped area.

  The projection surface 10 may have any shape. Even when the projection surface 10 has irregularities, the projection system can project an image without distortion onto the projection surface 10 including the irregularities. The projection surface 10 may be a wall surface of a room close to a plane or a projection surface that is concave.

  The projection control device 3 in this projection system is configured as shown in FIG. 2, for example.

  The projection control device 3 includes an image input unit 21, an input image memory 22, a calculation unit 23, a memory 24, a storage device 25, an I / O control unit 26, an output image memory 27, and an image output unit 28. The projection control device 3 is connected to a display device 4, a keyboard 5, and a mouse 6 as user operation interfaces for creating a projection image.

  In the projection control device 3, the captured image captured by the camera 2 is input by the image input unit 21, digitized, and stored in the input image memory 22. Here, the projection control device 3 can acquire a captured image by supplying a command to the camera 2 via a signal line (not shown) and controlling the imaging operation of the camera 2.

  Further, the projection control device 3 stores the projection image created by the calculation unit 23 in the output image memory 27 and causes the projector 1 to output it from the image output unit 28. Thereby, the projection control apparatus 3 projects a projection image from the projector 1 onto the projection area.

  Further, the calculation unit 23 creates a projection image using the memory 24 as a work area for creating a projection image. At this time, the calculation unit 23 creates a projection image using the projection pattern, projection sequence, content image data, conversion table Tcb, and distortion conversion matrix H stored in the storage device 25. The projection pattern is a fringe pattern image for measurement projected onto the projection plane 10. The content image data is various image data and video data projected on the projection plane 10. For example, the content image data includes presentation image data. The projection sequence is data indicating the projection pattern and the projection order of the content image data. The selection of the projection pattern, content image data, and projection sequence can be set by the calculation unit 23 by an operation input signal from the I / O control unit 26 in response to the operation of the keyboard 5 and the mouse 6.

  Furthermore, in order to project content image data without distortion on the projection plane 10, the calculation unit 23 converts a captured image into a projected image, a conversion table Tcb (first conversion information), and converts the content image data into a captured image. A distortion transformation matrix H (second transformation information) to be calculated is calculated. Then, the calculation unit 23 stores the conversion table Tcb and the distortion conversion matrix H in the storage device 25.

  The projection system shown as 1st Embodiment makes the same position as an observer's viewpoint position the imaging position of the camera 2, as shown in FIG. Then, the projection system makes it possible to view the content image without distortion when viewing the content image projected onto the projection plane 10 from the imaging position of the camera 2.

  The projected image having no distortion when viewed from the image pickup position of the camera 2 is shown in FIG. 4 at the vertices C1, C2, C3, and C4 of the image projection area 111 that is a rectangular area in the picked-up image 101 shown in FIG. In (c), the vertices I1, I2, I3, and I4 of the content image region 113 in the content image 103 are projected in a consistent manner. Here, the content image area 113 shows the shape in the content image 103, but it may be the same as the entire area of the content image 103. On the other hand, the viewpoint position of the observer and the installation position of the projector 1 are inevitably different. For this reason, the shape of the projection surface 10 on the projection image 102 in FIG. 4B and the projection surface 10 on the captured image 101 are different. First, conversion information (conversion table Tcb) from the captured image 101 to the projected image 102 is obtained. Next, the content image area 113 on the content image 103 is set (content image area setting means). Next, conversion information (distortion conversion matrix H) from the content image 103 to the captured image 101 is obtained. Therefore, the content image area 113 is converted into the captured image area 111 by the distortion conversion matrix H, and the converted captured image area 111 is converted into the projected image area 112 using the conversion table Tcb. By projecting the projection image thus created, as a result, it is possible to see the content image projected on the projection plane 10 without distortion from the imaging position of the camera 2, that is, the viewpoint position of the observer.

  The projection system calculates the relationship between the vertices C1, C2, C3, and C3 of the image projection area 111 in the captured image 101 and the vertices P1, P2, P3, and P4 of the image projection area 112 in the projection image 102. In the three-dimensional shape measurement, a method known as a time stripe pattern projection phase shift method is used. The process for obtaining the correspondence between the captured image 101 and the projected image 102 is to associate the position on the imaging element of the camera 2 with the position on the projection element of the projector 1.

  In this time fringe pattern projection phase shift method, projection and imaging are performed over a plurality of sheets while shifting the phase of the fringe pattern whose luminance value changes in a sine wave shape. Next, in the time-striped pattern projection phase shift method, the initial phase of the luminance value is estimated in each pixel of the image sensor from a plurality of captured images. Since this initial phase is set on the projection element, the estimated value of the initial phase obtained for each pixel of the image sensor directly represents the position on the projection element. Can be associated.

  In order to increase the accuracy of the position where the captured image and the projected image are associated with each other, if a plurality of phases are allocated on the projection element, phase connection is required at the joint of the phases at the time of estimation. There are many known phase connection methods, but the projection system of this embodiment uses a spatial encoding method that is simple in processing and highly reliable.

  In addition, the calculation unit 23 uses projective transformation to obtain the relationship between the vertices C1, C2, C3, and C4 of the rectangular area in the captured image and the areas I1, I2, I3, and I4 of the content image.

  Specifically, the projection control device 3 projects a fringe pattern whose brightness periodically changes in a first direction orthogonal to the light projecting direction of the projector 1 and performs correspondence using the phase shift method in the first direction. Do. In addition, the projection control device 3 projects a fringe pattern whose brightness periodically changes in a second direction orthogonal to the first direction within a plane orthogonal to the light projecting direction, and uses the phase shift method in the second direction. Perform correspondence. Thereby, the projection control apparatus 3 calculates | requires the conversion table Tcb (conversion information) which defines the correspondence between the pixels of a captured image and a projection image. This conversion table Tcb is stored in the storage device 25 (first conversion information creating means).

  By using the conversion table Tcb, the projection control device 3 can perform coordinate conversion on the captured image 101 in FIG. 5A and convert it into a projection image 102 as shown in FIG. Here, each pixel of the captured image 101 acquired by the projection control device 3 is converted to a position shifted from the sampling grid point of the pixel included in the projection image 102. For this reason, in order to obtain a luminance value at each pixel of the projection image 102, it is necessary to interpolate and obtain the luminance value of the captured image 101 converted around the sampling point of each pixel of the projection image 102.

  As a method of interpolating the projection image 102 with the pixels of the captured image 101, the projection control device 3 uses a conversion table Tcb stored in the storage device 25. As shown in FIG. 6, the conversion table Tcb is configured by registering the pixel coordinates of the captured image 101 converted around each pixel of the projection image 102. Specifically, the coordinates (1, 1) of the captured image correspond to the coordinates (1, 1) and (1, 2) of the projection image.

  When creating the projection image 102, the arithmetic unit 23 averages the luminance values of the pixels of the captured image 101 used to create each pixel of the projection image 102, and the luminance values of the pixels constituting the projection image 102 are calculated. Asking. When the captured image 101 is a color image, an average value is obtained for each element of RGB, and a process for creating the projection image 102 is performed.

  Further, the calculation unit 23 sets an image projection area 111 on the captured image 101 captured by the camera 2 provided at a predetermined viewpoint position (projection area setting means). Next, the content image area 113 on the content image 103 is set (content image area setting means). Next, based on the relationship between the set content image area 113 and the image projection area 111 on the captured image 101, the calculation unit 23 performs distortion conversion that converts the content image data so as to project an image onto the image projection area. A matrix H is created. At the time of actual projection, the content image area 113 is converted into the image projection area 111 on the captured image by the distortion conversion matrix H (image conversion means), and then the image projection area 111 on the converted captured image is converted into the conversion table. Using Tcb, the image is converted into an image projection area 112 on the projection image (area conversion means), and the projection image is projected from the projector 1.

  As shown in FIG. 7, the process of converting the data in the content image area 113 into the data in the image projection area 111 on the captured image 101 using the distortion transformation matrix H created by the calculation unit 23 is performed as shown in FIG. For each of the pixels, an arithmetic operation using a distortion transformation matrix H is performed to convert it onto a content image. At the time of conversion using the distortion conversion matrix H, the captured image 101 is converted to a position shifted from the sampling grid point of the pixel of the content image 103. For this reason, in order to obtain the luminance value of each pixel of the image projection area 111 on the captured image, the calculation unit 23 interpolates using the luminance value of the content image 103 around the coordinates obtained by converting the captured image. As a luminance value interpolation method, any of a bilinear method, a bicubic method, and a method of interpolating by creating a table may be used.

  As described above, the projection system converts the content image data into the captured image data using the distortion conversion matrix H, and then converts the captured image data into the projected image data using the conversion table Tcb. An image without distortion can be projected on the surface 10.

[Projection system operation]
Next, various operations performed by the projection system configured as described above will be described with reference to the flowcharts of FIGS.

"Projection of projection pattern"
First, the projection operation of the projection pattern by the projection system will be described with reference to FIG.

  As described above, the projection system creates the conversion table Tcb and the distortion conversion matrix H and stores them in the storage device 25 as preprocessing for projecting the content image.

  Then, when the timing for projecting the content image comes due to the user's operation or the like, the processing after step S1 is started. First, the projection control device 3 reads the conversion table Tcb and the distortion conversion matrix H stored in the storage device 25 into the memory 24.

  In the next step S <b> 2, the projection control device 3 reads content image data that the user desires to be projected from the storage device 25 into the memory 24.

  In the next step S3, the projection control device 3 converts the content image data read in step S2 into a projected image using the conversion table Tcb and distortion conversion matrix H read in step S1.

  In the next step S4, the projection control device 3 outputs the projection image, which is the content image converted in step S3, to the projector 1, and causes the projector 1 to project the projection image. Thereby, the projection system can project an image without distortion by looking at the image projected on the projection plane 10 from the imaging position of the camera 2 installed when the conversion table Tcb and the distortion conversion matrix H are created.

"Conversion table Tcb creation operation"
Next, the creation operation of the conversion table Tcb will be described.

  The conversion table Tcb indicates the correspondence between the table data in which the pixels of the captured image and the pixels of the projection image are associated, that is, the position data on the image sensor and the position data on the projection element. As described above, in order to associate the captured image with the projected image, the projection system according to the present embodiment applies a method called a time-striped pattern projection phase shift method that is generally used in measurement of a three-dimensional shape.

  As shown in FIG. 14, the camera coordinate system 202 has a horizontal direction as an x-axis and a vertical direction as a y-axis, and the size of a captured image is m pixels in the x direction and n pixels in the y direction. In the projector coordinate system 201, the horizontal direction is the u axis, the vertical direction is the v axis, and the size of the projected image is k pixels in the u direction and l pixels in the v direction.

  In step S 11 shown in FIG. 9, the projection system projects a stripe pattern in the first direction (projected image 301 in the first direction) as shown in FIG. 14 from the projector 1, and images the projection area with the camera 2. As described above, the camera 2 is installed such that the viewpoint position of the user with respect to the projection plane 10 is the imaging position, as shown in FIG. And the calculating part 23 calculates | requires matching of the camera coordinate and projector coordinate along a 1st direction using the luminance value of each pixel of a captured image. Here, the coordinates along the first direction are the x axis in the camera coordinate system 202 and the u axis in the projector coordinate system 201 as shown in FIG.

  In the next step S12, the projector 1 projects a fringe pattern in the second direction (projected image 302 in the second direction) orthogonal to the first direction as shown in FIG. And the calculating part 23 calculates | requires matching with the camera coordinate and projector coordinate along a 2nd direction using the luminance value of each pixel of a captured image. Here, the coordinates along the second direction are the y axis in the camera coordinate system 202 and the v axis in the projector coordinate system 201, as shown in FIG.

  By such step S11 and step S12, the corresponding pixel in the projector coordinates for each pixel of the captured image in the camera coordinates is obtained.

  In the next step S13, the captured image necessary for calculating the luminance value of each pixel of the projection image with respect to the projector coordinates using the correspondence relationship between the captured image obtained in step S11 and step S12 and the projection image. A conversion table Tcb that associates the pixels is created. Thereby, the conversion table Tcb as shown in FIG. 6 can be created.

  As described above, the principle of the time fringe pattern projection phase shift method is used for associating the captured image with the projected image. Hereinafter, the principle of the time stripe pattern projection phase shift method will be described.

  FIG. 16 shows the arrangement of the projection element 1 a of the projector 1, the imaging element 2 a of the camera 2, and the projection plane AB of the projection plane 10 in the projection system. FIG. 16 is a schematic diagram of a cross-section taken along a plane formed by the principal point of the projector 1, the principal point of the camera 2, and the point R that is an image on the projection surface of the point P on the projection element 1a. . This cross section is a straight line AB on the projection plane of the projection plane 10. Projection light emitted from a point P on the projection element 1 a forms an image at a point R on the projection plane on the projection plane 10. The projection light reflected from the point R on the projection surface forms an image at the point Q on the image sensor 2a.

  In the time fringe pattern projection phase shift method, the position P on the projection element 1a of the projector 1 is related to the initial phase of the sinusoidal fringe pattern, and the phase is changed from 0 to 3π / 2 as shown in FIG. The stripe pattern is projected sequentially while shifting by two. In the time-striped pattern projection phase shift method, the luminance at each pixel of a captured image obtained by capturing a projection image projected onto the projection surface AB of the projection surface 10 having an arbitrary shape and deformed by the three-dimensional shape of the projection surface AB. The value is detected, and the initial phase value of the sine wave of the fringe pattern is estimated for each pixel. Next, the time fringe pattern projection phase shift method uses the same initial value from the initial phase value estimated for each pixel (position Q) on the image sensor 2a and the initial phase of the fringe pattern at the position P on the projection element 1a. A position P on the phase projection element 1a and a position Q on the imaging element 2a are obtained. Thereby, the time-striped pattern projection phase shift method specifies from which pixel P on the projection element 1a the light received by a certain pixel Q of the imaging element 2a is light.

  The initial phase is deformed according to the three-dimensional shape of the projection surface of the projection surface 10. For this reason, the initial phase value is a value representing the three-dimensional shape of the projection surface. When calculating the initial phase value, in order to sequentially project the fringe pattern while shifting the phase, the adjective “time” is attached, and this is called a time fringe pattern projection phase shift method. In the present embodiment, a method of associating a captured image with a projected image is used in the time-striped pattern projection phase shift method.

  As described above, in the projection system, when the initial phase of the fringe pattern at the position Q on the image sensor 2a can be estimated, the position P on the projection element 1a on which the estimated initial phase is projected is known. In the time fringe pattern phase shift method, it is known that the phase can be estimated by projecting a fringe pattern in which at least three phases are shifted. In the present embodiment, as shown in FIG. 17, a case where four stripe patterns whose phases are shifted from 0 by π / 2 is used is described as an example. However, the position Q on the image sensor 2a corresponding to the position P on the projection element 1a may be obtained by estimating the initial phase of each position on the image sensor 2a using three stripe patterns.

  A process for associating the captured image with the projected image in the first direction (X direction) in step S11 in FIG. 9 described above with reference to FIG. explain.

  First, in step S21, the projection control device 3 calculates a global phase by a spatial coding method using the stripe patterns of the binary pattern 1, the binary pattern 2, the binary pattern 3, and the binary pattern 4 shown in FIG. This global phase roughly represents the phase of the entire screen.

In the next step S22, the projection control device 3 sequentially projects and images the fringe pattern in the first direction using the time fringe pattern projection phase shift method, and the phase Φ ₁ for each pixel (local) on the captured image. (X) is obtained.

At this time, as shown in FIG. 17, luminance values I ₀ (x), I ₁ (x) on the picked-up image obtained by picking up stripe patterns of no shift, π / 2 shift, π shift, and 3π / 2 shift, I ₂ (x) and I ₃ (x) are represented by the following formulas 1 to 4.

In Equations 1 to 4, B (x) represents a bias component, A (x) represents an amplitude component, and Φ ₁ (x) represents an initial phase. From these four luminance values I ₀ (x), I ₁ (x), I ₂ (x), and I ₃ (x), the initial phase Φ ₁ (x) can be obtained by the following Equation 5.

In the next step S23, the projection control apparatus 3 refers to the global phase obtained in step S21, performs unwrapping processing for connecting the phases of the local phases obtained in step S22, and obtains the absolute phase Φ ₁ (x ) Is calculated.

In the time fringe pattern projection phase shift method, when there are a plurality of fringe patterns, the phase estimated in the captured image is a so-called wrapping in which there is a phase jump every 2π. For this reason, an unwrapping process for connecting the wrapped phases is required. In this unwrapping process, the local phase Φ ₁ (x) in the first direction shown in FIG. 19A and the global phase in the first direction shown in FIG. The absolute phase Φ ₁ (x) in the first direction as shown in FIG. Thereby, the projection control apparatus 3 obtains a global phase that is a rough phase over the entire captured image, and then compares the global phase with the wrapped local phase to obtain the absolute phase. Can do. The method of calculating the global phase by the spatial coding method is shown in “Seiji Iguchi, Kosuke Sato, 3D image measurement, Shogodo, pp. 80-119”.

In the next step S24, the projection control device 3 obtains the position of the u-axis component in the projection image. Here, as illustrated in FIG. 14, the u-axis component of the projection image corresponds to the first direction (x direction) of the captured image. If the number of fringes of the fringe pattern projected and imaged sequentially is M, the number of pixels in the u-axis direction of the projected image is k pixels, and therefore u on the projected image corresponding to the position of x on the captured image. Can be calculated as shown in Equation 6 below.

  As described above, the projection control device 3 can associate the coordinates of the captured image with the coordinates of the projection image in the first direction by using both the spatial coding method and the time-striped pattern projection phase shift method. .

  Further, the processing for obtaining the correspondence between the camera coordinates and the projector coordinates in the second direction in step S12 is the same as that in the first direction described above. The process of step S12 is as shown in FIG. First, in step S31, as shown in FIG. 15, the projection control apparatus 3 calculates a global phase by a spatial coding method using a fringe pattern in a direction orthogonal to the first direction.

In the next step S32, the projection control device 3 sequentially projects and images the fringe pattern in the second direction using the time fringe pattern projection phase shift method, and the phase Φ ₂ for each pixel (local) on the captured image. (Y) is obtained.

At this time, as shown in FIG. 17, luminance values I ₀ (y), I ₁ (y) on a captured image obtained by capturing a stripe pattern of no shift, π / 2 shift, π shift, and 3π / 2 shift, I ₂ (y) and I ₃ (y) are represented by the following formulas 7 to 10.

From these four luminance values I ₀ (y), I ₁ (y), I ₂ (y), and I ₃ (y), the initial phase Φ ₂ (x) can be obtained by the following Expression 11.

In the next step S33, the projection control device 3 refers to the global phase obtained in step S31, performs unwrapping processing for connecting the phases of the local phases obtained in step S32, and performs absolute phase in the second direction. Φ ₂ (x) is calculated.

In the next step S34, the projection control device 3 obtains the position of the v-axis component in the projection image. Here, as illustrated in FIG. 14, the v-axis component of the projection image corresponds to the second direction (y direction) of the captured image. Assuming that the number of fringes of the fringe pattern to be projected and imaged sequentially is M, the number of pixels in the v-axis direction of the projected image is 1 pixel, and v on the projected image corresponding to the position of y on the captured image. Can be calculated as shown in Equation 13 below.

  As described above, the projection control device 3 can associate the coordinates of the captured image with the coordinates of the projection image in the second direction by using both the spatial coding method and the time-striped pattern projection phase shift method. .

  After step S11 in FIG. 9 including step S21 to step S24 in FIG. 10 and step S12 in FIG. 9 including step S31 to step S34 in FIG. 11 described above, the projection control device 3 performs the process in FIG. By doing so, the conversion table Tcb is created in step S13.

In step S41 shown in FIG. 12, the projection control device 3 reads the coordinates (u, v) on the projection image calculated in step S24 in step S11 and step S34 in step S12. The coordinates (u, v) on this projected image are as shown in the following equation (13):

It is expressed as

  In the next step S42, the projection control device 3 rounds off the decimal point of the coordinates (u, v) on the projection image and corresponds to the coordinates (u, v) column of the conversion table Tcb having the corresponding integer part. The coordinates (x, y) on the captured image to be stored are stored. Here, the coordinates (u, v) on the projection image converted from the coordinates (x, y) on the captured image shown in Expression 13 are the projection images that are discrete sampling lattice points as shown in FIG. The position deviates from the upper coordinates. For this reason, the projection control apparatus 3 needs to interpolate the luminance value of the captured image converted around the sampling point of each pixel of the projection image to obtain the luminance value at each pixel of the projection image.

  Therefore, in the conversion table Tcb shown in FIG. 6, the coordinates on the plurality of captured images are stored in the column of the coordinates (x, y) on the captured image corresponding to the coordinates (u, v) on the projection image. . When actually converting the coordinates on the captured image to the coordinates on the projected image, the pixels on the one or more captured images can be averaged to obtain the luminance value on the projected image.

  In the next step S43, the projection control device 3 determines whether or not the processing of step S41 and step S42 has been performed over the entire surface of the captured image. If the processes of step S41 and step S42 are performed over the entire surface of the captured image, the process ends. If not, the processes of step S41 and step S42 are performed.

  Thereby, the projection control apparatus 3 can create the conversion table Tcb and store it in the storage device 25.

"Creation operation of distortion transformation matrix H"
Next, an operation of creating a distortion transformation matrix H for converting a content image into a projection image will be described with reference to FIG.

  First, in step S51, the projection system images the projection area of the projection plane 10 with the camera 2 according to the control of the projection control device 3, and acquires the captured image. As described above, the camera 2 is installed such that the viewpoint position of the user with respect to the projection plane 10 is the imaging position, as shown in FIG.

  In the next step S52, the projection control device 3 sets an area on which the content image is to be projected on the image of the projection plane in the captured image acquired in step S51. The area where the content image is to be projected is an image projection area 111 in FIG. At this time, the projection control device 3 sets a vertex position that identifies the image projection area 111 on the captured image. At this time, the projection control device 3 displays the captured image captured by the camera 2 on the display device 4, and the four vertices C1, C2, C3, and C4 designated on the captured image by operating the keyboard 5 and the mouse 6 are displayed. Recognize For example, the keyboard 5 and the mouse 6 may be operated so as to overwrite the rectangle on the captured image, and the vertices of the rectangle may be recognized, or only the four vertices of the image projection area may be selected by the mouse 6. .

Note that the vertices of the rectangular image projection area 111 set in the captured image are represented by C1 (x ₁ , y ₁ ), C2 (x ₂ , y ₂ ), C3 (x ₃ , y ₃ ), C4 (x ₄ , y _4). ), And apexes of the image projection area 112 on the projection image (the image projection area 112 in FIG. 4) are P1 (u ₁ , v ₁ ), P2 (u ₂ , v ₂ ), P3 (u ₃ , v _3). ), P4 (u ₄ , v ₄ ), and the vertices of the content image area 113 are I1 (s ₁ , t ₁ ), I2 (s ₂ , t ₂ ), I3 (s ₃ , t ₃ ), I4 (s ₄ , t ₄ ).

  In the next step S53, the projection control device 3 converts from the relationship between the vertices C1, C2, C3, and C4 of the image projection area 111 on the captured image and the vertices I1, I2, I3, and I4 of the content image area 113. Information (distortion transformation matrix H) is obtained. This distortion transformation matrix H is a so-called projective transformation matrix. Therefore, the distortion transformation matrix H is obtained as follows. This distortion transformation matrix H is used for converting content image data into a captured image. However, in order to simplify the processing at the time of using the distortion transformation matrix H, in the following, obtaining the distortion transformation matrix H as a transformation from a captured image to a content image will be described.

Since the projective transformation matrix is expressed by an equivalence relation and the difference of constant multiples has no meaning, it can be expressed as the following Expression 14. It should be noted that “˜” in the expression 14 represents an equivalence relation, which means that they are equal by allowing a constant multiple difference.

In Expression 14, s and t indicate the coordinates of the content image 103, and x and y indicate the coordinates of the captured image 101. This shows that the image projection area 111 on the captured image can be converted into the content image area 113 by the distortion conversion matrix H composed of h ₁₁ to h _{32 in} Expression 14. That is, the projective transformation matrix of the above equation 14 becomes the distortion transformation matrix H, and the distortion transformation matrix H is defined as the following equation 15.

If the coordinates (s, t) of the content image area 113 are expressed using the distortion transformation matrix H of the above equation 15 and the coordinates (x, y) of the image projection area 111 on the captured image, s and t are not the same. The following expressions are corrected and expressed as Expressions 16 and 17 below.

When the above Expression 16 and Expression 17 are expanded, the operation expression of Expression 18 is obtained for s, and the operation expression of Expression 19 is obtained for t.

If the arithmetic expressions of the above expressions 18 and 19 are rearranged, s is expressed by the following expression 20 and t is expressed by the following expression 21.

When the above formulas 20 and 21 are represented by determinants, the following formula 22 is obtained.

Vertex C1 (x ₁ , y ₁ ), C2 (x ₂ , y ₂ ), C3 (x ₃ , y ₃ ), C4 (x ₄ , y ₄ ) of the image projection area 111 on the captured image, and the content image area 113 vertices of _{I1 (s 1, t 1)} , I2 (s 2, t 2), I3 (s 3, t 3), I4 (s 4, t 4) corresponding four coordinates the expression of the vertices of 22

Equation 23 is obtained. Therefore, a transformation matrix that defines the distortion transformation matrix H is obtained as shown in Equation 24.

  When the correspondence relationship between the captured image and the content image is determined using the above equation 14, the correspondence relationship between the captured image 101 and the content image 103 can be defined as shown in FIG. When the distortion transformation matrix H is calculated in this way, the projection control device 3 stores the distortion transformation matrix H in the storage device 25. When the projection system actually projects the content image on the projection plane 10, the projection system converts the content image data into captured image data using the distortion transformation matrix H of Expression 14, and then converts the captured image data. The projection image data is converted into projection image data using the table Tcb, and is supplied to the projector 1.

  As described in detail above, according to the projection system to which the present invention is applied, the first stripe pattern whose brightness changes periodically in the first direction orthogonal to the light projecting direction of the projector 1 is projected and imaged. The captured image and the projected image are associated with each other using the phase shift method for the direction, and the fringe pattern whose brightness changes periodically in the second direction orthogonal to the first direction is projected and imaged, and the phase in the second direction The captured image and the projection image are associated with each other using the shift method. Thereby, the conversion table Tcb (first conversion information) that defines the correspondence between the captured image and the projected image is created. Next, the projection system sets an image projection area 111 as a projection area on the captured image 101 captured by the camera 2 provided at a predetermined viewpoint position as shown in FIG. 3 (S51, S52). Next, a content image area 113 on the content image 103 is set. Next, based on the relationship between the set content image region 113 and the image projection region 111 on the captured image 101, a distortion transformation matrix H (first step) for converting the content image data so as to project an image onto the image projection region. 2 conversion information). At the time of actual projection, the content image area 113 is converted into the image projection area 111 on the captured image by the distortion transformation matrix H, and then the image projection area 111 on the converted captured image is converted using the conversion table Tcb. The image is converted into the image projection area 112 on the projection image, and the content image is finally projected from the projector 1 onto the projection plane 10.

In the conventional projection system, if the projection surface 10 has irregularities, the content image will appear distorted by the observer. However, if this projection system is used, the projection surface will have an irregular shape using the phase shift method. A corresponding conversion table Tcb is created, and the image projection region 111 on the captured image obtained by converting the content image using the distortion conversion matrix H is converted. No content image can be projected. Therefore, according to this projection system, even when the projection surface 10 is uneven or when the projector 1 and the projection surface 10 do not face each other, distortion correction of the image can be performed easily and accurately. According to the projection system shown as the first embodiment, the correspondence between the captured image and the projection image performed using the phase shift method is corrected using the spatial coding method, and the conversion table Tcb is created. Therefore, a phase image without phase jump can be created, and the correspondence between the captured image and the projected image can be made highly accurate. Therefore, according to the projection system, it is possible to project a content image without distortion even when the unevenness degree of the projection surface 10 is large.

[Second Embodiment]
Next, a projection system shown as the second embodiment will be described. In addition, about the part similar to the above-mentioned 1st Embodiment, the detailed description is abbreviate | omitted by attaching | subjecting the same code | symbol.

  In the projection system shown as the second embodiment, as shown in FIG. 20, an observer who sees a projection image faces the projection plane 10, and a content image projected onto the projection plane 10 from a position facing the projection plane 10. When the image is viewed, a projection image without distortion is provided. The projected image without distortion when viewed from the position facing the projection plane 10 is the vertices S1, S2, S3, and S4 of the rectangular image on which the image on the projection plane 10 is to be projected, and the vertices I1 and I2 of the content image , I3, and I4 are projected in coincidence. In the following description, for example, a case where the vertices S1, S2, S3, and S4 of the projection plane 10 are set as four points that can be arbitrarily selected by a user on a surface such as a wall will be described.

  For this purpose, as shown in FIG. 21, the content image area 113 is placed on the vertices C1, C2, C3, and C4 of the rectangular area in the captured image 101 corresponding to the vertices S1, S2, S3, and S4 in the projection plane 10. It is necessary to project the vertices I1, I2, I3, and I4 in agreement.

  On the other hand, in the projection system, the imaging position of the camera 2 and the projection position of the projector 1 are different. For this reason, the projection plane 10 on the projection image 102 and the projection plane 10 on the captured image 101 have different shapes. In order for an observer to view a content image without distortion from a position directly facing the projection plane 10, the content image region 113 may be projected in alignment with the projection regions S1, S2, S3, and S4. In order to project in this way, the relationship between the vertices S1, S2, S3, S4 of the projection area on the projection plane and the vertices C1, C2, C3, C4 of the image projection area on the captured image, the image on the captured image The relationship between the vertices C1, C2, C3, and C4 of the projection area and the vertices I1, I2, I3, and I4 of the content image area 113 is known, and each pixel of the image projection area 111 on the captured image is converted into an image projection area on the projection image. If it can be associated with each pixel 112, the projection system can project a content image without distortion when viewed from the position facing the projection plane 10.

  The relationship between the vertices S1, S2, S3, and S4 of the projection plane 10 and the vertices C1, C2, C3, and C4 of the image projection area 111 in the captured image 101 corresponding to the vertices S1, S2, S3, and S4 is first imaged. It is obtained by reading the coordinates of the vertices C1, C2, C3, C4 of the image projection area 111 in the image 101. Next, the relationship between the image projection area 111 in the captured image 101 and the image projection area 112 in the projection image 102 is obtained by using the time-striped pattern projection phase shift method described in the first embodiment, and is obtained from the conversion table Tcb. Defined. Next, the relationship between the vertices I1, I2, I3, and I4 of the content image area 113 in the content image 103 and the vertices C1, C2, C3, and C4 of the image projection area 111 on the captured image will be described in the first embodiment. The transformation matrix obtained and calculated using the concept of the projective transformation is defined as a distortion transformation matrix Q.

[Projection system operation]
Next, various operations performed by the projection system configured as described above will be described.

"Projection of projection pattern"
The projection system performs an operation as shown in FIG. 22 when projecting a content image in a state where the observer is actually facing.

  First, in step S <b> 61, the projection control device 3 reads the conversion table Tcb and distortion conversion matrix Q stored in the storage device 25 into the memory 24. This distortion transformation matrix Q is for converting the content image data in the same manner as the distortion transformation matrix H in the first embodiment described above. This distortion transformation matrix Q distorts the content image so that the content image can be observed without distortion when the content image projected onto the projection surface 10 is viewed from an observer facing the projection surface 10. .

  In the next step S <b> 62, the projection control device 3 reads the content image from the storage device 25 into the memory 24.

  In the next step S63, the projection control device 3 converts the content image read in step S62 into a projection image using the conversion table Tcb and the distortion conversion matrix Q read in step S61. Thereby, the projection control device 3 creates a projection image to be projected from the projector 1.

  In the next step S64, the projection control device 3 outputs the projection image, which is the content image converted in step S63, to the projector 1. As a result, the projection image is projected from the projector 1 onto the projection plane 10.

"Distortion transformation matrix Q creation operation"
Next, the operation of creating the distortion transformation matrix Q by the projection system will be described with reference to FIG.

  First, in step S71, the projection system images an area including the projection area of the projection plane 10 with the camera 2, and acquires a captured image. Here, the imaging area by the camera 2 and the projection area where a content image such as a wall of a room is desired to be projected do not have to coincide.

  In the next step S72, as shown in FIG. 21A, the projection control device 3 sets an image projection area 111 on which the content image is to be projected on the captured image 101 acquired in step S71. At this time, the projection control device 3 detects an operation of the keyboard 5 and the mouse 6 that designates the image projection region 111 while the captured image 101 is displayed on the display device 4. Thereby, the projection control device 3 detects the vertices C1, C2, C3, and C4 of the image projection area 111 corresponding to the vertices S1, S2, S3, and S4 of the projection plane 10. Here, in the second embodiment, since the observer's viewpoint position and the projection plane 10 are opposed to each other and the camera 2 and the projection plane 10 are not opposed, the image projection area 111 appearing in the captured image 101 is It becomes a distorted shape.

  At this time, similarly to the first embodiment, the projection control device 3 performs conversion using the conversion table Tcb, thereby converting the image data of the image projection area 111 in the captured image 101 into the image projection area 112 in the projection image 102. Convert to image data.

  In the next step S73, the projection control device 3 determines the vertexes C1, C2, C3, and C4 of the image projection area 111 on the captured image 101 obtained in step S72 and the vertex I1 of the content image area 113 in the content image 103, A distortion transformation matrix Q representing the relationship with I2, I3, and I4 is calculated. The processing for calculating the distortion transformation matrix Q is as follows.

Here, the projective transformation matrix is expressed by an equivalence relation, and the difference of constant multiples has no meaning, and therefore can be expressed as the following Expression 25. It should be noted that “˜” in the expression 25 represents an equivalence relationship, which means that they are equal by allowing a constant multiple difference.

In Expression 25, s and t indicate the coordinates of the content image area 113, and x and y indicate the coordinates of the image projection area 111 on the captured image. This shows that the image projection area 111 can be converted into the content image area 113 by the distortion transformation matrix Q composed of q11 to q32 in Expression 25. That is, the projective transformation matrix of the above equation 14 becomes the distortion transformation matrix Q, and the distortion transformation matrix Q is defined as the following equation 26.

When the coordinates (s, t) of the content image area 113 are expressed using the distortion transformation matrix Q of the above equation 26 and the coordinates (x, y) of the image projection area 111, s and t are expressed in a non-homogeneous expression. Are expressed as the following Expression 27 and Expression 28, respectively.

When each of Expression 27 and Expression 28 is expanded, the operation expression of Expression 29 is obtained for s, and the operation expression of Expression 30 is obtained for t.

If the arithmetic expressions of the above formulas 29 and 30 are arranged, s is represented by the following formula 31 and t is represented by the following formula 32.

When Expression 31 and Expression 32 are expressed as a determinant, the following Expression 33 is obtained.

Vertex C1 (x ₁ , y ₁ ), C2 (x ₂ , y ₂ ), C3 (x ₃ , y ₃ ), C4 (x ₄ , y ₄ ) of image projection area 111 and vertex I1 of content image area 113 When the coordinates of the four corresponding vertices of (s ₁ , t ₁ ), I 2 (s ₂ , t ₂ ), I 3 (s ₃ , t ₃ ), and I 4 (s ₄ , t ₄ ) are applied to the above equation 33, ,

It becomes. Therefore, a transformation matrix that defines the distortion transformation matrix Q is obtained as shown in Equation 35.

  When the correspondence relationship between the captured image and the content image is determined using the above Expression 35, the correspondence relationship between the captured image 101 and the content image 103 can be defined as shown in FIG. Thus, when the distortion transformation matrix Q is calculated, the projection control device 3 stores the distortion transformation matrix Q in the storage device 25. When the projection system actually projects the content image on the projection plane 10, the projection system converts the content image data into captured image data using the distortion conversion matrix Q, and then converts the captured image data into the conversion table Tcb. The projection image data is converted into projection image data, and a projection image is created and supplied to the projector 1.

  Further, the distortion transformation matrix Q is transformed to a position where the transformed pixel is shifted with respect to the sampling grid point of the pixel of the content image 103. For this reason, in order to obtain the luminance value of each pixel of the captured image 101, the projection control device 3 interpolates using the luminance value of the content image 103 around the converted coordinates. As a luminance value interpolation method, any of a bilinear method, a bicubic method, and a method of interpolating by creating a table may be used.

  As described above, the projection system can project the projection image having no distortion as viewed from the viewer facing the projection plane 10 by converting the content image data using the conversion table Tcb and the distortion conversion matrix Q. .

  As described above in detail, according to the projection system shown as the second embodiment to which the present invention is applied, the content image is the same as in the first embodiment even at the viewpoint position directly facing the projection plane 10. Data can be transformed by the distortion transformation matrix Q to project a content image without distortion on the projection plane 10.

[Third Embodiment]
Next, a projection system shown as the third embodiment will be described. Note that parts similar to those in the above-described embodiment are denoted by the same reference numerals, and detailed description thereof is omitted.

  The projection system shown as the third embodiment corresponds to the vertices S1, S2, S3, and S4 of the projection plane 10 by performing predetermined image processing on the captured image 101 in the process of calculating the distortion transformation matrix Q. The vertexes C1, C2, C3, and C4 of the image projection area 111 are detected.

  In this projection system, as shown in FIG. 24, a projection frame 10 a is provided at the edge of the projection surface 10. The projection frame 10 a has a role representing the area of the projection plane 10 and has a color different from the color of the projection plane 10. Examples of the projection frame 10a include a screen that is used when the projector 1 is normally used and has an inner projection surface 10 that is white and has a black edge.

  When such a projection plane 10 and projection frame 10a are captured by the camera 2, a large contrast appears in the captured image 101 between the projection plane 10 and the projection frame 10a. Therefore, the projection control device 3 can detect the vertices C1, C2, C3, and C4 of the image projection area 111 of the projection surface 10 by image processing. As this image processing, for example, the captured image 101 is differentiated to extract an edge, a straight line is detected from the extracted edge, and the vertices C1, C2, C3, and C4 of the image projection region 111 are obtained as intersections of the straight lines. Is mentioned. As another example of the image processing, there is a method of directly detecting the vertices C1, C2, C3, and C4 of the image projection area 111 using a vertex detection operator.

  In such a projection system, as shown in FIG. 25, in step S81, the projection area of the projection plane 10 is imaged by the camera 2 and a captured image 101 is acquired.

  In the next step S82, the projection control device 3 sets an image projection area 111 on which the content image is to be projected on the captured image 101 acquired in step S81. At this time, the projection control apparatus 3 automatically detects the vertices C1, C2, C3, and C4 of the image projection area 111 in the captured image 101 by performing image processing on the captured image 101. “Automatically” means “without manual operation”.

  In the next step S83, the projection control device 3 determines the vertices C1, C2, C3, C4 of the image projection area 111 on the projection image 101 and the vertices I1, I2, I3, I4 of the content image area 113 in the content image 103. A distortion transformation matrix Q representing the relationship is calculated.

  As described above, according to the projection system shown as the third embodiment, the vertices S1, S2, S3, and S4 can be automatically detected from the captured image 101 by predetermined image processing. The operation of setting 111 can be performed more efficiently and in a shorter time than the manual operation.

[Fourth Embodiment]
Next, a projection system shown as the fourth embodiment will be described. Note that parts similar to those in the above-described embodiment are denoted by the same reference numerals, and detailed description thereof is omitted.

  The projection system shown as the fourth embodiment detects the position of a mark attached to the image projection area of the projector 1 and obtains the vertex of the image projection area 111 in the captured image 101.

  The projection system shown as the fourth embodiment assumes a scene in which a projection image is projected in an environment where there is no indication of the vertices S1, S2, S3, and S4 that specify the projection plane 10. For example, the projection surface 10 may be a wall of a room. Then, when recognizing the vertices S1, S2, S3, and S4 of the projection plane 10 and creating the distortion transformation matrix Q, the user can obtain the vertices S1, S2, S3, and S4 of the projection plane 10 as shown in FIG. A mark 106 is pasted at a desired position. This mark 106 is like a removable seal. The mark 106 may be a color having a high contrast with the color of the projection surface 10. Further, the shape of the mark 106 is preferably a circle or a rectangle with high detection accuracy by image processing, but is not limited thereto.

  In a state where the mark 106 is pasted, the projection system captures an image within the imaging range including all the marks 106 pasted by the camera 2 and acquires the captured image 101. Then, the projection system automatically detects the position in the image of the mark 106 included in the captured image 101 as the vertices C1, C2, C3, and C4 of the image projection area 111. The detection method of the mark 106 is predetermined image processing as in the third embodiment. Also, the coordinates of the mark 106 in the captured image 101 can be easily detected by obtaining the center of gravity of the mark 106 in the captured image 101 when the mark 106 is a circle.

  Then, the projection system calculates the distortion transformation matrix Q using the captured image 101 and then removes the mark 106 attached to the projection surface 10. As a result, it is possible to project a content image without distortion from the viewpoint position directly facing the projection plane 10 onto an area that the user wants to project.

  In such a projection system, as shown in FIG. 27, when a mark 106 is first pasted at a position to be set as the vertices S1, S2, S3, S4 of the projection surface 10 by the user in step S91, in step S92, An area including the mark 106 is imaged by the camera 2.

  In the next step S93, the projection control device 3 sets an image projection area 111 on which the content image is to be projected on the captured image 101 acquired in step S92. At this time, the projection control device 3 automatically detects the mark 106 in the captured image 101 by performing image processing on the captured image 101, and detects the vertices C1, C2, C3, and C4 of the image projection region 111. Detect automatically.

  In the next step S94, the projection control apparatus 3 determines the vertexes C1, C2, C3, and C4 of the image projection area 111 on the captured image 101 obtained in step 93 and the vertex I1 of the content image area 113 in the content image 103, A distortion transformation matrix Q representing the relationship with I2, I3, and I4 is calculated.

  In subsequent step S95, the mark 106 attached to the projection surface 10 is removed by the user.

  As described above, according to the projection system shown as the third embodiment, the operation of setting the image projection area 111 in the captured image 101 can be performed more efficiently and in a shorter time than that performed manually. In particular, even when there is no vertex indicating the projection surface 10, the user can put the mark 106 at a desired position and width with respect to a surface such as a wall. A content image can be projected.

  The above-described embodiment is an example of the present invention. For this reason, the present invention is not limited to the above-described embodiment, and various modifications can be made depending on the design and the like as long as the technical idea according to the present invention is not deviated from this embodiment. Of course, it is possible to change.

DESCRIPTION OF SYMBOLS 1 Projector 2 Camera 3 Projection control apparatus 4 Display apparatus 5 Keyboard 6 Mouse 10 Projection surface 21 Image input part 22 Input image memory 23 Calculation part 24 Memory 25 Storage device 26 Control part 27 Output image memory 28 Image output part 101 Captured image 102 Projection Image 103 Content image 111, 112 Image projection area 113 Content image area

Claims (6)

A projection device that projects a projection image using content image data; and an imaging device that captures a projection image projected by the projection device and obtains a captured image.
A fringe pattern whose brightness changes periodically in a first direction orthogonal to the light projecting direction of the projection device is projected and imaged, and the captured image and the projected image are associated with each other using the phase shift method in the first direction. And projecting and capturing a fringe pattern whose brightness changes periodically in a second direction orthogonal to the first direction, and associating the captured image with the projected image using the phase shift method in the second direction First conversion information creating means for creating first conversion information that defines correspondence between pixels of the captured image and the projected image,
A projection area setting means for setting an image projection area on a captured image captured by the imaging device provided at a predetermined viewpoint position;
Area conversion means for converting an image projection area on the captured image set by the projection area setting means into an image projection area on the projection image using the first conversion information;
Content image area setting means for setting a content image area indicating the shape of the content image;
Based on the relationship between the image projection area set on the captured image by the projection area setting means and the content image area set by the content image area setting means, the content image area is an image projection area on the captured image. Second conversion information creating means for creating second conversion information for converting the content image data to be projected onto
A projection system comprising: image conversion means for converting content image data using the second conversion information and projecting a projection image from the projection device.   The projection system according to claim 1, wherein the imaging device is provided at a viewpoint position of an observer who views the projection image or an arbitrary position around the projection device.   The projection system according to claim 1, wherein the projection area setting unit detects an image projection area from the captured image by predetermined image processing.   The projection system according to claim 3, wherein the projection area setting unit detects a position of a mark attached to an image projection area of the projection device and obtains a vertex of the image projection area.   The first conversion information creation means creates the first conversion information by correcting the association between the captured image and the projection image performed using the phase shift method using a spatial coding method. The projection system according to any one of claims 1 to 4, wherein the projection system is characterized in that: A fringe pattern whose brightness changes periodically in a first direction orthogonal to the light projecting direction of the projection device is projected and imaged, and the captured image and the projected image are associated with each other using the phase shift method in the first direction. By projecting and capturing a fringe pattern whose brightness changes periodically in a second direction orthogonal to the first direction and associating the captured image with the projected image using the phase shift method in the second direction , Preliminarily creating first conversion information that defines the correspondence between the captured image and the projected image,
An image projection area is set on a captured image captured by an imaging device provided at a predetermined viewpoint position,
Converting the image projection area on the set captured image into an image projection area on the projection image using the first conversion information;
Set the content image area that indicates the shape of the content image,
Based on the relationship between the image projection area on the converted captured image and the set content image area, the content image data is converted to project the content image area onto the image projection area on the captured image. Create the second conversion information,
When actually projecting a content image, the content image data is converted using the second conversion information, and a projection image is projected from the projection device.