JP4554231B2 - Distortion parameter generation method, video generation method, distortion parameter generation apparatus, and video generation apparatus - Google Patents

Description

  The present invention relates to a distortion parameter generation method and a video generation method used in parallel video generation by computer graphics, to these apparatuses, and to applications thereof.

There are some which divide a target to handle video, generate parallel video by each processor, project each video on a screen by a plurality of projectors, and obtain one synthesized video.
In order to obtain a single continuous image by projecting onto a screen with a plurality of projectors, it is necessary to know the spatial relationship between the display screen and the projector. However, as shown in FIG. There is a situation where the relationship is not confronted. In such a case, the image is projected with distortion. As shown in FIG. 3B, if reverse distortion is applied at the image generation source, the image will be correctly captured when projected.

However, in the conventional video generation, there is no feedback of information between the video generator and the displayed video, and parameters are given to distort the video assuming a screen to be displayed in advance. There are problems such as inability to change, complicated adjustment to maintain the continuity of the display video generated in divided parts.
Further, as shown in Non-Patent Document 1, there are three lines of an image projected from a projector i (1... N) whose installation position is unknown and a distortion image on the screen imaged by a camera whose installation position is unknown. There is a matrix that is associated with a matrix of three columns and uses this matrix to apply reverse distortion to the image of the projector coordinates to correct the distortion of the image, but this is limited to a single screen.
Information Processing Society of Japan, “FIT2003 Information Science and Technology Forum General Lecture Collection Vol. 3”, published on August 25, 2003, p. 401-403

  The present invention performs feedback of information between the display screen and the image generation unit using an image captured by a camera, and can easily change the arrangement of the plurality of screens and the shape of the screen, and the continuity of the generated display image divided into parts It is an object of the present invention to provide a method and means that facilitate adjustment to be maintained.

In order to solve the above problem, a distortion parameter generating method according to claim 1 is a projection space configured by combining a plurality of n screens, and a plurality of p projectors provided corresponding to each of the plurality of n screens, A white and black rectangle created by a computer i having a plurality of m computers provided corresponding to each of the plurality of p projectors, a digital camera connected to the computer, and a storage device, and corresponding to the projector. A first process for projecting a test pattern common to each projector in a checkered pattern arranged vertically and horizontally on a screen corresponding to the projector, and the screen on which the test pattern is projected from a position as a viewpoint or a viewpoint position a second course of imaging by arranged the digital camera to the rear, the shooting From images, display necessary for matching between the image and the screen, the vertex constituting the screen, necessary for synthesizing the correction into one smooth image distortion of each projection image, of the rectangular portion of the test pattern and corners four is adjacent portions constitutes, detected through image recognition, the vertex of its position, and a third course determined using the computer connected to the digital camera, obtained by the third program the said corner and a conversion matrix for matching the displayed image distortion adjustment synthesis and screen and the scale matrix and movement matrix, and a fourth program obtained by the computer connected to the digital camera, the first course the fourth course, for each screen of the plurality n, after the projector is running multiple p provided, each projection Each transformation matrix obtained by the fourth program for meeting the synthesized and screen images into a single smooth image, a scale matrix and synthetic synthetic matrix taking the product of the moving matrix, as distortion parameters, for each projector And a fifth course stored in the storage device.

  In order to solve the above problem, a video generation method according to a second aspect of the present invention is a projection space configured by combining a plurality of n screens, a plurality of p projectors provided corresponding to each of the plurality of n screens, and the plurality of p A first process of reading an image from a frame memory in which an image generated by the computer is stored in the processing of each of the plurality of computers. A second process of rotating a virtual surface for texture mapping of the read image using the distortion parameter generated in claim 1, and a third step of pasting the read image as a texture pattern on the rotated virtual surface And storing the video pasted on the rotated virtual surface in the frame memory. And course of the video in the fourth frame memory storing at course and is characterized in that comprising a fifth program for projecting the corresponding screen by the projector corresponding to the computer.

In order to solve the above-described problem, a distortion parameter generation device according to a third aspect of the present invention is a projection space configured by combining a plurality of n screens, a plurality of p projectors provided corresponding to each of the plurality of n screens, and the plurality of projectors. A distortion parameter generation device having a plurality of m computers provided corresponding to each projector of p, a digital camera connected to the computer, and a storage device, wherein white and black rectangles are alternately arranged vertically and horizontally A plurality of projectors that project a common test pattern of each projector for each corresponding screen, and a digital camera that is positioned behind the viewpoint or behind the viewpoint and images the test pattern projected on the screen And a computer connected to the digital camera. From the captured image, necessary for matching the displayed image and the screen, the vertex constituting each screen, required to be combined into one smooth image to correct distortion of the projected image, the rectangular test pattern a corner constituting four are adjacent portions of the section, detected by the image recognition, a position detecting means for determining the position by the said corner the apex of each screen which has been determined by the position detection means, the strain transformation matrix according to the screen to match the adjustment and synthesis, and screen display image, scale matrix and movement matrix, and the matrix calculation means for calculating a computer connected to the digital camera, the transformation matrix determined by the matrix calculation means , take the product of the scale matrix and movement matrix synthesis and synthetic The Torikusu, as distortion parameters, is characterized in that a storage device for storing for each projector.

  In order to solve the above-described problem, a video generation device according to a fourth aspect of the present invention is a projection space configured by combining a plurality of n screens, a plurality of p projectors provided corresponding to each of the plurality of n screens, and the plurality of p A plurality of m computers provided for each projector, and each of the plurality of m computers includes an image reading unit that reads an image from a frame memory in which a generated image is stored, and the read image Virtual surface rotating means for rotating the virtual surface for texture mapping using the distortion parameter generated in claim 3, image pasting means for pasting the read image as a texture pattern on the rotated virtual surface, A frame memory for storing video pasted on the rotated virtual surface; The image of the frame memory is characterized in that comprising a projector for projecting onto a screen corresponding to the computer.

In order to solve the above-described problem, a distortion parameter generation method according to claim 5 is provided for a left-eye projector, a right-eye projector, the left-eye projector, and a right-eye projector in a projection space including a screen. A first process of having a left-eye computer, a right-eye computer, a digital camera connected to the computer, and a storage device, and imaging the screen with the digital camera, and a display video from the captured image A second step of detecting a vertex necessary for matching between the screen and the screen by image recognition and obtaining the position using the computer connected to the digital camera, and the computer for the left eye or the right eye each projector common of the hands of the checkered pattern that the rectangle of black and white that have been created are arranged in vertical and horizontal directions alternately by A third program for projecting either one projector for the left or right eye bets pattern on the screen, the image of which an image is captured, required to be combined into one smooth image to correct distortion of the projected image , A fourth step of detecting a corner formed by an adjacent part of the four rectangular portions of the test pattern by image recognition and obtaining the position using the computer connected to the digital camera; Either the left or right eye conversion matrix, the scale matrix, and the distortion adjustment and synthesis, and the screen and the display image are matched by the vertex obtained in the process and the corner obtained in the fourth process. the movement matrix, and fifth courses determined by the computer connected to the digital camera, created by a computer Re one projector for other eyes than one projector left or right eye according to the third course of each projector common test pattern to the screen of the checkerboard arranged rectangular white and black vertically and horizontally alternately And a portion adjacent to the four rectangular portions of the test pattern necessary for correcting the distortion of each projected image and synthesizing it into one smooth image from the captured image. The corners to be configured are detected by image recognition, and the positions thereof are determined by the seventh process obtained using the computer connected to the digital camera, the vertexes obtained in the second process, and the seventh process. In addition to the left or right eye according to the fifth step depending on the corner, the left or right eye for adjusting the distortion and synthesis and matching the screen and the display image An eighth process for obtaining the other conversion matrix, scale matrix, and movement matrix for use with the computer connected to the digital camera, and for combining the projected image into one smooth image and matching it to the screen And a ninth process of storing a composite matrix obtained by combining the products of the conversion matrix, the scale matrix and the movement matrix obtained in the fifth process and the eighth process in the storage device as a distortion parameter. It is a feature.

  In order to solve the above-described problem, a video generation method according to a sixth aspect of the present invention is provided corresponding to each of the left-eye projector, the right-eye projector, the left-eye projector, and the right-eye projector in a projection space including a screen. A first process of reading an image from a frame memory in which either the left-eye image or the right-eye image generated by the computer is stored in the processing of the computer; and A second step of rotating either one of the left-eye or right-eye virtual surfaces to be texture-mapped using either the left-eye or right-eye distortion parameter generated in claim 5; and the second step. And pasting the read image as a texture pattern on the virtual plane rotated by The fourth process for storing the image pasted on the virtual surface in the third process in the frame memory, and the image in the frame memory stored in the fourth process for either the left eye or the right eye A frame in which a fifth process for projecting onto a screen by one projector and an image for another eye other than one of the left and right eye images generated by the computer and read by the first process are stored. A virtual surface other than the virtual surface for either the left eye or the right eye in the sixth process for reading an image from the memory and the second process, and for either the left eye or the right eye for texture mapping the read image One of the virtual planes is the distortion parameter for either the left eye or the right eye generated in claim 5, and is used for the left eye or the right eye in the second process. A seventh process for rotating using a parameter other than any one of the parameters, and an eighth process for pasting the image read out in the sixth process as a texture pattern on the virtual surface rotated in the seventh process The process, the ninth process for storing the image pasted on the virtual surface in the eighth process in the frame memory, and the image in the frame memory stored in the ninth process are converted into the left eye by the fifth process. And a tenth process of projecting onto a screen by either the left-eye or right-eye projector other than the right-eye or right-eye projector.

In order to solve the above-described problem, a distortion parameter generation device according to a seventh aspect includes a left-eye projector, a right-eye projector, and a left-eye projector and a right-eye projector in a projection space including a screen. A test pattern common to each of the checkered projectors in which white and black rectangles are arranged alternately in the vertical and horizontal directions on the screen, and the left-eye computer, the right-eye computer, the digital camera connected to the computer, and the storage device. A projector for the left eye that projects, a projector for the right eye that projects a test pattern common to each projector in a checkered pattern in which white and black rectangles are arranged alternately in the vertical and horizontal directions , and a position serving as a viewpoint. Screen from left eye projector and right eye projector Said digital camera for capturing each test pattern being projected, is configured by a computer connected to the digital camera, from the screen of an image captured by the digital camera, required to conform with the displayed image and the screen, the Detect the vertices that make up the screen by image recognition, determine the position, and correct the distortion of each projected video from the images captured by the digital camera with the test patterns from the left-eye projector and right-eye projector. needed to be combined into one smooth video, corners four are adjacent portions of the rectangular portion of the test pattern constituting detected by the image recognition, a position detecting means for determining the position, the position detecting means Before the projection of the vertex and the left and right eye projectors obtained in step By the corners, each of the transformation matrix for the left and right eye for matching the synthesis and screen display image distortion adjustment, a matrix calculating means for calculating the scale matrix and movement matrix by the computer, the projection obtained by the matrix calculation means As a distortion parameter for the left and right eyes , a composite matrix that is the product of the transformation matrix, scale matrix, and movement matrix for the left and right eyes to synthesize the video into one smooth image and match the screen. And a storage device for storing.

  In order to solve the above problem, an image generation apparatus according to claim 8 is provided in a projection space including a screen, corresponding to each of a left-eye projector, a right-eye projector, and the left-eye projector and right-eye projector. A computer for a left eye and a right eye; in the computer, a left-eye image reading means for reading an image from a frame memory in which a left-eye image is stored; and a frame memory in which a right-eye image is stored 8. A right-eye image reading unit for reading an image, and a virtual surface rotating unit for the left eye that rotates a virtual surface for texture mapping of the image read by the left-eye image reading unit using the distortion parameter for the left eye generated in claim 7. And the image read by the right-eye image reading means The virtual surface to be mapped is rotated using the right-eye virtual surface rotating means generated by the right-eye distortion parameter generated in claim 7 and the left-eye image read out to the virtual surface rotated by the left-eye virtual surface rotating means. The image read by the means is pasted as a texture pattern, and the image read by the right-eye image read-out means is pasted as a texture pattern on the virtual plane rotated by the virtual plane rotation means for the right eye. A right-eye image pasting unit, a left-eye frame memory for storing a video pasted on the virtual plane by the left-eye image pasting unit, and a video pasted on the virtual plane by the right-eye image pasting unit are stored. Project the video in the right-eye frame memory and the stored left-eye frame memory onto the screen. A left-eye projector and is characterized in that it consists of a right-eye projector for projecting an image of the storage was the right eye frame memory to the screen.

  In order to solve the above-mentioned problem, a distortion parameter generation method according to claim 9 is the method according to claim 5, and corresponds to each of the plurality of n screens in a projection space configured by combining a plurality of n screens. A plurality of p / 2 sets of projectors for left and right eyes, and a plurality of m sets of computers for left and right eyes provided corresponding to the two projectors of the left and right eyes, respectively. And a digital camera connected to the computer and a storage device.

  In order to solve the above problem, an image generation method according to claim 10 is the method according to claim 6, wherein a projection space configured by combining a plurality of n screens corresponds to each of the plurality of n screens. Two left and right eye plural projectors for the left and right eyes, and two right and left eye plural projectors for the left and right eyes, and a plurality of m sets of computers for the left and right eyes, respectively. It is characterized by that.

  In order to solve the above problem, a distortion parameter generating apparatus according to claim 11 is the apparatus according to claim 7, and corresponds to each of the plurality of n screens in a projection space configured by combining a plurality of n screens. A plurality of p / 2 sets of projectors for left and right eyes, and a plurality of m sets of computers for left and right eyes provided corresponding to the two projectors of the left and right eyes, respectively. And a digital camera connected to the computer and a storage device.

  In order to solve the above problem, an image generating device according to claim 12 is the device according to claim 8, and corresponds to each of the plurality of n screens in a projection space configured by combining a plurality of n screens. Two left and right eye plural projectors for the left and right eyes, and two right and left eye plural projectors for the left and right eyes, and a plurality of m sets of computers for the left and right eyes, respectively. It is characterized by that.

  In order to solve the above-mentioned problem, a video generation method according to claim 13 is the video generation method according to any one of claim 2, claim 6, or claim 10, wherein each projector that projects on a screen is adjacent to each other. It is characterized in that a part of the video is overlapped.

  In order to solve the above problem, an image generation device according to claim 14 is the device according to any one of claims 4, 8, or 12, wherein an adjacent image of each projector projected on a screen is displayed. This is characterized in that a part of the regions is overlapped.

  According to the distortion parameter generating method according to claim 1, a predetermined test pattern is projected on a plurality of screens, and the vertex of each screen viewed from the viewpoint and the corner of the test pattern projected according to the shape of the screen are detected, A conversion matrix, a scale matrix, and a movement matrix of the positional relationship between the camera at the viewpoint and each of the plurality of screens can be obtained as distortion parameters.

  According to the video generation method according to claim 2, by using the distortion parameter generated in claim 1 for the image generated for each screen, the virtual plane for texture mapping of the image is rotated to give reverse distortion. Since the image is pasted and stored on the virtual plane, the image with the reverse distortion is projected onto the screen, and the image of one image is continuously displayed on the multiple screens from the viewpoint. It can be copied so that it looks correct to maintain.

  According to the distortion parameter generating apparatus according to claim 3, a predetermined test pattern is projected on a plurality of screens by a projector, the test pattern is imaged by a digital camera, and the apex of each screen and the screen viewed from the viewpoint by a position detection unit The corner of the test pattern projected according to the shape of the image is detected, and the matrix calculation means can determine the transformation matrix, the scale matrix and the movement matrix of the positional relationship between the camera at the viewpoint position and each of the plurality of screens as the distortion parameter, This can be stored in a storage device.

  According to the video generation device of the fourth aspect, the image is read from the frame memory in which the image generated for each screen is stored by the image reading means, while the virtual plane rotating means is a virtual for texture mapping the read image. The surface can be rotated using the distortion parameter generated in claim 3, and the image pasting means can paste the read image as a texture pattern on the rotated virtual surface to give the image a reverse distortion. Since the image is pasted on the virtual plane and stored in the frame memory, the image with the reverse distortion is projected from the projector onto the screen, and one video image from the viewpoint is displayed on the plurality of screens as a whole. It can be copied so that it looks correct so as to maintain continuity. Since one large screen image is synthesized using a plurality of projectors, the projection distance can be shortened.

  According to the distortion parameter generation method according to claim 5, a predetermined test pattern for right and left is projected on the screen, and a vertex of each screen viewed from the viewpoint and a corner of the projected test pattern are detected according to the shape of the screen. Thus, the conversion matrix, scale matrix, and movement matrix of the positional relationship between the camera at the viewpoint position and the screen can be obtained as the left and right distortion parameters.

  According to the video generation method of claim 6, by using the distortion parameters for the left and right eyes generated in claim 5 for the images generated for the left and right eyes, the left and right eyes for texture mapping are used. The virtual plane can be rotated to give the reverse distortion, and the image is pasted and stored on the virtual plane, so the image with this reverse distortion is projected onto the screen and left and right with respect to the screen from the viewpoint. Therefore, it is possible to make a stereoscopic view so that the image of the image can be seen correctly.

  According to the distortion parameter generating device of the seventh aspect, a predetermined test pattern for the left and right eyes is projected on the screen by the projector, the test pattern is imaged by the digital camera, and the vertex of the screen viewed from the viewpoint by the position detection unit. And a corner of the test pattern projected according to the shape of the screen is detected, and a conversion matrix, a scale matrix, and a movement matrix of the positional relationship between the camera at the viewpoint and each of the plurality of screens are obtained as distortion parameters by matrix calculation means. This can be stored in a storage device.

  According to the video generation device of the eighth aspect, the image is read from the frame memory in which the images generated for the left and right eyes are stored by the image reading unit, while the virtual plane rotating unit is a virtual that performs texture mapping on the read image. The surface can be rotated using the distortion parameter generated in claim 7, and the image pasting means can paste the read image as a texture pattern on the rotated virtual surface to give the image a reverse distortion. Since the image is pasted on the virtual surface and stored in the frame memory, the image with the reverse distortion is projected from the projector onto the screen so that the right and left images can be correctly viewed from the viewpoint to the screen. It is possible to take a stereoscopic view.

  According to the distortion parameter generation method according to claim 9, in addition to the effect of the method according to claim 5, a predetermined test pattern is projected on a plurality of screens, and the vertexes of each screen and the shape of the screen viewed from the viewpoint. The corner of the test pattern projected according to the above is detected, and the conversion matrix, scale matrix, and movement matrix of the positional relationship between the camera at the viewpoint and each of the plurality of screens can be obtained as the distortion parameters.

  According to the video generation method of claim 10, in addition to the effect of the method of claim 6, the image generated for each screen is texture-mapped by using the distortion parameter generated in claim 5. The virtual plane can be rotated to give reverse distortion, and the image is pasted and stored on the virtual plane. On the other hand, the image of one image as a whole can be copied so that it looks correct so as to maintain continuity.

  According to the distortion parameter generation device according to claim 11, in addition to the effect of the device according to claim 7, a predetermined test pattern is projected on a plurality of screens by a projector, the test pattern is imaged by a digital camera, and the position The detection means detects the vertex of each screen viewed from the viewpoint and the corner of the test pattern projected according to the shape of the screen, and the matrix calculation means converts the positional relationship between the camera at the viewpoint and the plurality of screens, the scale The matrix and the movement matrix can be obtained as distortion parameters, and can be stored in the storage device.

  According to the video generation device of claim 12, in addition to the effect of the device of claim 8, the image is read out from the frame memory in which the image generated for each screen is stored by the image reading means, The surface rotation means rotates a virtual surface on which the read image is texture-mapped using the distortion parameter generated in claim 3, and the image pasting means pastes the read image on the rotated virtual surface as a texture pattern. In addition, the image can be given an inverse distortion, and the image is pasted on the virtual plane and stored in the frame memory. Make sure that the image of one image is correctly displayed on multiple screens so as to maintain continuity. It can be.

  According to the image generation method according to claim 13, in addition to the effect of the method according to claim 2, claim 6, or claim 10, a plurality of projectors are arranged in an overlapping manner in the overlap region. By performing color mixing, it is possible to make it difficult to visually recognize the difference in characteristics between adjacent projectors.

  According to the video generation device according to claim 14, in addition to the effect of the device according to any one of claims 4, 8, or 12, when projecting from a plurality of projectors arranged in an overlapping manner, By performing color mixing in the overlap region, it is possible to make it difficult to visually recognize the difference in the characteristics of the projectors adjacent to the projector.

([1] Distortion parameters and video generation for multiple screens)
Embodiments of the present invention will be described below with reference to the drawings. FIG. 1A is a flowchart for explaining an embodiment of a method for generating distortion parameters for a plurality of screens, and FIG. 1B is a flowchart for explaining an embodiment of a method for generating a video for a plurality of screens. FIG. 2 is a functional block diagram for explaining an embodiment of a distortion parameter generating apparatus and a video generating apparatus relating to a plurality of screens. In FIG. 2, 101i (i = 1,2,3, ... n) is a screen (or _S i), 102i projector plurality p provided in correspondence to each of the screen 101i (or _P i), 103i is the A plurality of m computers (or PC _i ) provided corresponding to each of the plurality of p projectors 102i. FIG. 2A shows the case where p = m = n = 3. 104 is a digital camera, 105 is a computer to which the digital camera 104 is connected, 106 is a position detecting means constituted by the computer 105, 107 is a matrix calculating means constituted by the computer 105, 110 is a storage device, 111 is each This is a network for connecting the computer 103i. 112i is an image reading unit configured by each computer 103i, 113i is a virtual plane rotating unit configured by each computer 103i, 114i is an image pasting unit configured by each computer 103i, and 115i is provided in each computer 103i. 1 frame memory, 116i is a second frame memory provided in each computer 103i, 118i is a graphics board provided in each computer 103i, and has, for example, the first frame memory 115i and the second frame memory 116i. . Reference numeral 117i denotes a three-dimensional graphics generation means constituted by each computer 103i, each computer 103i has a node i relationship with respect to the network, and the computer 105 serves as a host.
The plurality of screens 101i are, for example, a quadrangle, and the number of surfaces is three in FIG. 2A, i = 1, 2, 3, and placed in an arrangement according to the image projection site. Construct a projection space. The projector 102i does not necessarily face the screen 101i, and the optical axis of the projector 102i and the surface of the screen 101i do not have to be orthogonal. The digital camera 104 is arranged at a viewpoint that is a position where a person views the screen image on the screen. The positional relationship of the camera 104, the screen 101i, and the projector 102i, that is, the spatial arrangement is unknown.

([1-1] Generation of distortion parameters)
Generation of distortion parameters by the above configuration will be described.
In a composite display space, it is necessary to know the spatial relationship between the display screen 101i and the projector 102i corresponding to the display screen 101i in order to make the display image continuous.
As described above, in order to know the spatial relationship between the display screen 101 i and the projector 102 i, a test pattern having a known shape is projected from the projector projector 102 i and captured by the digital camera 104, thereby deforming the figure. Find the spatial relationship by degree.
By repeating this operation for all screens 101i (i = 1, 2, 3) constituting the display space, the relationship between all projectors 102i and the screen 101i can be obtained. Conversely, distortion can be corrected by distorting and projecting the image before projection using this relationship.

First, distortion parameter generation for image generation with distortion correction for a single screen will be described. In principle, the combined display space is implemented by repeating the method of a single plane for the number of screens constituting the space.
FIG. 4 shows a method for obtaining a spatial relationship using a known test pattern when the spatial arrangement of the camera 104, the screen 101, and the projector 102 is unknown.
The spatial relationship between the imaging surface of the camera 104 (assuming the viewpoint) and the imaging surface of the projector 102 is related by a 3 × 3 conversion matrix (Homography Matrix).
FIG. 5 shows how to obtain the Homography Matrix. In order to obtain this, a predetermined point of the test pattern is mathematically obtained by detecting a point C by the camera 104 corresponding to the point P by the projector 102.
FIG. 6 shows a mathematical calculation by detecting a camera point Cij corresponding to the projector point Pij in the case where there are a plurality of screens and projectors, corresponding to FIG.

Each computer 103i creates a test pattern and projects it onto the corresponding screen 101i by each corresponding projector 102i (P101 in FIG. 1A). An example of a test pattern is shown in FIG. 7 ((i) and (ii) display does not constitute a test pattern). The test pattern is a checkered pattern in which two types of rectangles (squares in the embodiment) with high contrast are arranged vertically and horizontally, such as black and white. The corner of the test pattern is a point (for example, 701 in FIG. 7) through which the sides of the black and white squares constituting the test pattern are interposed.
At this time, the test pattern is distorted and projected according to the spatial relationship between the display screen 101i and the projector 102i.

  The screen 101i on which the test pattern is projected is imaged by the digital camera 104 arranged at a position to be a viewpoint (P102 in FIG. 1A).

The position detection means 106 detects, from the captured image, image recognition of a vertex that constitutes the screen 101i and a corner that is formed by a portion where the four rectangular portions of the test pattern are adjacent to each other, and the position is detected by the digital camera. 104 is obtained by the computer 105 connected to the terminal 104 (P103 in FIG. 1A).
The vertex constituting the screen 101i is the end of the side constituting the rectangle of the screen 101i. This vertex is detected by image recognition.

The corner point detection of the test pattern will be described.
An image on which the test pattern is projected is _denoted by F _Tcam , and an image on which the test pattern is not projected is _denoted by F _cam .
The difference between F _Tcam and F _cam is taken, and the image F _DT of only the test pattern with the background removed is created.
A rectangular area shown in FIG. 8 is set in the image data of the digital camera 104, and the corner of the test pattern is detected by searching the image _FDT . In FIG. 8, L represents the vertical and horizontal sizes of the search rectangular area, and the unit is “pixel”. “L” can be changed in the program and can be arbitrarily determined. The search is automatically performed by the program. The search is performed, for example, from the upper left of the image toward the lower right sequentially as shown in FIG. Search is performed while shifting the position by half (L / 2) of the rectangular area.
As a search means, use is made of the feature that the test pattern is a combination of white and black portions, and the corner points have high contrast. The corner point is considered to have a large variation in luminance because of its high structural contrast. Therefore, the variation coefficient Cν, which is known as a statistic used when comparing the variations of the groups, is detected as an evaluation value. That is, as the luminance variation is larger, the value of the variation coefficient is larger and approaches 1. The variation coefficient Cν is expressed by the following equation.
Cν = σ / μ (1)
Here, μ represents the average luminance within the region, and σ represents the standard deviation of luminance.
First, among the rectangular areas shown in FIG. 8, the coefficient of variation in the area is evaluated by a combination of the upper half area and the left half area of the rectangular area shown in FIG. 10A to detect candidate points. Here, the candidate point is the center of the rectangular area shown in FIG. When the contrast of the test pattern is ensured, at the corner points, the coefficient of variation of the two rectangular areas shown in FIG. 10 (a) is due to the left half area consisting of 101 and 102 and the upper half consisting of 101 and 103. This depends on the area of the test pattern. FIG. 7 shows the position of the search area at a certain time when searching within the test pattern using a combination of the upper half and the left half of the rectangular area. In FIG. 7, the position according to (i) is close to the corner point of the test pattern, so that the variation in luminance is large and the value of the coefficient of variation is also large. However, at the position according to (ii), a portion with high luminance occupies most of the area, so that variation in luminance is small and the value of the coefficient of variation is small. Thus, it is possible to search for a corner point using the coefficient of variation.
When it is determined that the value of the coefficient of variation is large in the search by the area shown in FIG. 10A, the evaluation is further performed by the search by the combination of the lower half area and the right half area of the rectangular area shown in FIG. Improve the reliability.
・ Candidate points do not necessarily match the corner points of the test pattern, so a rectangular area is set around the candidate points, and the vertical and horizontal edges (the line where the black and white areas that make up the test pattern touch each other) are set using luminance information. ) Is detected. Edge detection is performed within a rectangular area centered on the candidate point. In the left half area as shown in FIG. 11A and the upper half area as shown in FIG. The position where the difference is maximum is determined as the edge position. The position where the vertical edge line and the horizontal edge line obtained in this way intersect is used as the corner position of the test pattern.

  The computer in which the digital camera 104 is connected to a transformation matrix, a scale matrix, and a movement matrix depending on the spatial relationship between the screen 101i and the projector 102i with respect to the viewpoint at the point where the digital camera 104 is located by the vertex and the corner obtained in the process of P103 105 (P104 in FIG. 1A).

となるマトリクス（３×３）

を推測する。 (Homography Matrix)
If a plurality of coordinates C _{ij of the} camera 104 corresponding to the coordinates P _ij can be detected for each projector 102 i by image processing, the following expression (2) is established.

Matrix (3x3)

Guess.

の要素を未知数とする連立方程式（３）となる
Ａ《ｈ′》＝《ｂ》 （３）
ここで、例えば《ｈ′》はベクトルｈ′を示す表記である。なお、この明細書、図面においてベクトル表記を、《 》を用いた場合と、白抜き又は肉太の場合とで併用して表す。 Since the corresponding points P _ij (x _ij ^p , y _ij ^p ) and C _ij (x _ij ^C , y _ij ^c ) are known, the equation (1) is

It becomes simultaneous equations (3) with the elements of A as unknowns A << h '>><< b >> (3)
Here, for example, << h '>> is a notation indicating the vector h'. In this specification and drawings, vector notation is used in combination with <<>> and white or thick.

の要素を連続して並べた列ベクトルであり未知な値である。マトリクスＡ、ベクトル《ｂ》は既知のＰ_ｉｊ，Ｃ_ｉｊより決まる。《ｈ′》は９個の未知数を持つが（３）式がスケールに対して同値であるため、
《ｈ》＝［ｈ_１１／ｈ_３３，ｈ_１２／ｈ_３３・・・ｈ_３３／ｈ_３３］
を導入すれば、
Ａ《ｈ》＝《ｂ》 （４）
である。これは８個の未知数を持つ。従って４点以上のＰ_ｉｊ，Ｃ_ｉｊ（ｊ≧４）を用いれば（４）式の連立方程式は数値的に解ける。この数値的解法は良く知られている。 << h '>>

Is a column vector in which the elements of are consecutively arranged and is an unknown value. The matrix A and the vector << b >> are determined from known P _ij and C _ij . << h '>> has 9 unknowns, but because Eq. (3) is equivalent to the scale,
<< h >> = [h ₁₁ / h ₃₃ , h ₁₂ / h ₃₃ ... H ₃₃ / h ₃₃ ]
If you introduce
A << h >> = << b >> (4)
It is. This has 8 unknowns. Therefore, if four or more points of P _ij and C _ij (j ≧ 4) are used, the simultaneous equations of equation (4) can be solved numerically. This numerical solution is well known.

とし、この

を用いて、カメラ画像上の歪が無いパターンの点Ｃ_ｉｊに次の変換

を施せば、撮像時にカメラ位置より見て、歪の無い正しい映像を作ることができる（図５、図６）。
従って、発生した映像に

を操作すれば、撮影時に映像が正しく表示される。 Return the matrix “h” obtained above to matrix

And this

Is used to convert the point C _ij of the pattern without distortion on the camera image into the next transformation

If it is applied, it is possible to create a correct video image without distortion when viewed from the camera position during imaging (FIGS. 5 and 6).
Therefore, in the generated video

If you operate, the image will be displayed correctly when shooting.

(Compound distortion correction)
FIG. 12 shows a process for obtaining the relationship between each projector 102i and the screen 101i by applying the above process to the composite display space a plurality of times.
(1) Copy the test pattern from the projector 102i to the screen 101i. Let the captured image be _<< F _si >>.
_{However, "F s" = [ "} C 1", "C 2" ···, "C M"], "C i" = [r i, g i, b i]
It is. Here, [<< C ₁ >>, << C ₂ >>... << C _M >>] represents pixel information, and the subscript “M” represents the number of pixels of the image. If the image size is 640 × 480 pixels, the value of “M” is 640 × 480 = 307200. Each piece of pixel information further includes three pieces of information of r, g, and b (red, green, and blue).
(2) Erase the test pattern of the projector 102i. Images were captured photograph on the screen 101i and _{"F bi".}
(3) Take the difference and remove the background image _<< F _i >> = _<< F _si >> − _<< F _bi >>
(4) For the image << Fi >>, the matrix H _i ′ is obtained using the “Homography Matrix” derivation process.

《Ｓ_ｉｊ》は画像輝度であり、座標位置（ｉ，ｊ）を中心とした指定範囲ｍ，ｌ内の輝度を用いて平滑化した値となる。
・画像の輝度《Ｓ_ｉｊ》にラプラシアンをかける。すなわち、画像を精鋭化するため、ラプラシアンフィルタを施す。
・メモリを辿り、上位のＭ個を抜き出す。
但し、Ｍはスクリーンの頂点数であり、この場合４である。
上位４個の点の格納アドレスをａ_ｉ（ａ_１，ａ_２，ａ_３，ａ_４）とすれば、

となる。
以上の処理により、点群

を求める。
（２）

で規定される変形された点群とスクリーンを規定する正規な（歪のない）点群《ｍ^Ｃ》との対応より、

前記説明の方法により、Ｈ_ｄを求める。
（３）Ｈ_ｄを用いて、視点Ｅのカメラ座標と各プロジェクタＰ_ｉの座標系との関係Ｈ_ｉは
Ｈ_ｉ′＝Ｈ_ｉＨ_ｄより
Ｈ_ｉ ＝Ｈ_ｄ ^−１Ｈ_ｉ となる。 The Homography matrix H _i ′ obtained above indicates the correspondence between the projector coordinates P _i and the camera coordinates C at the camera position F.
By converting the coordinate points of the model to be displayed using this inverse matrix, an image without distortion seen from the position F can be seen.
However, since the viewpoint is viewed at the viewpoint position E, it is necessary to correct this for the viewpoint position E. The correction procedure is shown below.
(1) determine the vertices of the front of the screen 1012 from the image _{"F b".} In FIG. 12, “F _C ” represents a point group at the vertex of the screen detected based on the image obtained from the camera 104.
-Apply Laplacian to << F _b >> to enhance the edge.
・ Extract edges by binarizing white and black.
・ Vertex detection.
・

_<< S _ij >> is the image brightness, which is a value smoothed using the brightness within the designated ranges m and l with the coordinate position (i, j) as the center.
-Apply Laplacian to the image brightness _{<< Sij} >>. That is, a Laplacian filter is applied to sharpen the image.
Trace the memory and extract the top M.
However, M is the number of vertices of the screen, and is 4 in this case.
If the storage addresses of the upper four points are a _i (a ₁ , a ₂ , a ₃ , a ₄ ),

It becomes.
With the above processing, the point cloud

Ask for.
(2)

From the correspondence between the deformed point group defined in (1) and the normal (undistorted) point group << m ^C >> that defines the screen,

H _d is obtained by the method described above.
3 with _{H d,} the relationship _{H i} of the camera coordinate of the viewpoint E and the coordinate system of each projector _{P i} becomes H _{i '=} H i _{H d} from H _i = _H ^d -1 _{H i.}

(Match screen and display image)
Than was used the H _i which has been determined by the not take a match between the displayed image and the screen. For this reason, in order to match the display image with the screen, the scale amount for reduction and enlargement and the parallel movement amount are obtained.

(1) Scale A scale is obtained according to FIG.
Perform the following calculation:
_<< m _1i '>> = H _i ^-1 _<< m _1i ^-1 >>
_<< m _2i '>> = H _i ^-1 _<< m _2i ^-1 >>
_<< m _1i >> and _<< m _2i >> are the upper vertices of the test pattern created with the coordinates of the projected image, and _<< m _1i ′ >> and _<< m _2i ′ >> are the upper vertices of the test pattern within the normal coordinates. H _i ⁻¹ represents a conversion matrix for converting the coordinates of the projected image into normal coordinates. By using H _i ⁻¹ , it is possible to calculate which position in the normal coordinates the upper vertices _<< m _1i >> and _<< m _2i >> of the test pattern created on the coordinates of the projected image are. This is a calculation necessary for correcting an image having no distortion as viewed from the viewpoint position for the viewpoint position. (See FIG. 14. Corresponds to the relationship between Hi, Hi ′, and Hd below FIG. 12A.)
_<< S _1i '>> = H _{d <<} S _1i >>
_<< S _2i '>> = H _{d <<} S _2i >>
_<< S _1i >> and _<< S _2i >> represent the upper vertices of the screen in the image coordinates of the camera, and _<< S _2i ′ >> and _<< S _2i ′ >> represent the upper vertices of the screen in the normal coordinates. H _d represents a conversion matrix for converting the image coordinates of the camera into normal coordinates. The use of H _d, it is possible to screen the upper apex "S _1i" in the image coordinates of the _camera, "S _2i" to calculate whether made at any position in the regular coordinates. This is a calculation necessary for correcting an image having no distortion as viewed from the viewpoint position for the viewpoint position. (See FIG. 15. Corresponds to the relationship between Hi, Hi ′, and Hd below FIG. 12A.)
Find the distance between two points.
l _mi = | _<< m _1i '>>- _<< m _2i '>> |
l _i = | _<< S _1i '>>- _<< S _2i '>> |
Find the scale from the distance.
k _i = l _mi / l _i
If the scale of the display test pattern _k xi _{', k yi'} and, seek scale _{k xi} = k _xi 'k _{i
k yi} = k _yi 'k _i
Therefore, the scale matrix H _Si is

となる。 (Translation amount)
In order to match the screen and the display area, it is necessary to match the base point of the screen and the base point of the corresponding image by translation. The following is the amount of translation.
_{"O ti '" = H i} "O ti"
<< O _Si '>> = H _d << O _Si >>
_{Here, "O ti", "O} Si" is the center position and the screen center position of the test pattern in the image coordinates of the camera. Further, _<< O _ti '>> and _<< O _Si '>> are the center position of the test pattern and the center position of the screen in normal coordinates.
The movement vector << T _i >>
<< T _i >> = ( _<< O _Si '>>- _<< O _ti '>>
Therefore, the movement matrix H _ti is

It becomes.

(Final matrix)
Using rotation (included in H _i ), translation H _ti , and scale matrix H _Si , the final matrix H _i ^TOTAL for matching the screen and the displayed image is:
H _i ^TOTAL = H _ti H _Si H _i
It is.

After repeating the first course (P101) to the second course (P102), the third course (P103), and the fourth course (P104) a plurality of n times, the fourth course (P104) 5th process (P105 of Fig.1 (a)) which stores each conversion matrix, scale matrix, and movement matrix which were calculated | required in a memory | storage device as a distortion parameter.
By the distortion correction of the composite surface described above,
H ₁ ^TOTAL , H ₂ ^TOTAL , H ₃ ^TOTAL
For each.
Here, the suffixes 1 to 3 indicate the numbers of the screen 101i. (The computer 103i number corresponds to this screen number.)
The obtained H ₁ ^TOTAL , H ₂ ^TOTAL , and H ₃ ^TOTAL are stored in the storage device 110 as a distortion parameter file.
When displaying the test pattern for each screen 101i, the test patterns are projected onto one screen, for example, the central screen 1012 from the projectors 1021, 1022, and 1023 corresponding to each screen 101i, and the same size is projected. Displayed on the screen 101i.

([1-2] Video generation)
This will be described with reference to FIGS. 1B, 2 and 4.
(A) The image reading unit 1121i of the computer 103i reads the contents of the first frame memory 115i in which the results rendered (image generation) using the three-dimensional graphics generation unit 117i are stored (FIG. 1 ( b) S101). As shown in FIG. 16A, an image without distortion is stored in the first frame memory 115i. Rendering (image generation) at this time is based on transmission projection.
(B) The virtual surface rotating unit 113i rotates the surface to be texture-mapped using [H _i ^TOTAL ] ⁻¹ from the distortion parameter generated in the above “[1] Generation of distortion parameter” (FIG. 16B). (S102 in FIG. 1B).
(C) The image pasting means 114i pastes the video of the contents of the first frame memory 115i as a texture pattern on the texture mapping surface of the step S102 using parallel projection as the video generation projection method (FIG. 1B). S103). The reverse distortion is applied by pasting.
(D) The video with reverse distortion, that is, the video pasted on the rotated texture mapping surface is stored in the second frame memory 116i (S104 in FIG. 1B). The rendering at this time is based on parallel projection.
(E) The contents of the second frame memory 116i are projected onto the screen 101i via the projector 102i (S105 in FIG. 1B). At this time, the distortion of the image is corrected.
The above steps (a) to (e) are repeated until i = 1 to 3 to display on all screens 101i.
FIG. 17 shows the image (a) without distortion (image in pass 1 in the left column) read by the plurality of computers 103i and the image subjected to reverse distortion in (d) (pass 2 in the right column). Image).

(Other)
In the above description, the case where the number of the screen 101i, the projector 102i, and the computer 103i is the same number, p = m = n = 3, has been described. The number of p, m, and n is not limited to the same number. For example, a plurality of projectors 102ij that project an image on each screen 101i, for example, two projectors 102ij are provided as shown in FIG. 18, for example, and images from these projectors 102ij are provided so as to be arranged vertically on each screen 101i. Also good. These images are projected so as to form one video as a whole. FIG. 19 shows a display example when there are a plurality of screens projected from a plurality of projectors.
As shown in FIG. 2E, a plurality of, for example, two graphics boards 118 are provided in one computer 103, and the graphics boards 118j and 118 (j + 1) are connected to the projector 102i. The computer 103 performs the necessary processing as described above for each of the graphics boards 118j and 118 (j + 1) by, for example, time division.
In the above description, the digital camera 104 is connected to the computer 105. However, the digital camera 104 is connected to any one of a plurality of n computers 103i provided for each projector 102i, and the host of the computer 105 is connected. The function may be shared.
Further, in the description of the above-described embodiment, the screen 101i is a quadrangle and the number of surfaces is three. However, the present invention does not depend on the shape and the number of surfaces of each screen. For example, the present invention can also be applied to a composite screen such as a regular dodecahedron (regular pentagon, 12 planes), a rectangular parallelepiped (tetragonal, 6 planes). Further, FIG. 2 shows an example of rear projection, but front projection can also be handled due to the method of taking corresponding points of the test pattern as shown in FIG.

([2] Application to stereoscopic vision, detection of distortion parameters)
Hereinafter, an embodiment applied to polarization-type stereoscopic vision will be described with reference to the drawings.
FIG. 20 is a flowchart for explaining an embodiment of a distortion parameter generation method for polarization type stereoscopic vision, and FIG. 21 is a flowchart for explaining an embodiment of an image generation method for stereoscopic vision. FIG. 22 is a functional block diagram illustrating an embodiment of a distortion parameter generation apparatus and a video generation apparatus related to stereoscopic viewing.
22A, 22B, and 22C, 2001 is a screen, 20021 is a left-eye projector that projects a left-eye image, 20022 is a right-eye projector that projects a right-eye image, and 20031 is the left-eye projector 20001. The computer for the left eye provided corresponding to the projector for the right eye, the computer for the right eye provided for the projector for the right eye 20022, the digital camera connected to the computer for the right eye 2004, and the computer for the right eye 2006. A position detection means constituted by a computer 20032, 2007 a matrix calculation means constituted by the right eye computer 20032, and 2010 a storage device. 20121 is an image reading unit configured by the left eye computer 20003, 20122 is an image reading unit configured by the right eye computer 20032, 20131 is a virtual surface rotating unit 20132 configured by the left eye computer 20031, 20132 Is a virtual surface rotating means constituted by the right eye computer 20032, 20141 is an image pasting means constituted by the left eye computer 20031, 20142 is an image pasting means constituted by the right eye computer 20032, Reference numeral 20151 denotes a first frame memory provided in the left-eye computer 20031; reference numeral 20152 denotes a first frame memory provided in the right-eye computer 20032; and reference numeral 1611 denotes a left-eye computer. The second frame memory provided in the data 20031, 20162 is the second frame memory provided in the right-eye computer 20032, 2010171 is a three-dimensional graphics generation means configured by the left-eye computer 20031, The three-dimensional graphics generation means includes the right eye computer 20032. Reference numeral 2021 denotes a graphics board provided in the computer 20031, and reference numeral 2022 denotes a graphics board provided in the computer 20032, which includes, for example, the first frame memory 2015i and the second frame memory 2016i.
FIG. 22D shows a state in which the left and right eye images are matched for stereoscopic viewing.

([2-1] Generation of distortion parameters)
Generation of distortion parameters by the above configuration will be described.
(A) The screen 2001 is imaged with the digital camera 2004 (P201 in FIG. 20).
(B) The right eye computer 20032 to which the digital camera 2004 is connected detects vertices (corners) constituting the captured screen 2001 by image recognition, and obtains the position (P202 in FIG. 20).
(C) For example, the left-eye projector 20001 projects a test pattern formed by the left-eye computer 20031 in which two types of rectangles with high contrast are arranged vertically and horizontally on the screen 2001 (P203 in FIG. 20). This test pattern is the same as that in the above-mentioned plural screens.
(D) The position detection means 2006 of the right-eye computer 20032 to which the digital camera 2004 is connected detects, from the captured image, a corner formed by a portion where four rectangular portions of the test pattern are adjacent by image recognition. The position is obtained (P204 in FIG. 20).
(E) The matrix calculation means 2007 of the right-eye computer 20032 connected to the digital camera 2004 uses the vertex obtained in the second process (P202) and the corner obtained in the fourth process (P204) to generate the left eye. Conversion matrix, scale matrix, and movement matrix are obtained (P205 in FIG. 20).
(F) The projector for the right eye 20022 projects on the screen 2001 a test pattern in which two types of rectangles with high contrast created by the right eye computer 20032 are arranged vertically and horizontally (P206 in FIG. 20). This test pattern is projected in accordance with the same size as that in the third process (P203).
(G) The position detection unit 2006 of the right-eye computer 20032 to which the digital camera 2004 is connected forms a portion where four of the rectangular portions of the test pattern are adjacent from the image captured in the sixth process (P206). A corner is detected by image recognition, and its position is obtained (P207 in FIG. 20).
(H) The right eye computer 20032 to which the digital camera 2004 is connected includes a right eye conversion matrix based on the vertex obtained in the second process (P202) and the corner obtained in the seventh process (P207), A scale matrix and a movement matrix are obtained (P208 in FIG. 20).
(I) Each conversion matrix, scale matrix, and movement matrix obtained in the fifth process (P205) and the eighth process (P208) are stored in the storage device 2010 as distortion parameters (P209 in FIG. 20).
In the above explanations (c) to (i), in addition to the above description, the same description as ([1] Generation of distortion parameters) explained for a plurality of screens is performed.

([2-2] Video generation)
A description will be given of the generation of a polarized stereoscopic image with the above configuration.
(A) For example, the left-eye image reading unit 20121 of the left-eye computer 20031 stores the contents of the first frame memory 20151 in which the result of rendering (image generation) using the three-dimensional graphics generation unit 20171 is stored. Is read (S201 in FIG. 21). As shown in FIG. 16A, an image without distortion is stored in the first frame memory 20151.
(B) The left-eye virtual surface rotation unit 20131 of the left-eye computer 20031 generates the left-eye virtual surface for texture mapping of the read image in the above ([2-1] Distortion parameter generation). The rotation is performed using the distortion parameter [H _i ^total ] ⁻¹ (S202 in FIG. 21). Refer to FIG.
(C) The left-eye image pasting means 20141 pastes the image read in the first process (S201) as a texture pattern on the virtual surface rotated in the second process (S202) (S203 in FIG. 21). ). At this time, the projection method of image generation is parallel projection. The reverse distortion is applied by pasting.
(D) The image subjected to reverse distortion, that is, the image pasted on the virtual surface (texture mapping surface) in the third process (S203) is stored in the second frame memory 20161 (S204 in FIG. 21).
(E) The contents of the second frame memory 20161 are projected on the screen 2001 via the left-eye projector 200001 (S205 in FIG. 21). At this time, the distortion of the image is corrected.
(F) For example, the right-eye image reading unit 20122 of the right-eye computer 20032 stores the contents of the first frame memory 20152 in which the result of rendering (image generation) using the three-dimensional graphics generation unit 20152 is stored. Is read (S206 in FIG. 21). As shown in FIG. 16A, an image without distortion is stored in the first frame memory 20152.
(G) The right-eye virtual surface rotation unit 20132 of the right-eye computer 20032 generates the right-eye virtual surface for texture mapping of the read image in the above ([2-1] Generation of distortion parameters). The rotation is performed using the distortion parameter [H _i ^total ] ⁻¹ (S207 in FIG. 21). Refer to FIG.
(H) The right-eye image pasting means 20142 pastes the image read in the sixth process (S206) as a texture pattern on the virtual surface rotated in the seventh process (S207) (S208 in FIG. 21). ). At this time, the projection method of image generation is parallel projection. The reverse distortion is applied by pasting.
(I) The image subjected to reverse distortion, that is, the image pasted on the virtual surface (texture mapping surface) in the sixth process (S206) is stored in the second frame memory 20162 (S209 in FIG. 21).
(J) The contents of the second frame memory 20162 are projected on the screen 2001 via the right-eye projector 200021 (S210 in FIG. 21). At this time, the distortion of the image is corrected.

(Other)
In the above description, the digital camera 2004 is connected to the right-eye computer 20032. However, the digital camera 2004 may be connected to the left-eye computer 20031 with the same function. In addition to 20032, a host computer may be prepared to function independently as in the first embodiment.

([3-1] Generation of distortion parameters related to stereoscopic viewing on multiple screens)
In the above example, a case of a plurality of screens and a case of application to polarization-type stereoscopic vision on a single screen have been described. However, the technology of polarization-type stereoscopic vision ([2-1] distortion parameter generation) By applying the technique of ([1-1] distortion parameter generation) to each screen, distortion parameters for the left and right eyes can be obtained for each screen. A plurality of screens, projectors for left and right eyes on each screen, a digital camera, and a storage device are provided, and the process of ([2-1] distortion parameter generation) described above ([1-1] distortion parameter generation) is performed. This is realized by performing each screen as described in the above.

([3-2] Video generation related to stereoscopic viewing on multiple screens)
Projecting stereoscopic images on a composite screen by generating left and right eye images on each screen using the parameters generated in the above ([3-1] Generation of distortion parameters related to stereoscopic viewing on multiple screens). Can do.

When displaying an image for each of a plurality of screens described in the first and third embodiments, adjacent projectors projected on the screen are arranged so as to overlap each other, and the image is displayed in an overlap area ( (Adjacent part) are displayed so that they overlap, and the color of the image of the overlapping part is set to << C _A >> for the projector _A , and << C _B >> for the color of the projector B,
<< C >> = <C _A >> α (x, y) + (1−α (x, y)) << C _B >>
Thus, the projector characteristics are gently changed to perform color mixing. For example, color mixing is performed in the portions 191 to 197 in the band on the wavy line shown in FIG.

It is a flowchart explaining one Example of the production | generation method of a distortion parameter, and a video generation method. It is a functional block diagram for explaining an embodiment of a distortion parameter generation device and a video generation device. It is a figure explaining the conventional image generation | occurrence | production. Principle diagram for determining spatial positional relationship It is explanatory drawing of how to obtain | require a conversion matrix. It is explanatory drawing of how to obtain | require a conversion matrix. It is explanatory drawing of a test pattern. It is explanatory drawing of the corner search area | region of a test pattern. It is explanatory drawing of the corner search of a test pattern. It is explanatory drawing of the corner search area | region of a test pattern. It is explanatory drawing of the corner search of a test pattern. It is explanatory drawing of the process which calculates | requires the relationship of the several screen of a projector. It is explanatory drawing of the agreement of a screen and a display image. It is a figure explaining calculation of a scale. It is a figure explaining calculation of a scale. It is explanatory drawing of rendering. It is explanatory drawing of the image generation by multiple computers. It is explanatory drawing of the example which projects the image of a several projector on a screen. It is explanatory drawing of the example of a display in the case of having a plurality of screens which projected the image of a plurality of projectors. It is a flowchart explaining one Example of the production | generation method of a distortion parameter. It is a flowchart explaining one Example of a video generation method. It is a functional block diagram for explaining an embodiment of a distortion parameter generation device and a video generation device. It is a figure explaining the display of the overlap area | region of a projector.

Explanation of symbols

DESCRIPTION OF SYMBOLS 101i ... Screen, 102i ... Projector, 103i ... Computer, 104 ... Digital camera, 105 ... Computer, 106 ... Position detection means, 107 ... Matrix calculation means, 117i ... Three-dimensional graphics production | generation means, 110 ... Memory | storage device, 112i ... Image Reading means 113i... Virtual surface rotating means 114i. Image pasting means 115i... First frame memory 116i.

Claims (14)

In a projection space configured by combining a plurality of n screens, a plurality of p projectors provided for each of the plurality of n screens, a plurality of m computers provided for each of the plurality of p projectors, and a computer A digital camera connected to the storage device and a storage device,
A first step of projecting a test pattern common to each of the projectors in a checkered pattern in which white and black rectangles alternately arranged vertically and horizontally created by a computer i corresponding to the projector onto a screen corresponding to the projector;
A second process in which the screen on which the test pattern is projected is imaged by the digital camera disposed at a position serving as a viewpoint or behind the viewpoint position ;
From the captured image, the vertex of the test pattern necessary for matching the display image and the screen, and the rectangle of the test pattern necessary for correcting the distortion of each projection image and synthesizing it into one smooth image a corner four portions constitute the adjacent portions, is detected through image recognition, and a third program for determining the position, using the computer connected to the digital camera,
The said third said corner and the apex determined in course of transformation matrix for matching the displayed image distortion adjustment and synthesis, and a screen, a scale matrix and moving matrix is obtained by the computer connected to the digital camera first 4 courses,
After the first to fourth processes are executed for each of the plurality of n screens by a plurality of p projectors provided, the projected images are combined into one smooth image and matched with the screen. A synthesis matrix obtained by combining the products of the conversion matrix, the scale matrix, and the movement matrix obtained in the fourth process is composed of a fifth process for storing in a storage device for each projector as a distortion parameter. Generation method of distortion parameters. In a projection space configured by combining a plurality of n screens, a plurality of p projectors provided corresponding to each of the plurality of n screens and a plurality of m computers provided corresponding to the plurality of p projectors are provided. And
In the processing of each of a plurality of computers,
A first process of reading an image from a frame memory in which an image generated by a computer is stored;
A second step of rotating a virtual surface for texture mapping of the read image using the distortion parameter generated in claim 1;
A third process of pasting the read image as a texture pattern on the rotated virtual surface;
A fourth process of storing the image pasted on the rotated virtual surface in a frame memory;
An image generation method comprising: a fifth process of projecting an image in a frame memory stored in the fourth process onto a corresponding screen by a projector corresponding to the computer. In a projection space configured by combining a plurality of n screens, a plurality of p projectors provided for each of the plurality of n screens, a plurality of m computers provided for each of the plurality of p projectors, and a computer A distortion parameter generation device having a digital camera connected to the storage device and a storage device,
The plurality of projectors that project a test pattern common to each projector in a checkered pattern in which white and black rectangles are alternately arranged vertically and horizontally for each screen,
A digital camera that images the test pattern, which is arranged behind the position that becomes the viewpoint or the viewpoint position , and is projected on the screen;
Wherein it is constituted by a digital camera connected to the computer, from the captured image, necessary for matching the displayed image and the screen, the vertex constituting each screen, one smooth image to correct distortion of the projected image The corners formed by the adjacent portions of the four rectangular portions of the test pattern , which are necessary for synthesis, are detected by image recognition , and position detection means for obtaining the positions thereof, and each of the positions obtained by the position detection means. the said and the apex of each screen corner distortion adjusting the synthesis and transformation matrix according to the screen display image on the screen to match the scale matrix and moving matrix, the matrix calculation means for calculating a computer connected to the digital camera When,
A distortion parameter generation apparatus comprising: a storage device that stores , as a distortion parameter , a composite matrix obtained by combining products of each conversion matrix, scale matrix, and movement matrix obtained by the matrix calculation unit , as a distortion parameter . In a projection space configured by combining a plurality of n screens, a plurality of p projectors provided corresponding to each of the plurality of n screens and a plurality of m computers provided corresponding to the plurality of p projectors are provided. And
In each of the plurality of m computers,
Image reading means for reading an image from a frame memory in which the generated image is stored;
A virtual surface rotating means for rotating the virtual surface for texture mapping the read image using the distortion parameter generated in claim 3;
Image pasting means for pasting the read image as a texture pattern on the rotated virtual surface;
A frame memory for storing video pasted on the rotated virtual surface;
An image generating apparatus comprising: a projector that projects an image in the stored frame memory onto a screen corresponding to the computer. In a projection space including a screen, a left-eye projector, a right-eye projector, a left-eye computer and a right-eye computer provided for each of the left-eye projector and the right-eye projector, and a digital camera connected to the computer And a storage device,
A first process of imaging the screen with the digital camera;
A second step of detecting, by image recognition, vertices constituting the screen necessary for matching the display image and the screen from the captured image, and obtaining the position using the computer connected to the digital camera;
A test pattern common to each of the checkered projectors in which white and black rectangles created by the left-eye or right-eye computer are alternately arranged vertically and horizontally is projected onto the screen by any one projector for left or right eyes. And course
From the captured image, a corner formed by the adjacent portions of the four rectangular portions of the test pattern necessary for correcting the distortion of each projected video and synthesizing it into one smooth video is detected by image recognition. A fourth process for determining the position using the computer connected to the digital camera;
Wherein said the said corner obtained in said fourth courses the vertices obtained in 2 courses, left or either the transformation matrix for the right eye for matching the strain regulating synthesis and a screen display image, A fifth process for obtaining a scale matrix and a movement matrix by the computer connected to the digital camera;
A test pattern common to each projector of a checkered pattern in which white and black rectangles are alternately arranged vertically and horizontally created on a computer is used for the eyes other than the left or right projector according to the third process. A sixth process of projecting from one projector;
From the captured image, a corner formed by the adjacent portions of the four rectangular portions of the test pattern necessary for correcting the distortion of each projected video and synthesizing it into one smooth video is detected by image recognition. A seventh step for determining the position using the computer connected to the digital camera;
Matching distortion adjustment and composition and screen and display image other than the left or right eye according to the fifth process based on the vertex obtained in the second process and the corner obtained in the seventh process An eighth process for obtaining the conversion matrix, scale matrix and movement matrix for the other of the left or right eye for using the computer connected to the digital camera;
A synthesized matrix obtained by synthesizing the product of each conversion matrix, scale matrix, and moving matrix obtained in the fifth and eighth processes to synthesize the projected image into one smooth image and match the screen . A distortion parameter generation method comprising the ninth step of storing the distortion parameter in a storage device. In a projection space including a screen, a left-eye projector, a right-eye projector, and a computer provided corresponding to each of the left-eye projector and the right-eye projector,
In the processing of the computer,
A first process of reading an image from a frame memory in which one of the left-eye image and the right-eye image generated by the computer is stored;
A second step of rotating either the left-eye or right-eye virtual surface for texture mapping the read image using either the left-eye or right-eye distortion parameter generated in claim 5;
A third process for pasting the read-out image as a texture pattern on the virtual surface rotated by the second process;
A fourth process of storing the video pasted on the virtual surface in the third process in a frame memory;
A fifth process of projecting the image in the frame memory stored in the fourth process onto the screen by either the left-eye projector or the right-eye projector;
A sixth process of reading an image from a frame memory in which an image for the eye other than one of the left and right eye images generated by the computer and read by the first process is stored;
The virtual surface other than the left-eye or right-eye virtual surface in the second course, which is one of the left-eye and right-eye virtual surfaces for texture mapping of the read image, A seventh process of rotating using one of the generated distortion parameters for the left eye or the right eye, other than the left eye parameter or the right eye parameter in the second process;
An eighth process of pasting the image read out in the sixth process as a texture pattern on the virtual surface rotated in the seventh process;
A ninth process for storing the image pasted on the virtual surface in the eighth process in a frame memory;
The image in the frame memory stored in the ninth process is projected onto the screen by one of the left-eye or right-eye projectors other than the left-eye or right-eye projector according to the fifth process. 10th course,
A video generation method comprising: In a projection space including a screen, a left-eye projector, a right-eye projector, a left-eye computer and a right-eye computer provided for each of the left-eye projector and the right-eye projector, and a digital camera connected to the computer And a storage device,
A projector for the left eye that projects a test pattern common to each checkered projector in which white and black rectangles are alternately arranged vertically and horizontally on the screen;
A right-eye projector that projects a test pattern common to each checkered projector in which white and black rectangles are alternately arranged vertically and horizontally on the screen;
The digital camera which is arranged at a position to be a viewpoint, images the screen and images each test pattern projected on the screen from the left-eye projector and the right-eye projector;
The computer is connected to the digital camera, and from the screen image captured by the digital camera, the vertices constituting the screen necessary for matching the display image and the screen are detected by image recognition, and the position is obtained. In addition, the test pattern necessary for correcting the distortion of each projection video and synthesizing it into one smooth video from the image captured by the digital camera with the test pattern from the left-eye projector and the right-eye projector. A corner formed by four adjacent rectangular portions is detected by image recognition, and a position detecting means for obtaining the position, and the vertex obtained by the position detecting means and each of the left and right eye projectors are projected. the said corner, match the synthetic and screen display image distortion adjustment Because each of the transformation matrix for the right and left eyes, scale and matrix and matrix calculation means for moving matrix calculated by the computer, the projected image obtained by the matrix calculation means is combined into a single smooth image and to match the screen A distortion parameter generation comprising: a storage device for storing a composite matrix obtained by combining products of conversion matrices, scale matrices, and movement matrices for left and right eyes as distortion parameters for left and right eyes apparatus. In a projection space including a screen, a left-eye projector, a right-eye projector, and a left-eye computer and a right-eye computer provided for each of the left-eye projector and the right-eye projector,
In the computer,
Left-eye image reading means for reading an image from a frame memory in which a left-eye image is stored;
Right-eye image reading means for reading an image from a frame memory in which an image for the right eye is stored;
Virtual surface rotation means for left eye that rotates a virtual surface for texture mapping the image read by the image read means for left eye using the distortion parameter for left eye generated in claim 7;
A virtual surface rotating means for the right eye that rotates a virtual surface for texture mapping the image read by the image reading means for the right eye using the distortion parameter for the right eye generated in claim 7;
A left-eye image pasting unit that pastes the image read out by the left-eye image reading unit on the virtual surface rotated by the left-eye virtual surface rotating unit;
Right-eye image pasting means for pasting the image read by the right-eye image reading means as a texture pattern on the virtual surface rotated by the right-eye virtual surface rotating means;
A frame memory for the left eye for storing the image pasted on the virtual plane by the image pasting means for the left eye;
A frame memory for the right eye for storing the image pasted on the virtual surface by the image pasting means for the right eye;
A left-eye projector that projects the image in the stored left-eye frame memory onto a screen;
An image generator comprising: a right-eye projector that projects the image in the stored right-eye frame memory onto a screen.   In a projection space configured by combining a plurality of n screens, two or more p / 2 sets of projectors for the left and right eyes provided corresponding to each of the plurality of n screens, and two or more projectors for the left and right eyes 6. The distortion parameter according to claim 5, comprising a plurality of m sets of computers for left and right eyes provided corresponding to each of two sets of projectors, a digital camera connected to the computer, and a storage device. Generation method.   In a projection space configured by combining a plurality of n screens, two or more p / 2 sets of projectors for the left and right eyes provided corresponding to each of the plurality of n screens, and two or more projectors for the left and right eyes The video generation method according to claim 6, further comprising: a plurality of m sets of computers for left and right eyes provided corresponding to each of the two sets of projectors.   In a projection space configured by combining a plurality of n screens, two or more p / 2 sets of projectors for the left and right eyes provided corresponding to each of the plurality of n screens, and two or more projectors for the left and right eyes 8. The distortion parameter according to claim 7, comprising a plurality of m sets of computers for left and right eyes provided corresponding to each of two sets of projectors, a digital camera connected to the computer, and a storage device. Generator.   In a projection space configured by combining a plurality of n screens, two or more p / 2 sets of projectors for the left and right eyes provided corresponding to each of the plurality of n screens, and two or more projectors for the left and right eyes 9. The image generating apparatus according to claim 8, further comprising a plurality of m sets of computers for left and right eyes provided corresponding to each of the two sets of projectors.   11. The image generation method according to claim 2, wherein adjacent areas of the projectors projected on the screen are partially overlapped with each other.   13. The video generation apparatus according to claim 4, wherein adjacent video images of the projectors projected onto the screen are partially overlapped with each other.

Images