JP5236219B2 - Distortion correction and integration method using divided imaging, mapping function generation method therefor, distortion correction and integration device using divided imaging, and mapping function generation apparatus therefor - Google Patents

Description

  The present invention relates to distortion correction and integration by division imaging in computer graphics, a mapping function generation method therefor, an apparatus thereof, and an application thereof.

A high resolution and a wide field of view are required to obtain an immersive and realistic feeling in training simulators and research simulators. As a method for realizing this with good cost performance, a projector mosaic or a multi-projection technique is anticipated in which images with reverse distortion are projected from a number of projectors onto a flat screen and integrated into a continuous image. Conventionally, a method of optically correcting distortion using an electronic circuit or a lens has been widely used. However, there are many problems such as a fixed screen size and complicated adjustment when a projector is installed. In addition, since multiple screens are used to form a large screen, an expensive projector with a dedicated circuit that relaxes the boundary is required, and a great deal of time and effort is required for alignment, adjustment of projector characteristics, and brightness shift due to aging. Cost.
Further, as shown in Non-Patent Document 1, there are three lines of an image projected from a projector j (1... N) whose installation position is unknown and a distortion image on the screen imaged by a camera whose installation position is unknown. There is a matrix that is associated with a matrix of three columns and uses this matrix to apply reverse distortion to the image of the projector coordinates to correct the distortion of the image, but this is limited to a flat screen.
Information Processing Society of Japan, “FIT2003 Information Science and Technology Forum General Lecture Collection Vol. 3”, published on August 25, 2003, p. 401-403

  The present invention provides a method and means for facilitating the generation of real-time video by displaying a high-resolution large screen on a quadratic curved screen such as a dome using a plurality of relatively low-resolution projects for projecting partial images. It is to provide.

In order to solve the above-described problem, a mapping function generation method for distortion correction and integration by divided imaging according to claim 1 is divided into a quadric curved screen, the screen is divided into adjacent screens, and adjacent projection portions include overlapping portions and are continuous. A plurality of n projectors arranged as described above, a stereo camera arranged so as to shoot a screen divided for each of the plurality of projectors, a computer, and a storage device. The first process of performing the three-dimensional restoration of the test points projected from the projectors pi on the screen by the stereo camera, and the three-dimensional points of the restored projectors pi are integrated into one coordinate system. A second step of estimating a quadric surface coefficient matrix (Q) representing the screen shape from the points, and each projector p after the integration From a two-dimensional point of the three-dimensional point and the test point corresponding to the third step of estimating the position and orientation matrix for each projector pi a (P), the whole by dividing the projected surface of the secondary curved screen shooting A quadratic surface coefficient matrix including a plurality of reference virtual cameras vm (m = 1... M) that capture the projection surface of the quadric surface screen projected by the projector pi and including the estimated error. (Q) and a calculation using the position and orientation matrix of the plurality of reference virtual cameras vm (m = 1... M) and the position and orientation matrix (P) of each projector pi including the estimated error. A fourth step of obtaining a mapping function Ψvmpi between the reference virtual camera vm and the projector pi , including an error of the surface coefficient matrix (Q) and an error of the position and orientation matrix (P) of each projector pi; A viewpoint virtual camera e having position and orientation information is set at a position where the clean is viewed, a mapping function Ψevm between the position and orientation of the viewpoint virtual camera e and the reference virtual camera vm having the position and orientation is obtained, and the fourth process is performed. A fifth process for obtaining the mapping function Ψepi between each projector and the viewpoint virtual camera e by the mapping function Ψvmpi, and a sixth process for obtaining the inverse function by the mapping function Ψepi of the fifth process. It is characterized by.

Mapping function generation method for distortion correction and integration by division imaging according to claim 2 for solving the foregoing problem resides in that in Claim 1, the fourth process, the mapping function Ψvmpi including an error as the initial value, the It is characterized by optimizing with minimum error.

  In order to solve the above problem, a distortion correction and integration method using divided imaging according to claim 3 is a quadratic curved screen, and a plurality of screens that are divided into the screen and arranged so that adjacent projection portions include overlapping portions and are continuous. 3. A projector comprising: n projectors; a computer provided for each projector; and a storage device, wherein each computer generates an image for each area projected by each projector in a pixel buffer and claim 1 or 2 in advance. A first process of generating, in a pixel buffer, an image obtained by correcting the distortion of the generated image using the inverse function of the mapping for each projector obtained in step 2, and the quadratic curved surface It consists of the 2nd process of projecting on a screen.

In order to solve the above problems, a mapping function generation method for distortion correction and integration by divided imaging according to claim 4 is divided into a quadratic curved screen, the screen is divided into the screens, and adjacent projection portions are continuous and include overlapping portions. A plurality of n projectors arranged as described above, a stereo camera arranged so as to shoot a screen divided for each of the plurality of projectors, a computer, and a storage device. The first process of performing the three-dimensional restoration of the test points projected from the projectors pi on the screen by the stereo camera, and the three-dimensional points of the restored projectors pi are integrated into one coordinate system. A second step of estimating a quadric surface coefficient matrix (Q) representing the screen shape from the points, and each projector p after the integration From a two-dimensional point of the three-dimensional point and the test point corresponding to the third step of estimating the position and orientation matrix for each projector pi a (P), the whole by dividing the projected surface of the secondary curved screen shooting A quadratic surface coefficient matrix including a plurality of reference virtual cameras vm (m = 1... M) that capture the projection surface of the quadric surface screen projected by the projector pi and including the estimated error. (Q) and a calculation using the position and orientation matrix of the plurality of reference virtual cameras vm (m = 1... M) and the position and orientation matrix (P) of each projector pi including the estimated error. A fourth step of obtaining a mapping function Ψvmpi between the reference virtual camera vm and the projector pi , including an error of the surface coefficient matrix (Q) and an error of the position and orientation matrix (P) of each projector pi; A right-eye (or left-eye) viewpoint virtual camera e having position and orientation information is set at either the right eye or the left eye to view the clean, and the position and orientation of the right-eye (or left-eye) viewpoint virtual camera e The mapping function Ψevm between the reference virtual cameras vm having the position and orientation is obtained, and the right eye (or left eye) between each projector and the viewpoint virtual camera e for the right eye (or left eye) is determined by the mapping function Ψvmpi in the fourth process. ) Mapping function Ψr (l) epi for the left eye (or right eye) viewpoint virtual camera e having position and orientation information at the other position different from either the right eye or the left eye viewing the quadric curved screen And a mapping function Ψevm between the position and orientation of the viewpoint virtual camera e for the left eye (or right eye) and the reference virtual camera vm having the position and orientation is obtained. A fifth step of obtaining a left eye (or right eye) mapping function ψl (r) epi between each projector and the left eye (or right eye) viewpoint virtual camera e using the mapping function ψvmpi in the fourth step. And the sixth step of obtaining the inverse function by the right eye and left eye mapping functions ψrepi and ψlepi of the fifth step.

In order to solve the above-mentioned problem, the mapping function generation method for distortion correction and integration by divided imaging according to claim 5 is the fourth process, wherein the fourth process uses the mapping function Ψvmpi including an error as an initial value , It is characterized by optimizing with minimum error.

  In order to solve the above-described problem, the distortion correction and integration method by divided imaging according to claim 6 includes a quadric curved screen, and a plurality of screens that are divided into the screen and arranged so that adjacent projection portions include overlapping portions and are continuous. n projectors, a computer provided for each projector, and a storage device, and each computer generates right-eye and left-eye images for each area projected by each projector in a pixel buffer and requests in advance. A first step of generating, in a pixel buffer, an image obtained by correcting the distortion of the generated image by using an inverse function of the mapping functions Ψ repi and Ψ lepi for the right eye and the left eye for each projector obtained in item 4 or claim 5; Right eye and left eye images corrected for distortion in the pixel buffer from each projector are converted into the secondary music. It consists of the 2nd process projected on a surface screen.

  In order to solve the above-described problem, the distortion correction and integration method using divided imaging according to claim 7 includes a quadratic curved screen and a plurality of screens that are divided into the screens and that are adjacent to each other so that adjacent projection portions include overlapping portions. An inverse function of the mapping for each projector obtained in advance in claim 1 or 2 and generating an image for each area projected by each projector by the computer having n projectors, a computer, and a storage device Accordingly, a first process of generating a distortion-corrected image of the generated image in the pixel buffer, and a projection of the distortion-corrected image in the pixel buffer from each projector onto the quadric curved screen overlaps in common. It consists of the 2nd process which enlarges the brightness of a part, It is characterized by the above-mentioned.

  In order to solve the above problem, a distortion correction and integration method using divided imaging according to claim 8 is a quadratic curved screen, and a plurality of screens that are divided into the screen and arranged so that adjacent projection portions include overlapping portions and are continuous. each of the projectors obtained in advance according to claim 1 or 2, wherein the projector includes n projectors and video devices, and the video generated from the video devices is divided and generated so as to be projected onto the plurality of n projectors onto the screen. And a second process of projecting the distortion-corrected video image from each projector onto the quadric curved screen. It is characterized by.

  In order to solve the above problem, a distortion correction and integration method using divided imaging according to claim 9 includes a quadratic curved screen, and a plurality of screens that are divided into the screen and arranged so that adjacent projection portions are continuous including overlapping portions. n projectors and video devices, and separately generating and projecting images from the video devices onto the screen onto the plurality of n projectors, and for the right eye for each projector previously obtained in claim 4 A first process of generating a distortion-corrected image of the divided and generated image by an inverse function of a left-eye mapping function, and the distortion-corrected right-eye and left-eye images from the projectors on the quadric surface screen It consists of the 2nd process to project.

  In order to solve the above-described problem, the distortion correction and integration method by divided imaging according to claim 10 has a quadratic curved screen and an overlapping portion that is located in front of the screen and has a common area projected onto the screen. 3. The apparatus according to claim 1, further comprising: a plurality of n projectors arranged in a plurality of projectors; and a video device, wherein the image generated from the video device is divided and generated so as to be projected onto the plurality of n projectors onto the screen. A first process of generating an image obtained by correcting the divisionally generated image by distortion correction using an inverse function of the mapping for each projector obtained, and projecting the distortion corrected image from each projector onto the quadric curved screen It is characterized by comprising the second process of increasing the brightness of the overlapping portions in common.

  In order to solve the above problem, the distortion correction and integration method by divided imaging according to claim 11 is the method according to claim 8, claim 9, or claim 10, wherein the video equipment is a television device, a video playback device, a personal computer. It is characterized by being a computer.

  In order to solve the above-described problem, the distortion correction and integration method by divided imaging according to claim 12 is described in claim 3, claim 6, claim 7, claim 8, claim 9, claim 10, or claim 11. A quadratic curved screen, a plurality of n projectors divided into the screens and arranged so that adjacent projections are continuous including overlapping portions, a computer, and a screen positioned in front of the screen. A stereo camera arranged to take a picture, and a first process of taking a picture with the stereo camera in a state where none of the projectors projects on the screen; and a black picture from each projector on the screen. A second process of projecting and shooting with a stereo camera and repeating n times by a plurality of n projectors; And obtaining the maximum value from the average value of the screen by each projector and the sum of the average values by the difference between the luminance of the image for each projector in the second process and the luminance of the image obtained in the first process. The third process of obtaining the offset value by the difference between the maximum value and the average value, and the computer measures the luminance of the image projected at a fixed period, determines whether it is a bright image or a dark image according to the average value, In this case, the image processing apparatus includes a fourth process of generating an image by reducing the contrast of the overlapping portion of the image of each projector by the offset value and reducing the luminance by the offset value.

  In order to solve the above-mentioned problem, the distortion correction and integration method by divided imaging according to claim 13 includes a quadratic curved screen and a plurality of screens that are divided into the screen and arranged so that adjacent projection portions are continuous including overlapping portions. n projectors, a polarizing plate provided in front of each of the projectors, a computer provided for each of the projectors, and a storage device, and a video for each area projected by each projector by each of the computers is used as a pixel buffer. A first step of generating an image in which a distortion correction of the generated image is generated in a pixel buffer by using an inverse function of the mapping function for each projector previously generated in claim 1 or 2, and from each projector, A second process of projecting a distortion corrected image in the pixel buffer onto the quadric curved screen , It is characterized in the that and a third process of dimmed by the polarizing plate to represent nighttime scene.

In order to solve the above problem, a mapping function generating apparatus for distortion correction and integration by divided imaging according to claim 14 is divided into a quadric curved screen and the screen, and adjacent projection portions are continuous including overlapping portions. A plurality of projectors arranged in such a manner, a stereo camera arranged so as to photograph the screen positioned in front of the screen, and arranged so as to photograph the whole by dividing the projection surface of the quadric curved screen And a plurality of reference virtual cameras vm (m = 1... M) for photographing the projection surface of the quadric curved screen projected by the projector pi, and position and orientation information arranged at the position where the quadric curved screen is viewed. A viewpoint virtual camera e, a computer, and a storage device, and a test point projected from each projector pi onto the screen A test point three-dimensional restoration means for performing three-dimensional restoration by imaging with a laser, and the restored three-dimensional point of each projector pi is integrated into one coordinate system, and a secondary representing the screen shape from the integrated three-dimensional point A quadratic surface coefficient matrix (Q) estimation means for estimating a surface coefficient matrix (Q), and a position and orientation of each projector pi from the two-dimensional points of the integrated three-dimensional points of the projectors pi and the corresponding test points. A projector position / orientation estimation means for estimating a matrix (P), a quadratic surface coefficient matrix (Q) including the estimated error, a position / orientation matrix of the plurality of reference virtual cameras vm (m = 1... M) , By using the position and orientation matrix (P) of each projector pi including the estimated error, a reference including an error of the quadratic surface coefficient matrix (Q) and an error of the position and orientation matrix (P) of each projector pi Virtual First mapping function calculation means for obtaining a mapping function ψvmpi between the camera pi and the projector pi, a mapping function ψevm between the position and orientation of the viewpoint virtual camera e and the reference virtual camera vm having the position and orientation, and the first The second mapping function calculating means for obtaining the mapping function Ψepi between each projector and the viewpoint virtual camera e by the mapping function Ψvmpi by the mapping function calculating means, and the mapping function Ψepi by the second mapping function calculating means Inverse function calculation means for obtaining the inverse function and a storage device for storing the mapping functions ψvmpi, ψepi and the inverse function for each of the plurality of projectors.

In order to solve the above problem, the mapping function generation apparatus for distortion correction and integration by divided imaging according to claim 15 is the mapping function generation apparatus according to claim 14, wherein the first mapping function calculation means includes a mapping function including an error. the Ψvmpi the initial value and is characterized in that it comprises optimizing means for optimizing and the minimum error.

  In order to solve the above problem, a mapping function generating apparatus for distortion correction and integration by divided imaging according to claim 16 is divided into a quadratic curved screen and the screen, and adjacent projection portions are continuous including overlapping portions. A plurality of projectors arranged in such a manner, a computer provided for each of the projectors, and a storage device, and each computer generates an image for each area projected by each projector in a pixel buffer and is charged in advance. The image generation means for generating, in a pixel buffer, an image obtained by correcting the distortion of the generated image, by the inverse function of the mapping for each projector obtained in item 14 or claim 15, and distortion correction from the pixel buffer of each computer A plurality of projectors for projecting each image onto the quadric curved screen, Those characterized by comprising.

In order to solve the above problem, a mapping function generating apparatus for distortion correction and integration by divided imaging according to claim 17 is divided into a quadric curved screen and the screen, and adjacent projection portions are continuous including overlapping portions. A plurality of projectors arranged in such a manner, a stereo camera arranged so as to photograph the screen positioned in front of the screen, and arranged so as to photograph the whole by dividing the projection surface of the quadric curved screen A plurality of reference virtual cameras vm (m = 1... M) for photographing the projection surface of the quadric curved screen projected by the projector pi, and a position and orientation arranged at the position of the right eye viewing the quadric curved screen. A right-eye viewpoint virtual camera er having information, a left-eye viewpoint virtual camera el having position and orientation information arranged at the position of the left eye, and a computer A test point projected on the screen from each projector pi with a stereo camera to perform a three-dimensional restoration, and a three-dimensional point of each restored projector pi. Quadratic surface coefficient matrix (Q) estimating means for estimating a quadratic surface coefficient matrix (Q) representing a screen shape from three-dimensional points after integration into one coordinate system, and for each projector pi after the integration Projector position / orientation estimation means for estimating the position / orientation matrix (P) of each projector pi from the three-dimensional points and the corresponding two-dimensional points of the test points; a quadratic surface coefficient matrix (Q) including the estimated error ; the position and orientation matrix of the plurality of reference virtual camera vm (m = 1 ... M) , and by calculation using the position and orientation matrix for each projector pi including an error that the estimated (P), 2-order songs Error and the position and orientation matrix for each projector pi of the coefficient matrix (Q) includes an error of (P), a first mapping function calculation means for calculating a mapping function Ψvmpi between the reference virtual camera vm and the projector pi, the right eye (or The mapping function Ψevm between the position and orientation of the viewpoint virtual camera e for the left eye) and the reference virtual camera vm having the position and orientation is obtained, and each projector and the right eye (or A right eye (or left eye) mapping function Ψr (l) epi between the left eye viewpoint virtual camera e and the position and orientation of the left eye (or right eye) viewpoint virtual camera e are obtained. A mapping function Ψevm between the reference virtual cameras vm is obtained, and the mapping function Ψvm by the first mapping function calculating means is obtained. a second mapping function calculating means for obtaining a left eye (or right eye) mapping function Ψl (r) epi between each projector and the left eye (or right eye) viewpoint virtual camera e by pi; Inverse function calculation means for obtaining the inverse function by the right eye and left eye mapping functions Ψrepi, Ψlepi of the mapping function calculation means of the above, and a storage device for storing the mapping functions Ψvmpi, Ψepi and the inverse function for each of the plurality of projectors It is characterized by.

In order to solve the above problem, the mapping function generating apparatus for distortion correction and integration by divided imaging according to claim 18 is the one according to claim 17, wherein the first mapping function calculation means includes a mapping function including an error. the Ψvmpi the initial value and is characterized in that it comprises optimizing means for optimizing and the minimum error.

  In order to solve the above-mentioned problem, the distortion correction and integration device by divided imaging according to claim 19 includes a quadratic curved screen, and a plurality of screens divided into the screen and arranged so that adjacent projection portions are continuous including overlapping portions. n projectors, a computer provided for each projector, and a storage device, and each computer generates right-eye and left-eye images for each area projected by each projector in a pixel buffer and requests in advance. An image generation means for generating in the pixel buffer an image obtained by correcting the distortion of the generated image by the inverse function of the mapping for the right eye and the left eye for each projector obtained in claim 17 or claim 18, and distortion in the pixel buffer A plurality of n programs that project the corrected right-eye and left-eye images onto the quadric curved screen. And it is characterized in that comprising the Ekuta.

  In order to solve the above-mentioned problem, the distortion correction and integration device by divided imaging according to claim 20 is a quadratic curved screen, and a plurality of screens that are divided into the screen and are arranged so that adjacent projection portions are continuous including overlapping portions. n projectors, a computer, and a storage device, and each computer generates an image of each area projected by each projector in a pixel buffer and each projector obtained in advance in claim 14 or claim 15 Image generation means for generating a distortion-corrected image of the generated image in a pixel buffer by an inverse function of mapping, and projecting each distortion-corrected image from the pixel buffer of each computer onto the quadric curved screen A plurality of n projects arranged so as to increase the brightness of common overlapping portions. And it is characterized in that comprising a.

  In order to solve the above-described problem, the distortion correction and integration apparatus according to divided imaging according to claim 21 includes a quadratic curved screen, and a plurality of divisions that are divided into the screen and that are adjacent to each other so that adjacent projection portions include overlapping portions. 16. Each projector having n projectors and a video device is divided and generated so that an image from the video device is projected onto the screen onto the plurality of n projectors. An image generation means for generating a distortion-corrected image of the divided and generated image, and a plurality of n projectors for projecting the distortion-corrected images onto the quadric curved screen. It is a feature.

  In order to solve the above-mentioned problem, the distortion correction and integration device by divided imaging according to claim 22 is a quadratic curved screen, and a plurality of the screens divided into the screens and arranged so that adjacent projection portions include overlapping portions and are continuous. n projectors and video devices, wherein the images from the video devices are divided and generated so as to be projected onto the screen onto the plurality of n projectors, and the right eye for each projector previously obtained in claim 17 and Image generating means for generating a distortion-corrected image of the divided and generated image by an inverse function of mapping for the left eye, and a plurality of n projections for projecting the distortion-corrected right-eye and left-eye images to the quadric surface screen It is characterized by comprising each projector.

  In order to solve the above-described problem, the distortion correction and integration apparatus according to divided imaging according to claim 23 includes a quadratic curved screen, and a plurality of screens that are divided into the screens and that are adjacent to each other so that adjacent projection portions include overlapping portions. 16. Each projector having n projectors and a video device is divided and generated so that an image from the video device is projected onto the screen onto the plurality of n projectors. Image generation means for generating a distortion-corrected image of the divided and generated image by an inverse function of the mapping, and brightness of the common overlapping portion by projecting the distortion-corrected image on the quadric curved screen It is characterized by comprising a plurality of n projectors arranged so as to increase.

  In order to solve the above-mentioned problem, the distortion correction and integration device by divided imaging according to claim 24 is the one according to claim 21 or claim 22 or claim 23, wherein the video equipment is a television device, a video playback device, a personal computer. It is characterized by being a computer.

  In order to solve the above-described problem, the distortion correction and integration device by divided imaging according to claim 25 is described in claim 16, claim 19, claim 20, claim 21, claim 22, claim 23, or claim 24. A quadratic curved screen, a plurality of n projectors that are divided into the screen and are arranged so that adjacent projections are continuous including an overlapping portion, and are located in front of the screen so as to photograph the screen And a stereo camera arranged on the projector for photographing a screen in a first state that is not projected from any of the projectors, and photographing a second state of the screen in which a black state is projected from each projector for each projector. And the first state from the brightness of the image of each projector that projects the second state captured by the camera. By each computer, the offset calculation means for obtaining an offset value by the difference between the maximum value and the average value and obtaining the maximum value by the sum of the average value and the average value of the screen by each projector due to the difference between the brightness of the captured video, The brightness of the image projected at a fixed period is measured, and it is determined whether it is a bright image or a dark image according to the average value. In the case of a dark image, the contrast of the overlapping portion of the image for each projector is calculated by the offset value. And a video generation means for generating a video by lowering the luminance and increasing the offset value.

  In order to solve the above-mentioned problem, the distortion correction and integration device by divided imaging according to claim 26 is a quadratic curved screen, and a plurality of screens that are divided into the screen and arranged so that adjacent projection portions include overlapping portions and are continuous. 15. A projector comprising: n projectors; a polarizing plate; a computer provided for each projector; and a storage device, wherein each computer generates an image for each area projected by each projector in a pixel buffer and Alternatively, the image generation means for generating, in a pixel buffer, an image obtained by correcting the distortion of the generated image by the inverse function of the mapping for each projector obtained in claim 15, and each image after distortion correction from the pixel buffer of each computer A plurality of projectors for projecting the image onto the quadric curved screen, and each of the projectors Provided on the front and is characterized by comprising a polarizing plate for dimming the quantity of light from the projector to represent a night scene.

  In order to solve the above-mentioned problem, a mapping function generating apparatus for distortion correction and integration by divided imaging according to claim 27 is the projector according to claim 14, claim 15, claim 17, or claim 18. Is installed on the floor surface by a flexible arm having a degree of freedom.

  In order to solve the above-mentioned problem, the distortion correction and integration device by divided imaging according to claim 28 is the one according to any one of claims 16 or 19 to 26, wherein each projector is a flexible arm having a degree of freedom. It is characterized by being installed on the floor.

  According to the mapping function generation method for distortion correction and integration by division imaging according to claim 1, the screen shape is estimated based on the test points projected on the quadric curved screen, and the spatial position between the viewpoint position and the projector position is estimated. A relative function is obtained, a mapping function Ψvmpi between a plurality of reference virtual cameras vm and a projector pi for photographing the projection surface of the quadric curved screen is obtained, and a viewpoint virtual camera e having position and orientation information at a position where the quadric curved screen is viewed is obtained. Since the mapping function Ψepi is set between each projector and the viewpoint virtual camera e, even if the camera position and the actual viewpoint position are different from each other on the parametric screen, the reference point is determined without measuring the camera position. Mathematically determined mapping function can be obtained with high accuracy and can be applied to wide viewing angle screens. it can.

  According to the mapping function generation method for distortion correction and integration by divided imaging according to claim 2, the fourth process in claim 1 is to optimize the mapping function Ψvmpi with an initial value as the minimum value. Can do.

  According to the distortion correction and integration method by divided imaging according to claim 3, the image to be projected on one quadric curved screen is generated and projected by a plurality of projectors and a computer equipped therewith, so that the resolution can be increased. It is possible to display a detailed part of the image, and the processing of the computer equipped with multiple projectors can be rendered by correcting and rendering so that different images can be seen correctly when projected on the screen Can reduce the load. Further, since partial videos in different projection areas can be rendered with high resolution, the resolution of the video projected on the screen can be improved without depending on the resolution of the database video as in the past. Thereby, the resolution of the image on the screen can be improved by increasing the number of projectors.

  According to the mapping function generation method for distortion correction and integration by divided imaging according to claim 4, the screen shape is estimated based on the test points projected on the quadric curved screen, and the spatial position between the viewpoint position and the projector position is estimated. A relative function is obtained, a mapping function Ψvmpi between a plurality of reference virtual cameras vm and a projector pi for photographing the projection surface of the quadric curved screen is obtained, and for the right eye and the left eye having position and orientation information at a position where the quadric curved screen is viewed Since the viewpoint virtual camera e is set and the mapping function Ψ repi, lepi between each projector and the right-eye and left-eye viewpoint virtual cameras e is obtained, the camera position differs from the actual viewpoint position with respect to the parametric screen. Even if the camera position is not measured, the mapping function is obtained with high accuracy by mathematically determining the reference point. And can be applied to a wide viewing angle screen.

  According to the mapping function generation method for distortion correction and integration by divided imaging according to claim 5, the fourth process in claim 4 is to optimize the mapping function Ψvmpi as an initial value with the error minimized. Can do.

  According to the distortion correction and integration method by divided imaging according to the sixth aspect, the image to be projected on one quadric curved screen is generated and projected by a plurality of projectors and a computer equipped with them, so that the resolution can be increased. It is possible to display a detailed portion of the image, and the processing of the computer equipped with the multiple projectors is corrected so that it can be correctly viewed when images of different areas are projected on the screen, and the right eye and the left eye Rendering can be used to reduce the rendering load. Further, since partial videos in different projection areas can be rendered with high resolution, the resolution of the video projected on the screen can be improved without depending on the resolution of the database video as in the past. Thereby, the resolution of the image on the screen can be improved by increasing the number of projectors.

  According to the distortion correction and integration method by divided imaging according to claim 7, since the brightness of the projection area is improved by using several projectors and overlapping the projection areas, the necessary brightness without using an expensive projector. The brightness of the projection area can be adjusted by changing the number of projectors, and the brightness at a fixed position can be improved by combining the projector overlay and distortion correction.

  According to the distortion correction and integration method by divided imaging according to claim 8, since the video to be projected onto one quadric curved screen is generated and projected by the video equipment, the resolution can be increased, and a fine portion of the video In addition, the video processing of the video device can reduce the rendering load by rendering a different video for each divided projector. In addition, since partial videos in different projection areas can be rendered with high resolution, the resolution of the video projected on the screen can be improved. Thereby, the resolution of the image on the screen can be improved by increasing the number of projectors.

  According to the distortion correction and integration method by divided imaging according to claim 9, since the video to be projected on one quadric curved screen is generated and projected by the video equipment, the resolution can be increased, and the detailed portion of the video In the video processing of the video equipment, the rendering load can be reduced by rendering different videos for each of the divided projectors for the right eye and the left eye. In addition, since partial videos in different projection areas can be rendered with high resolution, the resolution of the video projected on the screen can be improved. Thereby, the resolution of the image on the screen can be improved by increasing the number of projectors.

  According to the distortion correction and integration method by divided imaging according to claim 10, since the brightness of the projection area is improved by using several projectors and overlapping the projection areas, the necessary brightness is obtained without using an expensive projector. The brightness of the projection area can be adjusted by changing the number of projectors, and the brightness at a fixed position can be improved by combining the projector overlay and distortion correction.

  According to the distortion correction and integration method using divided imaging according to the eleventh aspect, it can be easily realized by using household equipment such as a television apparatus, a video reproduction apparatus, and a personal computer as the video equipment.

  According to the distortion correction and integration method by divided imaging according to claim 12, the brightness of the image projected at a constant period is measured, and whether the image is a bright image or a dark image is determined for each scene according to the average value. In addition, since the offset is changed, it is possible to reduce the conspicuous brightness in the overlapping area of the projector projection area in the state where the black is projected, and for example, the function of the graphics board without requiring a special blending device. It can be realized using.

  According to the distortion correction and integration method by divided imaging according to claim 13, since the light can be gradually reduced, the optimum light attenuation rate can be adjusted, so that the overlapping portion of the projection area by each projector is more conspicuous. It becomes possible to eliminate.

  According to the mapping function generating apparatus for distortion correction and integration by divided imaging according to claim 14, the screen shape is estimated based on the test points projected on the quadric curved screen, and the spatial position between the viewpoint position and the projector position is estimated. A relative function is obtained, a mapping function Ψvmpi between a plurality of reference virtual cameras vm and a projector pi for photographing the projection surface of the quadric curved screen is obtained, and a viewpoint virtual camera e having position and orientation information at a position where the quadric curved screen is viewed is obtained. Since the mapping function Ψepi is set between each projector and the viewpoint virtual camera e, even if the camera position and the actual viewpoint position are different from each other on the parametric screen, the reference point is determined without measuring the camera position. Mathematically determined mapping function can be obtained with high accuracy and applied to a wide viewing angle screen Can do.

  According to the mapping function generating apparatus for distortion correction and integration by divided imaging according to the fifteenth aspect, in the fourteenth aspect, the mapping function Ψvmpi can be set as an initial value and the error can be minimized to be optimized.

  According to the distortion correction and integration device by divided imaging according to claim 16, the image to be projected on one quadratic curved screen is generated and projected by a plurality of projectors and a computer equipped therewith, so that the resolution can be increased. It is possible to display a detailed portion of the video, and the processing of the computer provided in the plurality of projectors can reduce the rendering load by rendering each different video. Further, since partial videos in different projection areas can be rendered with high resolution, the resolution of the video projected on the screen can be improved without depending on the resolution of the database video as in the past. Thereby, the resolution of the image on the screen can be improved by increasing the number of projectors.

  According to the mapping function generation device for distortion correction and integration by divided imaging according to claim 17, the screen shape is estimated based on the test points projected on the quadric curved screen, and the spatial position between the viewpoint position and the projector position is estimated. A relative function is obtained, a mapping function Ψvmpi between a plurality of reference virtual cameras vm and a projector pi for photographing the projection surface of the quadric curved screen is obtained, and for the right eye and the left eye having position and orientation information at a position where the quadric curved screen is viewed Since the viewpoint virtual camera e is set and the mapping function Ψ repi, lepi between each projector and the right-eye and left-eye viewpoint virtual cameras e is obtained, the camera position differs from the actual viewpoint position with respect to the parametric screen. Even in this case, the mapping function can be obtained with high accuracy by mathematically determining the reference point without measuring the camera position. It can be applied to a wide viewing angle screen.

  According to the mapping function generation device for distortion correction and integration by divided imaging according to claim 18, the fourth process in claim 4 is to optimize the mapping function Ψvmpi with an initial value as a minimum and an error thereof. Can do.

  According to the distortion correction and integration device by divided imaging according to claim 19, the image to be projected on one quadric curved screen is generated and projected by a plurality of projectors and a computer equipped therewith, so that the resolution can be increased. It is possible to display a detailed part of the video, and the processing of the computer equipped with multiple projectors reduces the rendering load by rendering the video of different areas for the right eye and the left eye respectively. be able to. Further, since partial videos in different projection areas can be rendered with high resolution, the resolution of the video projected on the screen can be improved without depending on the resolution of the database video as in the past. Thereby, the resolution of the image on the screen can be improved by increasing the number of projectors.

  According to the distortion correction and integration device by divided imaging according to claim 20, since the brightness of the projection area is improved by using several projectors and overlaying the projection areas, the necessary brightness without using an expensive projector. The brightness of the projection area can be adjusted by changing the number of projectors, and the brightness at a fixed position can be improved by combining the projector overlay and distortion correction.

  According to the distortion correction and integration device by divided imaging according to claim 21, since the video to be projected onto one quadric curved screen is generated and projected by the video equipment, the resolution can be increased, and the detailed portion of the video In the video processing of the video equipment, the rendering load can be reduced by rendering different videos for each of the divided projectors for the right eye and the left eye. In addition, since partial videos in different projection areas can be rendered with high resolution, the resolution of the video projected on the screen can be improved. Thereby, the resolution of the image on the screen can be improved by increasing the number of projectors.

  According to the distortion correction and integration device by divided imaging according to claim 22, the video device generates and projects an image to be projected onto one quadric curved screen, so that the resolution can be increased, and a fine portion of the image In the video processing of the video equipment, the rendering load can be reduced by rendering different videos for each of the divided projectors for the right eye and the left eye. In addition, since partial videos in different projection areas can be rendered with high resolution, the resolution of the video projected on the screen can be improved. Thereby, the resolution of the image on the screen can be improved by increasing the number of projectors.

  According to the distortion correction and integration device by divided imaging according to claim 23, since the brightness of the projection area is improved by using several projectors and overlapping the projection areas, the necessary brightness without using an expensive projector. The brightness of the projection area can be adjusted by changing the number of projectors, and the brightness at a fixed position can be improved by combining the projector overlay and distortion correction.

  According to the distortion correction and integration device by division imaging according to the twenty-fourth aspect, it can be easily realized by using a household device such as a television device, a video reproduction device, or a personal computer as the video device.

  According to the distortion correction and integration device by divided imaging according to claim 25, the brightness of the image projected at a fixed period is measured, and whether the image is a bright image or a dark image is determined for each scene according to the average value. In addition, since the offset is changed, it is possible to reduce the conspicuous brightness in the overlapping area of the projector projection area in the state where the black is projected, and for example, the function of the graphics board without requiring a special blending device. It can be realized using.

  According to the distortion correction and integrated device according to the 26th aspect of the present invention, since the light can be gradually reduced, the optimum light attenuation rate can be adjusted, so that the overlapping portions of the projection areas of the projectors are more conspicuous. It becomes possible to eliminate.

  According to the mapping function generation device for distortion correction and integration by divided imaging according to claim 27, the adjustment range is wide by the flexible arm even if the positional relationship between the screen and the floor on which the projector is installed is not strictly set. Therefore, a display system can be configured in a short time.

  According to the distortion correction and integration device by split imaging according to claim 28, since the adjustment range is wide by the flexible arm without strictly setting the positional relationship between the screen and the floor on which the projector is installed, the display can be performed in a short time. The system can be configured.

([1] Mapping function related to quadratic curved screen, inverse function thereof and image generation)
Hereinafter, an example of a premise technique for explaining the present invention will be described with reference to the drawings. FIGS. 5A, 5B, and 5C are functional block diagrams illustrating an embodiment of a correction function generation device and a video generation device related to a quadratic curved screen. FIG. 6A is a flowchart for explaining an embodiment of a mapping function generating method related to a quadric curved screen, and FIG. 6B shows an embodiment of an image generating method related to a quadric curved screen. FIG. In FIG. 5, 505 is a quadric curved screen, 502j (j = 1, 2, 3,... N) is a projector provided in front of the screen, and 503j is a plurality of computers provided corresponding to each projector 502j. . 5041 and 5042 are digital cameras, 505 is a computer to which the digital camera 504 is connected, 506 is a transformation matrix calculation means in the computer 505, 507 is a quadric surface parameter (Qv) calculation means in the computer 505, and 508 is in the computer 505. 509j is a mapping function and inverse function calculation means in each computer 503j, 510j is a storage device in each computer 503j, 511j is an image generation means provided in each computer 503j, and 512aj and 512bj are provided in each computer 503j. A pixel buffer, and constitutes a video generation means together with the computer 503j and the image generation means 511j. A network 513 connects the computers 503j and 505. Reference numeral 514j denotes offset calculation means of each computer 503j.

(Projector mounting)
The surface of the secondary curved screen 505 forms a secondary curved surface, for example, a cylindrical shape or a spherical surface, and the periphery is formed in a circular shape. FIG. 7A shows a cylindrical shape with a part of the side surface omitted, and the secondary curved screen 505 is fixed on a base 701 that is a floor surface with the side surface vertical. Columns 7031, 7032, 7033, 7034, and 7035 are provided around the screen 505 at predetermined intervals. Flexible arms 7051, 7052, 7053, 7054, and 7055 are attached to the columns 7031, 7032, 7033, 7034, and 7035, respectively. Projectors 5021, 5022, 5023, 5024, and 5025 are attached to flexible arms 7051, 7052, 7053, 7054, and 7055, respectively, and are positioned around the convex surface of the secondary curved screen 505 so as to face the concave surface. . FIG. 7B shows a connected state of the flexible arm 7051 and the projector 5021, for example. When light is projected from the projectors 5021, 5022, 5023, 5024, and 5025 onto the quadric curved screen 505, as viewed from above on the quadric curved screen 505 as shown in FIG. The flexible arms 7051, 7052, 7053, 7054, and 7055 are adjusted so that areas adjacent to each other are continuous so as to have portions that overlap each other, and portions that do not receive light on the way are not generated. At this time, projection areas from the projectors 5021, 5022, 5023, 5024, and 5025 (each area is indicated by a one-dot chain line, two-dot chain line, three-dot chain line, and four-dot chain line) cover the screen. The shape and area need not be the same.
In the embodiment, the quadric curved screen 505 is cylindrical. However, the present invention is not limited to this, and it may be formed of a spherical surface, a hyperboloid, an ellipsoid, or the like. Furthermore, although the case where the number of projectors is five will be described, the present invention can be applied without limitation to the number.

(Principle of distortion correction)
When the viewpoint, the viewpoint, and the projection center match, there is no video distortion. However, distortion generally occurs because the viewpoint and the projection center do not match. In order to correct this distortion, the original image to be projected may be projected in a reverse distortion in advance so that the distortion disappears when viewed from the viewpoint.
FIG. 8 shows a schematic diagram of distortion correction by reverse distortion. FIG. 8 shows that if the distortion f generated by the projection of the projector is projected with reverse distortion at f ⁻¹ at the time of image generation, the distortion is offset by f (f ⁻¹ ). That is, a mapping function f that relates the projector image and the distortion on the screen is obtained. The inverse function f ⁻¹ is obtained from the mapping function. In FIG. 8, it is assumed that the position and direction of the camera installed at the stage of detecting distortion are the same as the viewpoint and the line-of-sight direction.

The mapping function is obtained by the following procedure.
([1-1] Measurement of 3D coordinate positions of multiple points on the screen)
A light spot is projected onto the quadric curved screen 505 from any one of the projectors 502j, for example. The light spot indicates a predetermined coordinate point constituting a predetermined test pattern. By imaging the light spot with a digital camera 5041,5042 constituting the stereo camera obtains a three-dimensional position P _i from the principle of triangulation. i indicates a score. At this time, the stereo camera is premised on that an internal parameter indicating the internal characteristics of the focal length and the number of pixels of each camera lens and an external parameter indicating the external positional relationship are known by a predetermined calibration process. The matrices representing the internal and external parameters of the left camera 5041 are A _c1 and M _c1 . Similarly, let A _c2 and M _c2 be matrices representing internal and external parameters of the right camera 5042. At this time, the relationship between the three-dimensional position P _i on the screen, the position p _i ^c1 of the light spot on the left camera image, and the position p _i ^c2 of the light spot on the right camera image is p _i ^c1 = A _c1 M _c1 P _i
p _i ^c2 = A _c2 M _c2 P _i (1)
It is represented by Here, M _c1 and M _c2 are M _c1 = [I | 0]
M _c2 = [R _c2 | T _c2 ] (2)
It is.
To the projection center of the left camera 5041 as the origin of coordinates, external parameters M _c1 of the left camera 5041 is represented using a unit matrix I. Further, since the projection center of the right camera 5042 is a position that is rotated R _c2 and translated T _c2 from the projection center of the left camera 5041, the external parameter M _c2 of the right camera 5042 is expressed using R _c2 and T _c2. Can do. 9 and 10, the projection center (Center Of Projection) of the left camera 5041 (hereinafter sometimes referred to as COP) is COP _c1 and the COP of the right camera 5042 is COP _c2 . Also, if the straight lines connecting the COP of each camera and the position of the light spot on the image are l _c1 and l _c2 , the three-dimensional position P _i on the screen is estimated at these intersections using l _c1 and l _c2. Can do. If no intersection point is found, a common normal that minimizes the distance between l _c1 and l _c2 is obtained, and the midpoint of the normal is determined as P _i . Also, assuming that the position of the light spot on the projector image coordinates is p _i ^pj and the matrix representing the internal parameters of the projector is A _pj , when p _i ^pj , A _pj , P _i , and the COP of the left camera 5041 are the origin. The rotation matrix R _pj and the translation matrix T _pj of the projector have the following relationship.
p _i ^pj = A _pj [R _pj | T _pj ] P _i (3)
Here, j represents the number of projectors. Also, the position p _i ^pj on the projector image coordinates is known because it is the position of the point of the test pattern. _Apj is known because it is obtained by a predetermined calibration process. Therefore, R _pj and T _pj can be obtained using p _i ^pj , A _pj and P _i . R _pj and T _pj are repeatedly calculated using all the corresponding p _i ^pj and P _i , and the values when the error is minimized are used. These calculations are performed in the computer 505 to which the cameras 5041 and 5042 are connected, and the calculation results are stored in the storage device 508.
Next, using the internal parameters of the projector obtained by the above method, a quadric surface parameter Q, which is information representing the type and shape of the quadric surface (spherical surface, hyperboloid, ellipsoid, etc.) and shape, is obtained.

であり、この関数をＱの要素を並べたベクトル《Ｖ》を用いて変形すれば、

（ここで、例えば《ｈ′》はベクトルｈ′を示す表記である。なお、この明細書、図面においてベクトル表記を、《 》を用いた場合と、矢印を上部に付した場合と白抜き又は肉太の場合とで併用して表す場合がある。）
Ｎ点（Ｎ≧９）用いれば、以下の連立方程式が成立する。
Ｘ《Ｖ》 ＝ 《０》
但し ＸはＮ行１０列のマトリクスである。
この連立方程式を解けば、《Ｖ》、即ちＱが求まる。 ([1-2] Calculation of quadric surface parameter Q)
In order to obtain the mapping function f (), it is necessary to obtain the quadric surface Q. Therefore, first, ^assuming that the projection matrix of the left camera 5041 is P ^c1 and the projection matrix of the projector is P _j ^prj , the projection matrix, the light spot position p _i ^c1 on the left camera image, and one of the test patterns on the projector image coordinates The three-dimensional position P _i is obtained using the corner position p _i ^pj which is a light spot. As a method for obtaining, a method similar to that for obtaining a three-dimensional position with a stereo camera is used. The calculation is performed in the computer 505 to which the cameras 5041 and 5042 are connected. Here, P ^c1 and P _j ^prj are expressed as follows.
P ^c1 = A _c1 [I | 0]
P _j ^prj = A _pj [R _pj | T _pj ] (4)
When the obtained three-dimensional position P _i is on the quadric surface Q,

If this function is transformed using a vector << V >> in which elements of Q are arranged,

(Here, for example, << h '>> is a notation indicating a vector h'. In this specification and drawings, the vector notation is indicated by using <<>>, the case where an arrow is attached at the top, and white or (It may be used in combination with meat thickness.)
If N points (N ≧ 9) are used, the following simultaneous equations are established.
X << V >> = << 0 >>
X is a matrix of N rows and 10 columns.
By solving these simultaneous equations, << V >>, that is, Q can be obtained.

を、異なる視点即ちプロジェクタ５０２ｊ及びカメラ５０４１又は５０４２から見たときの像ｐ_ｉ及びｐ_ｉ′の関係を示すマッピング関数ｆを求める。以下、図１１をもとに説明する。すなわち、カメラの位置及び方向が視点及び視線方向と同じである場合を考える。
今、射影空間Ｐ^２における点を

、射影空間ｐ^３における対応する点を

とすれば、

の対応がある。但し、式中のｔは転置を示す。
また、Ｐ^３の点

をプロジェクタの座標系Ψから見たｐ^２の点を

、カメラ座標系Ψ′から見た点を

とすれば、

ここで、

はΨ′におけるエピポールであり、Ｈはホモグラフィー行列である。
今、

が二次曲面上にあるとの制約を課すと、

より

これを展開して、次のλに関する二次方程式を得る。（λは図１１の

と

との関係を示す倍率

、Ｑ_３３は二次曲面規定マトリクスＱの上から３×３部を抜き出したもので回転を示す。また、《ｑ》は平行移動量、
である。）

ｑ_４４＝１として良いので、これを解き、λを求めると、

式（８）に、このλを代入して整理すれば、

となる。
右辺は

でまとめられ、次式を得る。

と置けば次式となる。

これが、カメラ設置位置と視点が一致する場合における、スクリーン５０５に投影された点

をプロジェクタ座標Ψから見た点

と、カメラ座標系Ψ′から見た点

の対応を示すマッピング関数

である。 ([1-3] Principle of Derivation of Mapping Function f ())
Quadratic surface, for example, one point on the screen

Is a mapping function f indicating the relationship between the images p _i and p _i ′ when viewed from different viewpoints, ie, the projector 502j and the camera 5041 or 5042. Hereinafter, a description will be given with reference to FIG. That is, consider the case where the camera position and direction are the same as the viewpoint and line-of-sight direction.
Now, the point in the projective space P ²

, The corresponding point in the projective space p ³

given that,

There is correspondence. However, t in a formula shows transposition.
In addition, in terms of ^{P 3}

P ² point from the projector coordinate system Ψ

, The point seen from the camera coordinate system Ψ ′

given that,

here,

Is the epipole in ψ ′ and H is the homography matrix.
now,

Imposes the constraint that is on a quadric surface

Than

This is expanded to obtain a quadratic equation for the following λ. (Λ is shown in FIG.

When

Magnification showing the relationship with

, Q ₃₃ is a 3 × 3 portion extracted from the top of the quadric surface defining matrix Q and indicates rotation. Moreover, << q >> is the amount of parallel movement,
It is. )

Since q ₄₄ = 1 is good, if this is solved and λ is obtained,

Substituting this λ into equation (8) and organizing it,

It becomes.
The right side is

And the following formula is obtained.

The following formula is obtained.

This is the point projected on the screen 505 when the camera installation position matches the viewpoint

From the projector coordinate Ψ

And the point seen from the camera coordinate system Ψ ′

Mapping function to show the correspondence of

It is.

([1-4] Mapping function by diagonalized virtual camera method)
([1-4-1] Principle)
Even when the camera installation position and the viewpoint do not match, it will be described that calibration can be performed by virtually moving the camera to the viewpoint using the screen shape information.
In a display system that requires accuracy such as a training apparatus, the viewpoint position is usually defined by a displacement from a reference point. For example, in the case of a dome screen, it is defined by 1 m behind the base line passing through the center of the dome and the displacement from the reference point. FIG. 12 shows the relationship between the reference point 1201, the installation position and orientation of the camera 1202, and the viewpoint and line-of-sight direction referenced by 1203, that is, the position and orientation of the virtual camera. Without thinking of finding the amount of rotation / translation to move directly to the specified viewpoint position / orientation from a camera with an uncertain installation position / orientation, move to the reference point accurately and finally use the displacement from this point. Thinking in two stages when moving to the specified viewpoint position / posture. Based on this idea, when the screen is a quadric surface, the reference point is generally the center of the screen, so moving the camera from any position / posture to the position / posture of the reference point is a quadric surface Corresponds to diagonalization of the prescribed matrix. If this correspondence is used, the amount of movement from the ambiguous position / posture to the reference point can be accurately obtained without actually measuring the posture position of the camera, and it is possible to move to the position / posture of the designated viewpoint. This corresponds to the fact that the camera is virtually placed at the designated viewpoint position and orientation for calibration.

このＳは式（１９）のように分解することができる。

ここで、Ｒ_３３は式（２０）に示すＱ_３３を対角化する回転行列であり、Ｑ_３３及びｑは式（２０）で示されたＱの成分である。また、ΛはＱ_３３を対角化した行列、すなわちＲ_３３ ^−１Ｑ_３３Ｒ_３３である。この回転行列Ｒ_３３によりＱ_３３が対角化されることは数学的に保証されている。

式（１７）の導出を説明する。
座標変換によるＱの対角化Ｓ^ｔＱＳを、Ｑを対角化する相似変換Ｓ^−１ＱＳと捉えて直接的にＳを求められない。なぜならば、Ｓは回転行列Ｒと平行移動行列Ｔの積からなるがＴによりＳが正規直交行列とならないからである。
しかしＳの３行３列の回転成分Ｒ２２は正規直交行列となるため座標変換による対角化問題を相似変換の問題へおきかえることが可能である。すなわち、座標変換によるＱ_３３の対角化Ｒ^−１ _３３Ｑ_３３Ｒ_３３＝Λは相似変換の問題に帰着できる。これにより、未知のＲ_３３がＱ_３３の固有ベクトルを並べたものとして決まり、

となる。展開すると

ここで対角行列Ｄは、

の関係から次の連立する方程式ができる。
ΛＴ＋Ｒ_３３ｑ ＝０
Ｔ^ｔΛ＋ｑ^ｔＲ_３３ ＝０ （２４）
Ｔ^ｔΛ＋Ｔ^ｔＲ^ｔ _３３ｑ＋ｑ^ｔＲ_３３Ｔ ＝０
上の２式より、
Ｔ ＝−Λ^−１Ｒ^ｔ _３３ｑ
Ｔ^ｔ ＝−ｑ^ｔＲ_３３Λ^−１ （２５）
Ｒ_３３及びＴを用いてＳを求めれば、

Ｍは設計時に指定される既知の変換行列であるのでＨが決まり、これによりＱｖが求まり、最終的にマッピング関数ｆ（）が式（１５）から計算できる。発明者はこの方法を「対角化仮想カメラ法」と呼ぶ。 ([1-4-2] Moving to the center by diagonalization)
The quadric surface parameter is information that represents the quadric surface type and shape determined by the camera coordinate system Q, a quadric surface parameter at the viewpoint coordinate system with its origin at the viewpoint and Q _v. The relationship between the two is given by the following equation using H as a transformation (movement, rotation) matrix from the viewpoint coordinate system to the camera coordinate system.
Q _v = H ^t QH (16)
The quadric surface parameter Q is obtained by the method described above ([1-2] Calculation of quadric surface parameter Q).
Considering a two-step conversion from S to the reference coordinate system with the center of the quadric surface as the origin from the camera coordinate system and M as the conversion matrix from the reference coordinate system to the viewpoint coordinate system, H is It is expressed by a formula.
H = MS (17)
S diagonalizes Q as shown in equation (18).

This S can be decomposed as shown in equation (19).

Here, R ₃₃ is a rotation matrix that diagonalizes Q ₃₃ shown in equation (20), and Q ₃₃ and q are components of Q shown in equation (20). Λ is a matrix obtained by diagonalizing Q ₃₃ , that is, R ₃₃ ⁻¹ Q ₃₃ R ₃₃ . It is mathematically guaranteed that Q ₃₃ is diagonalized by this rotation matrix R ₃₃ .

Derivation of Expression (17) will be described.
The diagonalization S ^t QS of Q by coordinate transformation is regarded as a similarity transformation S ⁻¹ QS that diagonalizes Q, and S cannot be obtained directly. This is because S is a product of the rotation matrix R and the translation matrix T, but S does not become an orthonormal matrix due to T.
However, since the rotation component R22 of S in 3 rows and 3 columns is an orthonormal matrix, it is possible to replace the diagonalization problem by coordinate transformation with the problem of similarity transformation. That is, the diagonalization of Q ₃₃ by coordinate transformation R ⁻¹ ₃₃ Q ₃₃ R ₃₃ = Λ can be reduced to the problem of similarity transformation. As a result, unknown R ₃₃ is determined as an array of Q ₃₃ eigenvectors,

It becomes. When expanded

Where the diagonal matrix D is

From this relationship, the following simultaneous equations can be obtained.
ΛT + R ₃₃ q = 0
T ^t Λ + q ^t R ₃₃ = 0 (24)
T ^t Λ + T ^t R ^t ₃₃ q + q ^t R ₃₃ T = 0
From the above two formulas,
T = −Λ ⁻¹ R ^t ₃₃ q
T ^t = −q ^t R ₃₃ Λ ⁻¹ (25)
If S is calculated using R ₃₃ and T,

Since M is a known transformation matrix designated at the time of design, H is determined, whereby Qv is obtained, and finally the mapping function f () can be calculated from the equation (15). The inventor calls this method the “diagonalized virtual camera method”.

  That is, since M is a known transformation matrix designated at the time of design, H is determined by the calculation of the transformation matrix calculation means 506 (step P601 in FIG. 6A), and thereby the quadratic surface parameter (Qv) calculation means. Qv is obtained in the calculation of 507 (step P602 in FIG. 6A). These results are stored in the storage device 508 as a table. In the mapping function and inverse function calculation means j of each computer 503j, the mapping function f () can be calculated from the equation (15) (step P603 in FIG. 6A). The mapping function and inverse function calculation means 509j obtains the inverse function from the mapping function f () (step P604 in FIG. 6A). After step P601 to step P602, step P603, and step P604 are repeated n times (in the embodiment, n = 4) for the projector 502j, the inverse function obtained in step P604 is stored in the storage device 510j of each computer 503j ( Step P605 in FIG. Mapping functions and inverse functions obtained in the series of processes are obtained in advance for a suitable number of virtual viewpoints and stored as a table.

([1-5] Example of the present invention / Example used for a wide-angle quadric curved screen)
Hereinafter, embodiments of the present invention based on the above-mentioned prerequisite technology will be described with reference to the drawings. FIGS. 1A, 1B, and 1C are functional block diagrams illustrating an embodiment of a correction function generation device and a video generation device related to a quadratic curved screen. FIG. 2A is a flowchart for explaining an embodiment of a mapping function generating method relating to a quadric curved screen, and FIG. FIG. In FIG. 1, 101 is a quadratic curved screen, 102i (i = 1, 2, 3,... N) is a projector arranged around the screen, and 103i is a plurality of computers provided corresponding to each projector 102i. is there. 1041i, 1042i (i = 1, 2, 3,... N) are digital cameras constituting a stereo camera capable of capturing images projected by the projector, 105 is a computer connecting the digital cameras 1041i, 1042i, 106 Is a test point three-dimensional restoration means in the computer 105, 107 is a quadric surface coefficient matrix estimation means in the computer 105, 108 is a storage device in the computer 105, 1091i and 1092i are first and second in each computer 103i. Mapping function and inverse function calculating means, 110i is a storage device in each computer 103i, 111i is an image generating means provided in each computer 103i, 112ai and 112bi are pixel buffers provided in each computer 103i. i, constitutes the image generating means with the image generating unit 111i. A network 113 connects the computers 103 i and 105. 114i is an offset calculation means of each computer 103i, 115m (m = 1, 2,... M) is a digital camera, and a reference virtual located on the outer periphery so as to photograph the surface of the secondary curved screen 101 projected by the projector 102i. It is a camera.

誤差がなければ右辺は零である。また、基準仮想カメラｖｍ（１１５ｍ）の画角及び解像度はステレオカメラと同じとし、レンズの歪み及び画像中心のずれがない理想的なカメラとする。これにより基準カメラｃｎ（１０４１ｉ）と基準仮想カメラｖｍ（１１５ｍ）間のスケールを考える必要がない。
ステップ（ｂ） 基準カメラｃｎから基準仮想カメラｖｍ（１１５ｍ）への変換には、ｃｎの座標系における３次元点をもとに決めた基準仮想カメラｖｍ（１１５ｍ）の位置姿勢を表す既知の《Ｒ_{ｃｖｍｃｎ}》，《ｔ_{ｃｖｍｃｎ}》を用いる。最終的に統合された基準仮想カメラｖｍ（１１５ｍ）の座標系における各プロジェクタの３次元点Ｘ_ｖｍｐｉ ^ｋは、ステップ（ａ）で求めた《Ｒ_ｃｎｃｍ》，《ｔ_ｃｎｃｍ》と前記《Ｒ_{ｃｖｍｃｎ}》，《ｔ_{ｃｖｍｃｎ}》を用いて次式から得る。
Ｘ_ｖｍｐｉ ^ｋ＝《Ｒ_ｖｍｃｎ》《Ｒ_ｃｎｃｍ》Ｘ_ｃｍｐｉ ^ｋ
＋《Ｒ_ｖｍｃｎ》《ｔ_ｃｎｃｍ》＋《ｔ_ｖｍｃｎ》 （２８）
これらの計算は、カメラ１０４１ｉ，１０４２ｉが接続されるコンピュータ１０５の二次曲面係数行列推定手段１０７において行われ、これらの計算結果は記憶装置１０８に記憶される。
（３）スクリーンの推定（図２（ａ）のステップＰ１０２）
統合後の各プロジェクタの３次元点Ｘ_ｖｍｐｉ ^ｋは２次曲面係数行列（パラメータマトリックス）Ｑの上にあるので、式（２９）の最適化問題を解いて未知の係数行列Ｑ（４×４）を得る。

これらの計算は、カメラ１０４１ｉ，１０４２ｉが接続されるコンピュータ１０５の二次曲面係数行列推定手段１０７において行われ、これらの計算結果は記憶装置１０８に記憶される。 ([1-5-1] 3D point reconstruction / integration / screen shape estimation)
(1) Restoration of three-dimensional points (Step P101 in FIG. 2A)
FIG. 3 shows restoration of a three-dimensional point by the stereo cameras 1041i and 1042i. A test point x _pi ^k (two-dimensional point: k = 1... K) of a known test pattern is projected from the projector pi (102i (i = 1, 2, 3,... N)). Based on the image obtained by the stereo camera cm (1041i, 1042i) at the position of cm, the three-dimensional position X _cmpi ^k on the screen 101 with the camera cm (1041i, 1042i) as the origin from the principle of stereo vision. From the principle of triangulation. For the projector pj (102j), X _cmpj ^k is restored in the same manner using the camera cm (1041i, 1042i). At this time, the position of the stereo camera cm (1041i, 1042i) is such that the entire projection area of the i-th projector pi (102i) as a target and a part of the projection area of the j-th projector pj (102j) adjacent to it. Any position that can be photographed including overlapping portions can be freely installed. The above procedure is performed for all projectors 102pi (102i) to obtain all three-dimensional points. These calculations are performed in the test point three-dimensional restoration means 106 of the computer 105 to which the cameras 1041 i and 1042 i are connected, and the calculation results are stored in the storage device 108.
(2) Integration of coordinate system (step P102 in FIG. 2A)
Next, in order to estimate the screen shape, the three-dimensional points of the projectors 102i restored in different camera coordinate systems are integrated into one camera coordinate system. Integration is (a) once integrated into the coordinate system of the reference camera cn (for example, the left camera of the stereo camera 1041i). (B) This is installed at a position that faces the screen 101 and the entire screen falls within the angle of view. This is performed in two stages of conversion to the coordinate system of the virtual camera vm (hereinafter referred to as “reference virtual camera vm”). (This is because the position and orientation of the reference virtual camera vm (115m) is determined based on the three-dimensional point of each projector integrated into the coordinate system of the left camera of the stereo camera 10413.)
Step (a) In order to convert the three-dimensional point of each camera coordinate system to the coordinate system of the reference camera cn (stereo camera 1041i), a relative relationship between each camera, rotation _<< R _cncm >>, translation _<< t _cncm >> is required. It is. The unknown _<< R _cncm >> and _<< t _cncm >> are the constraint conditions to be satisfied by the common three-dimensional points X _cmpi ^k and X _cnpi ^k of the same projector restored in the coordinate systems of the cameras cn and cm. It can be obtained by solving an optimization problem.

If there is no error, the right side is zero. The angle of view and resolution of the reference virtual camera vm (115m) are the same as those of the stereo camera, and the camera is an ideal camera free from lens distortion and image center shift. This eliminates the need to consider the scale between the reference camera cn (1041i) and the reference virtual camera vm (115m).
Step (b) For conversion from the reference camera cn to the reference virtual camera vm (115m), a known << representing the position and orientation of the reference virtual camera vm (115m) determined based on the three-dimensional point in the coordinate system of cn. R _cvmcn >>, _<< t _cvmcn >> is used. The 3D points _{X vmpi} ^k of each projector in the coordinate system of the finally integrated reference virtual camera vm (115m), obtained in step _{(a) "R cncm",} "t cncm" and the _{"R cvmcn"} , _<< t _cvmcn >> is obtained from the following equation.
^{_{X vmpi k = "R vmcn"}} "R cncm" X cmpi k
+ _<< R _vmcn >>_<< t _cncm >> + _<< t _vmcn >> (28)
These calculations are performed by the quadric surface coefficient matrix estimation means 107 of the computer 105 to which the cameras 1041 i and 1042 i are connected, and the calculation results are stored in the storage device 108.
(3) Screen estimation (step P102 in FIG. 2A)
Since the three-dimensional point X _vmp ^{i k} of each projector after the integration is on the quadratic surface coefficient matrix (parameter matrix) Q, the optimization problem of Equation (29) is solved and the unknown coefficient matrix Q (4 × 4) Get.

These calculations are performed by the quadric surface coefficient matrix estimation means 107 of the computer 105 to which the cameras 1041 i and 1042 i are connected, and the calculation results are stored in the storage device 108.

この射影行列《Ｐ_ｐｉ》は《Ｐ_ｐｉ》＝《Ｋ_ｐｉ》［《Ｒ_ｐｉ》｜《ｔ_ｐｉ》］のように内部パラメータ《Ｋ_ｐｉ》と位置姿勢を表す行列《Ｒ_ｐｉ》，《ｔ_ｐｉ》に分離でき、プロジェクタｐｉ（１０２ｉ）の内部パラメータは既知であるのでスクリーン１０１に対するプロジェクタｐｉ（１０２ｉ）の回転《Ｒ_ｐｉ》、並進《ｔ_ｐｉ》が得られる。これらの計算は、各プロジェクタ１０２ｉが接続されるコンピュータ１０３の第１のマッピング関数及び逆関数計算手段１０９１ｉにおいて行われ、これらの計算結果は記憶装置１１０ｉに記憶される。
（２）マッピング関数Ψｖｍｐｉの計算（図２（ａ）のステップＰ１０４）
投影による歪みを補正するには、予め投影する映像に歪みを与えておけば、投影時の歪みで逆歪みが相殺されて歪みのない映像となる。これが基本的な考えであり、このことは、プロジェクタｐｉの画素位置ｘｐｉの投影像がスクリーンを介してカメラ（即ち視点）の画素位置ｘｃに対応するマッピング関数Ψｐｉｃを見つけることに帰着する。ここで添え字ｐｉ，ｃはプロジェクタｉ及び実カメラの座標系を示し、変換方法はｃからｐｉである。また、ｖを視点、ｒを基準座標系とする。
このΨｐｉｃを得る方法はコンピュータビジョン技術の典型的な応用としてよく知られている。方法は各種あるが、発明者はカメラとプロジェクタの内部パラメータを事前に計測し既知であるとして、プロジェクタからテストパターンをスクリーンに投影し、これをカメラで撮影して対応関係を得ることは、上述した通りである。
平面の場合、このΨｐｉｃはホモグラフィマトリクスＨを用いて以下の関係にある。
ｘ_ｐｉ＝Ｈｘ_ｃ （３１−２）
また、２次曲面を対象とした場合は次式に拡張されることが知られている。第一項は平面と同様の式であり、第二項が曲面による補正項である。
ｘ_ｐｉ
＝Ｈｘ_ｃ−（ｑ^Ｔｘ_ｃ±√（（ｑ^Ｔｘ_ｃ）^２−ｘ^Ｔ _ｃＱ_３３ｘ_ｃ））ｅ_ｃ （３１−３）
ここで、Ｔは転置、ｅ_ｃはエピポール、Ｑ_３３，ｑは次式で決まる４行４列の２次曲面係数行列Ｑの各要素である。

式（３１−３）のΨｐｉｃで示された対応から、予め逆に歪ませる量が分かるので、映像を出すプロジェクタ側で逆歪みをかけておけば投影時の歪みで逆歪みが相殺され、歪みのない映像がカメラ位置（視点）から見える。
基準仮想カメラｖｍ（１１５ｍ）の位置姿勢、プロジェクタｐｉ（１０２ｉ）の位置姿勢《Ｒ_ｐｉ》，《ｔ_ｐｉ》、スクリーン形状を表す行列Ｑを用いて、補正に必要な各プロジェクタｐｉ（１０２ｉ）と基準仮想カメラｖｍ（１１５ｍ）間のマッピング関数Ψｖｍｐｉ，（ｍ＝１…Ｍ，ｉ＝１…Ｎ）が式（３２）（式（３１−３）を変形したもの）のように求まる。
ｘ^ｋ _ｖｍ
＝Ａ_ｖｍｐｉｘ^ｋ _ｐｉ±（√（ｘ^ｋ _ｐｉ ^ＴＥ_ｐｉｘ^ｋ _ｐｉ））ｅ_ｖｍ （３２）
ここで、Ａ_ｖｍｐｉ＝Ｈ_ｖｍｐｉ−ｅ_ｖｍ《ｑ^Ｔ _ｐｉ》，Ｅ_ｐｉ＝《ｑ_ｐｉ》《ｑ^Ｔ _ｐｉ》−Ｑ_３３ｐｉであり、ｐ_ｉ以外のプロジェクタｐｉ（１０２ｉ）のマッピング関数についても同様にして求めることができる。これらの計算は、各プロジェクタ１０２ｉが接続されるコンピュータ１０３の第１のマッピング関数及び逆関数計算手段１０９１ｉにおいて行われ、これらの計算結果は記憶装置１１０ｉに記憶される。
（３）マッピング関数Ψｖｍｐｉの最適化（図２（ａ）のステップＰ１０４−２）
以上の計算で得られたマッピング関数Ψｖｍｐｉは様々な誤差を含んでおり、そのままでは使えない。そこで、前出のマッピング関数Ψｖｍｐｉ、具体的にはＡ_ｖｍｐｉ，Ｅ_ｐｉ，ｅ_ｖｍを初期化して、以下の再投影誤差Ｃを計算しマッピング関数Ψｖｍｐｉの最適化を行う。
Ｃ
＝Σ_ｋ，ｉ｜ｘ^ｋ _ｖｍ−（Ａ_ｖｍｐｉｘ^ｋ _ｐｉ±（√（ｘ^ｋ _ｐｉ ^ＴＥ_ｐｉｘ^ｋ _ｐｉ））ｅ_ｖｍ）｜
（３３）
即ち、既知のｘ^ｋ _ｖｍとｘ^ｋ _ｐｉを用いて式（３３）を最小化するＡ_ｖｍｐｉ，Ｅ_ｐｉ，ｅ_ｖｍを求める。これらの計算は、各プロジェクタ１０２ｉが接続されるコンピュータ１０３の第１のマッピング関数及び逆関数計算手段１０９１ｉにおいて行われ、これらの計算結果は記憶装置１１０ｉに記憶される。 ([1-5-2] Mapping function derivation)
(1) Detecting the position and orientation of the projector (Step P103 in FIG. 2A)
If the projection matrix of the projector pi is _<< P _pi >>, the two-dimensional point x _pi ^k of the test pattern corresponding to the three-dimensional point X _vmpi ^k in the coordinate system of the reference virtual camera vm (115m) Have a relationship.
x _pi ^k = _<< P _pi >> X _vmp ^{i k} (30)
Therefore, the unknown projection matrix _<< P _pi >> can be obtained by solving the optimization problem of Expression (31) using the two-dimensional points x _pi ^k (both known) of the test pattern corresponding to X _vmp ^{i k} .

The projection matrix _{"P pi"} is _{"P pi" = "K pi} " [ "R pi" | "t pi"] matrix representing internal parameters _{"K pi"} and the position and orientation as _{"R pi", "t pi} >> and the internal parameters of the projector pi (102i) are known, so that the rotation << R _pi >> and the translation << t _pi >> of the projector pi (102i) with respect to the screen 101 are obtained. These calculations are performed in the first mapping function and inverse function calculation means 1091i of the computer 103 to which each projector 102i is connected, and these calculation results are stored in the storage device 110i.
(2) Calculation of mapping function Ψvmpi (step P104 in FIG. 2A)
In order to correct distortion due to projection, if distortion is given to an image to be projected in advance, reverse distortion is canceled out by distortion at the time of projection, resulting in an image without distortion. This is a basic idea, and this results in finding a mapping function Ψ pic where the projection image of the pixel position xpi of the projector pi corresponds to the pixel position xc of the camera (ie, the viewpoint) via the screen. Here, the subscripts pi and c indicate the coordinate system of the projector i and the real camera, and the conversion method is from c to pi. Further, v is a viewpoint and r is a reference coordinate system.
This method of obtaining Ψpic is well known as a typical application of computer vision technology. Although there are various methods, the inventor measures the internal parameters of the camera and the projector in advance and assumes that the parameters are known in advance. That's right.
In the case of a plane, this Ψpic has the following relationship using the homography matrix H.
x _pi = Hx _c (31-2)
Further, it is known that when a quadric surface is targeted, it is expanded to the following equation. The first term is an expression similar to a plane, and the second term is a correction term by a curved surface.
x _{pi
^{= Hx c - (q T x}} c ± √ ((q T x c) 2 -x T c Q 33 x c)) e c (31-3)
Here, T is the transpose, e _c is epipoles, Q _{33, q} is each element of the quadratic surface coefficient matrix Q of four rows and four columns determined by the following equation.

From the correspondence indicated by Ψpic in equation (31-3), the amount of reverse distortion is known in advance, so if reverse distortion is applied on the projector side that displays the image, the reverse distortion is canceled out by the distortion at the time of projection, and the distortion An image with no image can be seen from the camera position (viewpoint).
Using the position and orientation of the reference virtual camera vm (115m), the position and orientation of the projector pi (102i) << R _pi >>, << t _pi >>, and the matrix Q representing the screen shape, each projector pi (102i) required for correction A mapping function ψvmpi, (m = 1... M, i = 1... N) between the reference virtual cameras vm (115m) is obtained as in Expression (32) (which is a modification of Expression (31-3)).
x ^k _{vm
^{= A vmpi x k pi ± (}} √ (x k pi T E pi x k pi)) e vm (32)
^{_{Here, A vmpi = H vmpi -e vm}} "q T pi", E pi = "q pi"" a _q T _{pi" -Q} 33pi, also applies mapping function other than _{p i} projector pi (102i) Can be obtained. These calculations are performed in the first mapping function and inverse function calculation means 1091i of the computer 103 to which each projector 102i is connected, and these calculation results are stored in the storage device 110i.
(3) Optimization of mapping function Ψvmpi (step P104-2 in FIG. 2A)
The mapping function Ψvmpi obtained by the above calculation includes various errors and cannot be used as it is. Therefore, the above mapping function ψvmpi, specifically, A _vmpi , E _pi , and e _vm are initialized, the following reprojection error C is calculated, and the mapping function ψvmpi is optimized.
C
= Σ _{k, i} | x ^k _vm − (A _vmpix x ^k _pi ± (√ (x ^k _pi ^T e _pi x ^k _pi )) e _vm ) |
(33)
That is, A _vmpi , E _pi , and e _vm that minimize Equation (33) are obtained using known x ^k _vm and x ^k _pi . These calculations are performed in the first mapping function and inverse function calculation means 1091i of the computer 103 to which each projector 102i is connected, and these calculation results are stored in the storage device 110i.

([1-5-3] Sign Determination of Mapping Function Ψvmpi (Step P104 in FIG. 2A))
As shown in FIG. 4A, the quadric surface Q representing the screen shape is bounded by a plane plane where the optical axis of one reference virtual camera v1 (1151) is a ray and the center coincides with the center of Q. It has two surfaces, the front curved surface and the back curved surface. The sign in equation (32) is determined depending on which of the two surfaces is corrected. In this case, the front surface is-and the back surface is +. When looking at the projectors p1 (1021) and p5 (1025), it is difficult to determine where the projection area straddles the boundary plane 、 and whether the positive / negative determination of Expression (32) occurs.
In order to prevent this inconvenience, as shown in FIG. 4B, the problem of sign determination is avoided by installing a plurality of reference virtual cameras vm (115m). In the example of FIG. 4B, reference virtual cameras v1 (1151), v2 (1152), and v3 (1153) are arranged around the screen 101, and the respective reference virtual cameras v1 (1151), v2 (1152), and v3 are arranged. (1153) shares and expects the whole screen. As a result, when viewed from the newly installed reference virtual camera vm (115m), the corresponding projector can always project an image on the back surface of the quadric curved screen 101, and an ambiguous code state can be avoided. In the example of FIG. 4B, a mapping function with the reference virtual cameras v2 (1152) and v3 (1153) is required for the projectors p1 (1021) and p2 (1022). Further, the optimization processing is also performed for the projectors p2 to p4 (1022 to 1024) of the reference virtual camera v1 (1151), the projector p1 (1021) of the reference virtual camera v2 (1152), and the projector p5 (1025) of the reference virtual camera v3. It is necessary to use the two-dimensional point (1153). These calculations are performed together with the computer 105 connecting the reference virtual camera 115m in the first mapping function and inverse function calculation means 1091i of the computer 103 to which each projector 102i is connected, and the calculation results are stored in the storage device 110i. Is done.

(Calculation of [1-5-4] Mapping Function Ψepi (Step P105 in FIG. 2A))
A mapping function Ψepi between each projector pi (i = 1... I) and the actual viewpoint e is obtained in order to look continuous without distortion when viewed from the actual viewpoint position. Since the mapping function Ψvmpi (m = 1... M) between each projector pi (102i) and the reference virtual camera vm (115m) has already been obtained, the mapping function Ψevm between the reference virtual camera vm (115m) and the viewpoint is obtained. For example, Ψepi can be obtained indirectly from Expression (34).
Ψepi = Ψevm Ψvmpi (m = 1... M, i = 1... I) (34)
Since the position and orientation of the viewpoint e viewed from the reference virtual camera vm (115m) is obtained geometrically using the virtual camera method, the virtual camera (that is, the viewpoint virtual camera e) e having the position and orientation information and the reference virtual camera vm A mapping function Ψevm is also obtained. In the example of FIG. 4, since four virtual cameras (three reference virtual cameras v1 to v3 (1151 to 1153) and viewpoint virtual camera v0 (1154) ≡e) are used, mapping is performed as shown in FIG. 4B. As for the function, a mapping function Ψevm (m = 1... 3) between each reference virtual camera vm (115m) and the viewpoint virtual camera v0 (1154) is required, and the virtual camera method is used for each reference virtual camera vm (115m). Is applied to obtain Ψev1, Ψev2, and Ψev3. Since the reference virtual camera vm (115m) and the viewpoint virtual camera v0 (1154) are ideal cameras, and the position and orientation of the viewpoint virtual camera v0 (1154) can be obtained geometrically, the mapping functions Ψev1, Ψev2, and Ψev3 There is no need for optimization. The mapping function between each projector and the viewpoint virtual camera v0 (1154) in the case of FIG. 4 is expressed as follows.
Ψepi = Ψev1Ψv1pi (i = 2... 4)
Ψep1 = Ψev2Ψv2p1; Ψep5 = Ψev3Ψv3p5 (35)
These calculations are performed together with the computer 105 connecting the reference virtual camera 115m in the second mapping function and inverse function calculation means 1092i of the computer 103 to which each projector 102i is connected, and the calculation results are stored in the storage device 110i. Is done.
The second mapping function and inverse function calculation means 1092i obtains the inverse function from the mapping function f () (step P106 in FIG. 2A). The inverse function obtained in step P106 is stored in the storage device 110i of each computer 103i. Mapping functions and inverse functions obtained in the series of processes are obtained in advance for a suitable number of virtual viewpoints and stored as a table.

([2] Load distribution by parallel processing)
In the case of FIG. 6, the circumference of the screen is an ellipse unless it is looked down from a directly facing position. Since the diagonal viewpoint described in “[1-4-2] Moving to the center by diagonalization” described above makes it possible to face the virtual viewpoint position directly on the screen, the rectangle outlined in the circle is calculated. The field of view can be obtained directly and overlapped.

([2-1] Division of Viewing Angle Using Rectangle FIG. 13A shows the relationship between the rectangle circumscribing the circle representing the edge of the screen and the division and overlap of the field of view using the rectangle. (B) shows the viewing angle division method, which will be described below assuming that each division is included in the projectable region in the figure.

([2-2] Transformation matrix for combining)
In the following, the calculation of the transformation matrix is shown by taking 4 fields of view (4 parallel) as an example. However, the discussion here does not lose generality other than the four divisions. The mapping function f ^j () from the projector coordinates of the projector 502 j to the virtual camera coordinates (the region divided into four as shown in FIG. 13A) is obtained as follows.
(1) _Obtain the radius r of the screen and use this to calculate W _rct and H _rct shown in FIG.
(2) A rectangular matrix circumscribing the circle is divided into four so as to have overlapping portions H _over and W _over , and a transformation matrix H ^j _rct from the circumscribed rectangular display area to the corresponding area divided into four is calculated.
(3) The mapping function f ^j () is calculated in the mapping function and inverse function calculation means 509j of the computer 103i provided corresponding to the projector 502j using H ^j _rct .
Hereinafter, it demonstrates in order.

を用いた

はカメラ座標でのスクリーンの縁を表す楕円を示す式である。従って、カメラ座標系における二次曲面行列Ｑを、視点座標系に変換したＱ_ｖを用いて計算し直したＥに対応するＥ_ｖを用いた以下の式

は、スクリーンに正対する視点座標系でのスクリーンの縁に対応する円を示す式となる。
また、Ｅｖで規定される円と任意の点

の距離ｄは、

の関係がある。
ここで、

として展開すれば、
ｅ_１１ｘ^２ _ｉ＋２ｅ_１２ｘ_ｉｙ_ｉ
＋２ｅ_１３ｘ_ｉ＋２ｅ_２３ｙ_ｉ
＋ｅ_２２ｙ^２ _ｉ＋ｅ_３３
＝ｄ^２ （３７）
この距離ｄが最大となる点が、円の中心であり、また、このときの距離が半径となる。ｄを最大とする点

は次の関係を満足する。
∂ｄ^２／∂ｘ_ｉ＝２ｅ_１１ｘ_ｉ＋２ｅ_１２ｙ_ｉ＋２ｅ_１３＝０
∂ｄ^２／∂ｙ_ｉ＝２ｅ_１２ｘ_ｉ＋２ｅ_２２ｙ_ｉ＋２ｅ_２３＝０ （３８）
上式で示した連立方程式を解けば、
ｘ_ｉ＝（ｅ_１３ｅ_２２−ｅ_１２ｅ_２３）／（ｅ^２ _１２−ｅ_１１ｅ_２２）
ｙ_ｉ＝（ｅ_１１ｅ_２３−ｅ_１２ｅ_１３）／（ｅ^２ _１２−ｅ_１１ｅ_２２） （３９）
この座標点は、円の中心であるので、（ｘ_ｃ，ｙ_ｃ）と表記する。この中心点（ｘ_ｃ，ｙ_ｃ）を用いれば、半径ｒは、
ｒ＝√（ｅ_１１ｘ^２ _ｃ＋２ｅ_１２ｘ_ｃｙ_ｃ＋２ｅ_１３ｘ_ｃ
＋２ｅ_２３ｙ_ｃ＋ｅ_２２ｙ^２ _ｃ＋ｅ_３３） （４０）
となる。 ([2-2-1] Circle radius)
In the square root of the above equation (15)

Used

Is an equation representing an ellipse representing the edge of the screen in camera coordinates. Therefore, the following equation using E _v corresponding to E recalculated using Q _v converted from the quadratic surface matrix Q in the camera coordinate system into the viewpoint coordinate system:

Is an expression showing a circle corresponding to the edge of the screen in the viewpoint coordinate system directly facing the screen.
Also, the circle defined by Ev and any point

The distance d of

There is a relationship.
here,

Expand as
e ₁₁ x ² _i + 2e ₁₂ x _i y _i
+ 2e ₁₃ x _i + 2e ₂₃ y _i
+ E ₂₂ y ² _i + e ₃₃
= D ² (37)
The point where the distance d is maximum is the center of the circle, and the distance at this time is the radius. Point that maximizes d

Satisfies the following relationship:
∂d ² / ∂x _i = 2e ₁₁ x _i + 2e ₁₂ y _i + 2e ₁₃ = 0
∂d ² / ∂y _i = 2e ₁₂ x _i + 2e ₂₂ y _i + 2e ₂₃ = 0 (38)
Solving the simultaneous equations shown above,
_{x i = (e 13 e 22} -e 12 e 23) / (e 2 12 -e 11 e 22)
y _i = (e ₁₁ e ₂₃ -e ₁₂ e ₁₃ ) / (e ² ₁₂ -e ₁₁ e ₂₂ ) (39)
Since this coordinate point is the center of the circle, it is expressed as (x _c , y _c ). Using this center point (x _c , y _c ), the radius r is
r = √ (e ₁₁ x ² _c + 2e ₁₂ x _c y _c + 2e ₁₃ x _{c
^{+ 2e 23 y c + e 22}} y 2 c + e 33) (40)
It becomes.

各視野は、各プロジェクタ５０２ｊ毎の映像が投影される領域を決める。 ([2-2-2] Field division)
As shown in FIG. 13A, if the overlapping width of the projected images in the horizontal direction is W _over and the overlapping width in the vertical direction is H _over , the divided visual field sizes Wrct and Hrct are as follows.
Wrct = r + (Wovr / 2)
Hrct = r + (Hovr / 2) (41)
Using this, the transformation matrix from the coordinate system having the origin as O shown in FIG. 13A to the coordinate system of each visual field having the origin as O ^j is as follows.

Each field of view determines an area in which an image for each projector 502j is projected.

([2-2-3] Mapping function f ())
The following expression obtained by operating Hjrct to Expression (15) is a new mapping function f ^j () related to the projector 502j when the viewing angle is divided as described above.

([2-3] Brightness adjustment)
Since the overlapping portion of the projection area has high luminance, it is necessary to correct the luminance of the projected image. This luminance correction is performed by preliminarily holding a weighting coefficient at a pixel position of each projector overlapping on the screen as a table, and multiplying the pixel value and the corresponding weighting coefficient at the time of distortion correction. FIG. 14 shows a method for determining the weight. The program in FIG. 14 is written in a pseudo language.
Here, (p, q) is the pixel position in the camera coordinate system, i, j is the projector number of (1... 4), H is the number of pixels in the horizontal direction of the projector, W is the number of pixels in the vertical direction, and α ⁱ ( p ⁱ , q ⁱ ) represent weighting factors at the pixel position (p ⁱ , q ⁱ ) of the projector i, respectively. The overlapping part of the projector i and the projector j is determined, and the weight of the part is given in inverse proportion to the overlapping width.

([3] Reduction of black offset due to projector leakage light)
([3-1] Necessity of reducing black offset)
When a high-resolution video is generated using a plurality of projectors, an overlapping portion is generated in the projection area of each projector. The overlapping part is added with the brightness of each projector, and becomes brighter than the non-overlapping part. There is no problem as long as the projector can display complete black, but even if the projected image is black, the area projected by the leakage light of the projector is not completely black (see FIG. 15A). In addition, the overlapping area becomes brighter because the leakage light of the projectors overlaps. In other words, a black offset occurs due to light leaked from the projector, and the overlapped portion becomes conspicuous even when black is emitted.

([3-2] Reduction of black offset)
The black offset portion due to the leakage light of the projector depends on the projector and cannot be adjusted. Only the color of the projected image can be adjusted. If the color value becomes 1.0 or more with simple color raising (the color value is normalized to 0.0 to 1.0), the color is corrected to 1.0. It will be different from the color of the original video. Therefore, the contrast of the original image is lowered, and the color is not changed by raising the image while keeping the color (see FIG. 15B). As an implementation method, by using the fragment shader function of the graphics board using the Cg (C for Graphics by nVidia) language and correcting the color of the video to be captured, the brightness of the overlapping portion is conspicuous (FIG. 15 (c ), (See FIG. 15D).
The color correction formula used in Cg is as shown in the following formula (44). By using this equation, it is possible to use the α blend function for adjusting the black level and blending overlapping portions (see FIG. 15E). In addition, it is possible to reduce luminance deviation due to black offset. An example of the overlapping portion will be described with reference to FIG.
C_new
= K * C_org + C_offset (44)
However,
C_new: Corrected color (The corrected color)
K: the coefficient for reducing the contrast (the coefficient for reducing the contrast)
C_org: The original color of image (The original color of image)
C_offset: The brightness of color (The offset of color)

Ｃ^ｎ _{ｏｆｆｓｅｔ}＝（Ｉ_ｍａｘ−Ｉ^ｎ _ａｖｅ）／２５６ （４７）
（図１６のステップＰ１６０３）
なお、式（４５）の分母のＷｉｍｇ，Ｈｉｍｇはそれぞれカメラでスクリーン５０５又は１０１を撮像したときのスクリーン５０５又は１０１の周囲にある矩形の幅、高さである（図１３（ａ）参照）。式（４７）の分母の２５６は例えばカメラが持つ輝度値を８ｂｉｔで持つとした場合の２^８＝２５６である。
Ｋ＝１．０−Ｃ^ｎ _{ｏｆｆｓｅｔ} （４８）
これらにより式（４４）を得て、補正した色を得る。このとき、一定周期で投影する映像の輝度を計測し、その平均値に従って、ブラックオフセットにかさ上げするＣ＿ｏｆｆｓｅｔを前記一定の周期で計算して変更することで、シーンごとにＣ＿ｎｅｗを作成し、黒レベルの調整を行う。従って、明画像か暗画像かを判定し、暗画像の場合にオフセットを調整するため、ブラックオフセットによる輝度のずれを軽減することができる。また、重複する部分についてアルファブレンディングにより重複する部分の輝度を下げて他の領域と目立たなくしている。（図１６のステップＰ１６０４）。 ([3-3] K and C_offset calculation method)
Determined from the brightness of the camera image.
1. A state in which nothing is projected from the projectors 5021 to 5024 or 102i is captured by the camera 5041 or 1042j. (Used as Background Image.) Let the image be Img_bkgrnd (step P1601 in FIG. 16).
2. A state in which black is emitted from each projector 5021 to 5024 or 102i is photographed by the camera 5041 or 102i (step P1602 in FIG. 16). Let the image be Img_blkofst_n. n is a projector number. Assuming that the luminance obtained from the image is I, the offset calculator 514j or 114j performs the following calculation.

C ⁿ _offset = (I _max −I ⁿ _ave ) / 256 (47)
(Step P1603 in FIG. 16)
Note that Wimg and Himg in the denominator of Expression (45) are the width and height of a rectangle around the screen 505 or 101 when the screen 505 or 101 is imaged by the camera, respectively (see FIG. 13A). The denominator 256 in the equation (47) is, for example, 2 ⁸ = 256 when the luminance value of the camera is 8 bits.
K = 1.0−C ⁿ _offset (48)
Thus, the equation (44) is obtained, and a corrected color is obtained. At this time, the brightness of the projected image is measured at a fixed period, and C_offset to be raised to the black offset is calculated and changed at the fixed period according to the average value, thereby creating C_new for each scene. Adjust the level. Therefore, since it is determined whether the image is a bright image or a dark image, and the offset is adjusted in the case of a dark image, a luminance shift due to a black offset can be reduced. Further, the luminance of the overlapping portion is lowered by alpha blending to make it inconspicuous from other regions. (Step P1604 in FIG. 16).

とする）を利用し、プロジェクタ画像座標のある位置がテクスチャ座標のどの位置に対応するかを求めることができ、そのテクスチャ座標の位置の色をサンプリングして色情報として保持する（図１８参照）。
サンプリング結果を元に歪み補正を施した映像を作成し、プロジェクタから投影する。投影された映像はスクリーン上で正しく映像として表示される。
図５の各プロジェクタ５０３ｊ毎に備えられたコンピュータ５０３ｊの画像生成手段５１１ｊにより生成された画像を、ピクセルバッファ５１２ｊａにレンダリングし、この画像を当該プロジェクタ５０３ｊについて求めたマッピングの逆関数ｆ^−１（）により、歪み補正した映像をピクセルバッファ５１２ｊｂに生成する（図６（ｂ）のステップＳ６０１）。この歪み補正した映像を各プロジェクタ５０３ｊから二次曲面スクリーン５０５に投影する（図６（ｂ）のステップＳ６０２）。
図１による場合、各プロジェクタ１０２ｉ毎に備えられたコンピュータ１０３ｉの画像生成手段１１１ｉにより生成された画像を、ピクセルバッファ１１２ａｉにレンダリングし、この画像を当該プロジェクタ１０２ｉについて求めたマッピングの逆関数ｆ^−１（）により、歪み補正した映像をピクセルバッファ１１２ｉｂに生成する（図２（ｂ）のステップＳ１０１）。この歪み補正した映像を各プロジェクタ１０２ｉから二次曲面スクリーン１０に投影する（図２（ｂ）のステップＳ１０２）。図４（ｃ）に表示システムを示し、各プロジェクタ１０２ｉ（ｉ＝１〜５）は図示しないフレキシブルアームにより支柱４０１〜４０５を介して床面４０６上に設置されている。 ([4] Video generation)
Conventionally, when video rendering is performed, the video is generally written in a frame buffer (memory area) of the graphics board, and the buffer is displayed via a display or a projector.
In a projection cluster (the screen is covered by the entire projection area of a plurality of projectors), the video data stored in the buffer is sampled (video reference) when correcting the video distortion. The distortion is corrected by recreating the video based on the referenced data. The larger the buffer, the higher the resolution of the video. When this buffer is a frame buffer, the frame buffer depends on the resolution with the display or projector, and therefore the buffer cannot be made larger than the resolution of the display or projector. Therefore, it is difficult to increase the resolution beyond the function of the display or projector.
In the present invention, a pixel buffer is used instead of a frame buffer when storing video data. By storing directly in the pixel buffer and performing off-screen rendering, the resolution of the display or projector is not limited. Although the size of the pixel buffer is limited to a power of 2, it can be arbitrarily set. Actually, the size of the pixel buffer is limited because it depends on the function of the graphics card, but a buffer having a size larger than the frame buffer can be secured.
The size of the pixel buffer is arbitrarily determined, and video having a display resolution or higher is stored in the buffer. Here, the buffer size is 2048 × 2048.
When using a pixel buffer, it is not necessary to use a frame buffer because it can be rendered directly into the buffer. Further, by using an OpenGL extension instruction, data stored in the pixel buffer can be directly handled as a texture.
The pixel buffers 512aj and 512bj in FIG. 5 or the pixel buffers 112aj and 112bj in FIG. 1 perform data sampling using a quadratic transfer (QT). QT represents conversion from camera image coordinates to projector image coordinates, and inverse conversion (QT_inv) represents conversion from projector image coordinates to camera image coordinates. In the projection cluster, data sampling is performed using QT_inv.
As a sampling method, the coordinates from (0, 0) to (1024, 768) of the projector image coordinates are sequentially scanned, and the position corresponding to the position in the camera image coordinates is obtained. If the camera is considered as a viewpoint on OpenGL, an image stored in the texture memory can be regarded as an image viewed from the camera. This relationship (here transformation matrix

The position of the projector image coordinates corresponds to which position of the texture coordinates, and the color at the position of the texture coordinates is sampled and held as color information (see FIG. 18). .
Based on the sampling result, an image with distortion correction is created and projected from the projector. The projected image is correctly displayed as an image on the screen.
The image generated by the image generation means 511j of the computer 503j provided for each projector 503j in FIG. 5 is rendered in the pixel buffer 512ja, and the inverse function f ⁻¹ () of the mapping obtained for the projector 503j. Thus, a distortion-corrected video is generated in the pixel buffer 512jb (step S601 in FIG. 6B). This distortion-corrected image is projected from each projector 503j onto the secondary curved screen 505 (step S602 in FIG. 6B).
In the case of FIG. 1, the image generated by the image generation means 111i of the computer 103i provided for each projector 102i is rendered in the pixel buffer 112ai, and the inverse function f ^{−1 of the} mapping obtained for this projector 102i is obtained. By (), the distortion corrected video is generated in the pixel buffer 112ib (step S101 in FIG. 2B). This distortion-corrected image is projected from each projector 102i onto the quadric curved screen 10 (step S102 in FIG. 2B). FIG. 4C shows a display system, and each projector 102 i (i = 1 to 5) is installed on the floor surface 406 via a column 401 to 405 by a flexible arm (not shown).

In the above ([2-3] brightness adjustment), the width of the overlapping portion of the projection area is set to be relatively smaller than the non-overlapping portion as shown in FIG. As described above, for example, the overlapping portions of the projection areas 1703 and 1704 by the two projectors 1701 and 1702 can be enlarged to generate a composite image, and the brightness of the projection area can be improved by superposition. At this time, the screen may be a quadratic curved surface as described above, but may be a flat screen 1705.
Also in the case of using these two projectors, the two computers 5031 and 5032 in FIG. 5 generate images for each projector 5021 and 5022 and the generated images by the inverse function of the mapping for each projector 5021 and 5022. Is generated in the pixel buffer 512a (step S1701 in FIG. 17B), and the images corrected for distortion in the pixel buffer 512b are projected from the projectors 5021 and 5022 onto the quadric curved screen 505. Thus, the brightness of the overlapping portion is increased (step S1702 in FIG. 17B).
Also in the case of using these two projectors, for example, the two computers 1031 and 1032 in FIG. 1 generate images for each of the projectors 1021 and 1022, and the generated by the inverse function of the mapping for each of the projectors 1021 and 1022 A video whose distortion is corrected is generated in the pixel buffers 112a1 and 112a2 (step S1701 in FIG. 17B), and the video whose distortion is corrected in the pixel buffers 112b1 and 112b2 is projected from the projectors 1021 and 1022 onto the quadric curved screen 101. Then, the brightness of the common overlapping portion is increased (step S1702 in FIG. 17B).

In the above ([3] Reduction of black offset due to leakage light from projector), there is no problem if the projector can display perfect black, but even if the projected image is black, the area projected by the leakage light of the projector is I explained that it was not completely black. This is because when a dark scene image at night is projected, the human eye can be sensitive to even a slight difference in luminance, and thus tends to be more noticeable.
In this case, as a possible measure, by correcting the brightness of the original image to be projected softly, the brightness of the overlapping area caused by black offset can be reduced and the conspicuousness of the overlapping part can be reduced. It is difficult to prevent the overlap part from being completely recognized in the scene.
For this reason, it is possible to display an image in which the overlapping portion is not felt more by combining the polarizing plates in combination with the correction of the luminance of the original image and adjusting the dimming rate for the overlapping portion in a stepwise manner.
FIG. 19A shows a flow in which the light attenuation rate can be adjusted stepwise, and FIG. 19B shows the apparatus. In FIG. 19B, 1901 is a projector, 1902 is a polarizer, 1903 is an analyzer, 1904 is a rotating means including a motor, gears, other mechanical parts, and an electric circuit for rotating the analyzer 1903. Reference numeral 1906 denotes gear teeth provided around the analyzer so as to mesh with the gear of the rotating means 1904.
Polarizer 1902 covers the range of the area corresponding to the overlapping portion of the image area of projector 1901 where it is located with the image area of another projector (not shown) (indicated by hatching to cover). . Depending on the rotation state of the analyzer 1903, the brightness of only the overlapping portion of the projection light from the projector 1901 onto the screen is adjusted.
According to Malu's law, I (θ) = I (0) cos2θ
However, I (θ): intensity of light when passing through the polarizer and analyzer I (0): intensity of light when passing through the polarizer, so the luminance of the overlapping portion is changed by changing the angle of the analyzer Can be adjusted.
Each computer 503j in FIG. 5 generates an image for each projector 502j in the pixel buffer 512a, and generates an image obtained by correcting the distortion of the generated image in the pixel buffer 512b by an inverse function of the mapping for each projector 502j ( P1901). Each projector 502j projects the image corrected for distortion in the pixel buffer 512b onto the quadric curved screen 505 (P1902). When representing a night scene, the operator operates the rotating means 1904 to rotate the analyzer 1903. The light from the projector 502j is dimmed according to the rotation of the analyzer 1903 (P1903).
If the luminance of the overlapping portion is reduced to the same level as the luminance of each projector 502j alone, the overlapping portion becomes inconspicuous.
In the case of FIG. 1, each computer 103j generates an image for each projector 102j in the pixel buffer 112aj, and generates an image obtained by correcting the distortion of the generated image in the pixel buffer 112bj by an inverse function of the mapping for each projector 102j. (P1901). Each projector 102j projects the image corrected for distortion in the pixel buffer 112bj onto the quadric curved screen 101 (P1902). When representing a night scene, the operator operates the rotating means 1904 to rotate the analyzer 1903. In accordance with the rotation of the analyzer 1903, the light from the projector 102j is reduced (P1903).
If the luminance of the overlapping portion is reduced to the same level as the luminance of each projector 102j alone, the overlapping portion becomes inconspicuous.

(Application to stereo image display)
The above-described technique can also be realized in stereo video display. FIG. 20A is a flowchart for explaining an embodiment of a mapping function generating method related to a quadratic curved screen, and FIG. 20B is a flowchart explaining an embodiment of an image generating method related to a quadric curved screen. FIG. FIG. 21 is a functional block diagram for explaining an embodiment of a correction function generating apparatus and video generating apparatus relating to a quadratic curved screen. In FIG. 21, 2101 is a quadric curved screen, 21021i, 21022i (i = 1, 2, 3,... N) are projectors arranged around the screen, and 2103i is provided corresponding to each projector 21021i, 21022i. Multiple computers. 21041i, 21042i (i = 1, 2, 3,... N) are digital cameras constituting a stereo camera that can capture images projected by the projectors 21021i, 21022i, and 2105 connects the digital cameras 21041i, 21042i. Computer 2106 is a test point three-dimensional restoration means in the computer 2105, 2107 is a quadric surface coefficient matrix estimation means in the computer 2105, 2108 is a storage device in the computer 2105, 21091i and 21092i are first and second in each computer 2103i. Second mapping function and inverse function calculating means, 2110i is a storage device in each computer 2103i, 2111i is an image generating means provided in each computer 2103i, 2112ai, 2112bi is each computer A pixel buffer provided in 2103I, constitute the image generating means the computer 2103I, together with the image generating means 2111I. Reference numeral 2113 denotes a network connecting the computers 2103i and 2105. Reference numeral 2114i is an offset calculation means of each computer 2103i, 2115m (m = 1, 2,... M) is a digital camera, and a reference virtual located on the outer periphery so as to photograph the surface of the secondary curved screen 2101 projected by the projector 2102i. It is a camera.

([1] Mapping function related to quadratic surface for stereo display and its inverse function)
The stereo display is to generate a mapping function and an inverse mapping function for the viewpoints of both the left and right eyes instead of the one viewpoint described with reference to FIG. is there.
That is, a mapping function related to a quadric curved screen for stereo display and its inverse function are obtained as follows.
Three-dimensional restoration of the test points projected on the screen 2101 from each projector pi (21021i, 21022i) is performed by the stereo cameras 21021i, 21022i (step P2001 in FIG. 20A).
The restored three-dimensional points of each projector pi (21021i, 21022i) are integrated into the coordinate system of one (for example) camera 21021i, and a quadratic surface coefficient matrix (Q) representing the screen shape from the integrated three-dimensional points is obtained. Estimate (step P2002 in FIG. 20A).
The position and orientation (P) of each projector pi (21021i, 21022i) is estimated from the three-dimensional point of each projector pi (21021i, 21022i) after integration and the two-dimensional point corresponding to the test point (FIG. 20 (a)). ) Step P2003).
A plurality of reference virtual cameras that divide the projection surface of the quadric curved screen 2101 and shoot the entire image, and photograph the projection surface of the quadric curved screen 2101 projected by the projector pi (21021i, 21022i). Vm (2115m) (m = 1... M), the estimated quadric surface coefficient matrix (Q), the position and orientation of the plurality of reference virtual cameras Vm (2115m) (m = 1. From the position and orientation (P) of each projector pi (21021i, 21022i), the projected image at the pixel position of the projector pi (21021i, 21022i) corresponds to the pixel position of the reference virtual camera Vm (2115m) via the screen 2101. Map between virtual camera Vm (2115m) and projector pi (21021i, 21022i) Request grayed function Pusaivmpi (step P2004 in FIG. 20 (a)).
In step P2004, the above-described mapping function Ψvmpi, specifically, A _vmpi , E _pi , and e _vm are initialized by the expression (33) described in conjunction with FIG. The mapping function Ψvmpi is optimized (step P2004-2 in FIG. 20A).
A viewpoint virtual camera e (161514 or 161524) for right eye (or left eye) having position and orientation information is set at either position of the right eye or left eye looking at the quadric curved screen 2101, and for the right eye (or left eye). The mapping function Ψevm between the position and orientation of the viewpoint virtual camera e (161514 or 161524) and the reference virtual camera Vm (2115m) having the position and orientation is obtained, and by the mapping function Ψvmpi in step P2024,
Mapping function Ψr (l for right eye (or left eye) between each projector 21021i (for right eye) (or left eye projector 21022i) and right eye (or left eye) viewpoint virtual camera e (161514 or 161524) ) Find the epi and set the left-eye (or right-eye) viewpoint virtual camera e (161514 or 161524) having position and orientation information at the other position different from either the right eye or the left eye viewing the quadric curved screen 2101 Then, a mapping function Ψevm between the position and orientation of the viewpoint virtual camera e (161514 or 161524) for the left eye (or right eye) and the reference virtual camera Vm (2115m) having the position and orientation is obtained, and mapping according to the fourth process is performed. By the function Ψvmpi, each projector 21021i (21022i) and the left (Or right) obtaining the mapping function Ψl (r) epi for the left eye (or right) between for the viewpoint virtual camera e (one hundred sixty-one thousand five hundred and fourteen or 161524) (step P2005 in FIG. 20 (a)).
The inverse function is obtained by the right-eye and left-eye mapping functions ψrepi and ψlepi in step P2004 (step P2006 in FIG. 20A).

([2] Stereo image generation)
Stereo display is performed as follows.
The image generation means 2111i of each computer 2103i generates right-eye and left-eye images for each projector 21021i, 21022i in the pixel buffer 2112ai, and for the right-eye and left-eye for each projector 21021i, 21022i obtained as described above. An image obtained by correcting the distortion of the generated image is generated in the pixel buffer 2012 ib using an inverse function of mapping (step S2001 in FIG. 20B). The right and left eye images corrected for distortion in the pixel buffer 2012b are projected from the projectors 21021i and 21022i onto the quadric curved screen 2101 (step S2002 in FIG. 20B).

  The technique described as the second embodiment can also be applied to the stereo display. The technique described as the third embodiment can also be applied to stereo display.

FIG. 22A is a functional block diagram for explaining the fifth embodiment, and 2201 denotes a video device, which uses a television device, a video reproduction device, a personal computer, or the like. 2202 is a distortion-corrected video generation unit, 2203 is a converter that converts a signal of the video device 2201 so that it is suitable for processing by the distortion-corrected video generation unit 2202, 2204 is a memory that temporarily stores an output signal from the converter 2203, 2205, 2206, 2207, and 2208 are distortion correction means for correcting the distortion of the image by the inverse function of the mapping function as described in the first embodiment according to the position of the projector and the screen, and 2209, 2210, 2211, and 2212 are distortion correction. After correction, 2213, 2214, 2215, and 2216 are converters that convert the corrected video signals into signals suitable for projection onto the projector, 2217, 2218, 2219, and 2220 are projectors, and 2221 is the second memory. This is a second curved screen.
The mapping function and its inverse function are obtained by a computer (not shown) as described in the first embodiment.
The memory 2204 may be composed of a frame memory, but this need not be the case and may be a line memory. Further, the memory 2204 is not essential, and may be directly connected from the converter 2203 to the distortion correction means 2205 to 2208. Although the post-correction memories 2209, 2210, 2211, and 212 may be configured by frame memories, this need not be the case and may be line memories. Further, the post-correction memories 2209, 2210, 2211, and 2212 are not essential, and each of the distortion correction means 2205, 2206, 2207, and 2208 may be directly connected to the converters 2213, 2214, 2215, and 2216.
Arbitrary number n can be used as the number of projectors 2217, 2218, 2219, and 2220, and four examples are illustrated in this embodiment.

  Video from the video equipment 2201 (FIG. 22B) is converted by the converter 2203 so as to be suitable for processing, and is temporarily stored in the memory 2204. The video in the memory 2204 is divided by ranges projected onto the projectors 2217, 2218, 2219, and 2220, and distortion is corrected by the distortion correction means 2205, 2206, 2207, and 2208 within that range (step in FIG. 23). ST2301) and temporarily stored in the corrected memories 2209, 2210, 2211, and 2122 (FIG. 22C). Video signals are processed by converters 2213, 2214, 2215, and 2216, and projected onto screen 2221 by projectors 2217, 2218, 2219, and 2220 (step ST2302 in FIG. 23).

  In the converters 2213, 2214, 2215, and 2216, the brightness of the overlapping projection areas of the projectors 2217, 2218, 2219, and 2220 is adjusted by the brightness adjustment as described in ([2-3] brightness adjustment) of the first embodiment. Can be corrected.

  In FIG. 22, offset adjustment is performed by a computer (not shown) as described in the first embodiment ([3] Reduction of black offset due to leakage light of projector).

For example, the overlapping portion of the projection areas of two projectors can be enlarged, and the brightness of the overlapping portion can be increased as in the second embodiment.
The video from the video device 2201 is converted by the converter 2203 so as to be suitable for processing, and temporarily stored in the memory 2204. Images in the memory 2204 are divided by ranges projected onto the projectors 2217, 2218, 2219, and 2220, and distortion is corrected by the distortion correction units 2205, 2206, 2207, and 2208 within the ranges (steps in FIG. 24). ST2401) and temporarily stored in the corrected memories 2209, 2210, 2211 and 2122. Video signals are processed by converters 2213, 2214, 2215, and 2216, and are projected onto a screen 2221 by projectors 2217, 2218, 2219, and 2220 to increase the brightness of common overlapping portions (step ST2402 in FIG. 24). ).

  A polarizer and an analyzer as in the third embodiment are arranged in front of the projectors 2217, 2218, 2219, and 2220, and the light from the projector can be reduced to make the leaked light from the projector inconspicuous.

By dividing the even number of projectors 2217, 2218, 2219, and 2220 into those for the right eye and those for the left eye and applying the fourth embodiment, the mapping function and the inverse mapping function are obtained for the viewpoints of both the left and right eyes. Stereo images can be realized by using stereoscopic images as the images.
For example, projectors 2217 and 2219 are for the right eye, and projectors 2218 and 2220 are for the left eye.
Distortion correction is performed on the right-eye and left-eye images of the projectors 2217, 2218, 2219, and 2220 from the video equipment 2201 by the inverse functions of the right-eye and left-eye mappings of the projectors 2217, 2218, 2219, and 2220. (Step ST2501 in FIG. 25). The right-eye and left-eye images corrected for distortion are projected onto the quadric curved screen 2221 from the projectors 2217, 2218, 2219, and 2220 (step ST2502 in FIG. 25).

It is a functional block diagram explaining one Example of the production | generation apparatus of a mapping function, and an image generation apparatus. It is a flowchart explaining one Example of the production | generation method of a mapping function, and a video generation method. It is a figure explaining restoration | restoration of a three-dimensional point. It is a figure explaining arrangement | positioning of a camera, a projector, and a screen. It is a functional block diagram explaining one Example of the production | generation apparatus of a mapping function, and an image generation apparatus. It is a flowchart explaining one Example of the production | generation method of a mapping function, and a video generation method. It is a figure explaining the arrangement | positioning attachment of a projector. It is a schematic diagram of distortion correction by reverse distortion. It is a figure explaining the point on a screen. It is a figure explaining the point on a screen. It is a figure explaining the positional relationship in an image plane when the point on a screen is seen from a different viewpoint. It is a figure explaining the movement to a viewpoint position through a reference point from a camera installation position. It is a figure explaining division | segmentation and the viewing angle of the visual field of a screen. It is a program explaining the determination method of the weight of an overlap part. It is a figure explaining reduction of offset. It is a flowchart explaining the offset reduction method. It is a figure explaining the brightness improvement by the superimposition of a composite image. It is a figure explaining the sampling by a pixel buffer. It is a figure explaining the brightness | luminance adjustment of the duplication part at the time of the projection of a night scene. It is a flowchart explaining one Example of the production | generation method of a mapping function, and a video generation method. It is a functional block diagram explaining one Example of the production | generation apparatus of a mapping function, and an image generation apparatus. It is the functional block diagram explaining one Example of a video generator, and description of distortion correction. It is a flowchart explaining one Example of a video generation method. It is a flowchart explaining the brightness improvement by the superimposition of a composite image. It is a flowchart explaining one Example of a video generation method.

Explanation of symbols

  DESCRIPTION OF SYMBOLS 101 ... Quadric surface screen, 102i ... Projector, 103i ... Computer, 1041i, 1042i ... Digital camera (stereo camera), 105 ... Computer, 106 ... Test point three-dimensional reconstruction means, 107 ... Quadric surface coefficient matrix estimation means, 108 ... storage device, 1091i, 1092i ... mapping function and inverse function calculation means, 110i ... storage device in each computer 103i, 111i ... image generation means, 112ai, 112bi ... pixel buffer, 113 ... network. 114i: Offset calculation means, 115m: Reference virtual camera.

Claims (28)

Arranged to shoot a quadratic curved screen, a plurality of n projectors divided into the screens and arranged so that adjacent projections are continuous including overlapping portions, and a screen divided for each of the plurality of projectors A stereo camera, a computer, and a storage device,
Computer processing,
A first step of performing three-dimensional reconstruction of the test points projected from the projectors pi on the screen by the stereo camera;
A second step of integrating the restored three-dimensional points of each projector pi into one coordinate system and estimating a quadric surface coefficient matrix (Q) representing the screen shape from the integrated three-dimensional points;
A third step of estimating the position and orientation matrix (P) of each projector pi from the three-dimensional point of each projector pi after integration and the two-dimensional point of the test point corresponding thereto;
A plurality of reference virtual cameras vm (m = 1... M) that divide the projection surface of the quadratic curved screen and shoot the whole and image the projection surface of the quadric curved screen projected by the projector pi. ) wherein the and estimated quadratic surface coefficient matrix including an error (Q), the position and orientation matrix of the plurality of reference virtual camera vm (m = 1 ... M) , for each projector pi including an error that the estimated the calculation using the position and orientation matrix (P), 2 quadratic surface coefficient matrix error and the position and orientation matrix for each projector pi of (Q) includes an error of (P), the reference virtual camera vm and mapping functions between the projector pi A fourth process for obtaining Ψvmpi;
A viewpoint virtual camera e having position and orientation information is set at a position where the quadric curved screen is viewed, a mapping function Ψevm between the position and orientation of the viewpoint virtual camera e and the reference virtual camera vm having the position and orientation is obtained, A fifth step of obtaining a mapping function ψepi between each projector and the viewpoint virtual camera e by the mapping function ψvmpi in step 4;
A sixth step of obtaining the inverse function by the mapping function Ψepi of the fifth step;
A mapping function generation method for distortion correction and integration by divided imaging, characterized by comprising: In claim 1, the fourth process, the mapping function Ψvmpi including an error as the initial value, the mapping function generating method for integrating the distortion correction by dividing an imaging characterized by optimized the error to a minimum . A quadratic curved screen, a plurality of n projectors that are divided into the screens and arranged so that adjacent projection portions are continuous including overlapping portions, a computer provided for each projector, and a storage device ,
An image for each area projected by each projector is generated in the pixel buffer by each computer, and the generated image is distortion-corrected by an inverse function of the mapping for each projector previously obtained in claim 1 or 2. A first process for generating the pixel buffer;
A distortion correction and integration method by divided imaging, comprising a second process of projecting an image corrected for distortion in the pixel buffer from each projector onto the quadric curved screen. Arranged to shoot a quadratic curved screen, a plurality of n projectors divided into the screens and arranged so that adjacent projections are continuous including overlapping portions, and a screen divided for each of the plurality of projectors A stereo camera, a computer, and a storage device,
Computer processing,
A first step of performing three-dimensional reconstruction of the test points projected from the projectors pi on the screen by the stereo camera;
A second step of integrating the restored three-dimensional points of each projector pi into one coordinate system and estimating a quadric surface coefficient matrix (Q) representing the screen shape from the integrated three-dimensional points;
A third step of estimating the position and orientation matrix (P) of each projector pi from the three-dimensional point of each projector pi after integration and the two-dimensional point of the test point corresponding thereto;
A plurality of reference virtual cameras vm (m = 1... M) that divide the projection surface of the quadratic curved screen and shoot the whole and image the projection surface of the quadric curved screen projected by the projector pi. ) wherein the and estimated quadratic surface coefficient matrix including an error (Q), the position and orientation matrix of the plurality of reference virtual camera vm (m = 1 ... M) , for each projector pi including an error that the estimated the calculation using the position and orientation matrix (P), 2 quadratic surface coefficient matrix error and the position and orientation matrix for each projector pi of (Q) includes an error of (P), the reference virtual camera vm and mapping functions between the projector pi A fourth process for obtaining Ψvmpi;
A right-eye (or left-eye) viewpoint virtual camera e having position and orientation information is set at one of the positions of the right eye and the left eye viewing the quadratic curved screen, and the right-eye (or left-eye) viewpoint virtual camera e is set. A mapping function Ψevm between the position and orientation and the reference virtual camera vm having the position and orientation is obtained, and the mapping function Ψvmpi according to the fourth process is used to connect between each projector and the viewpoint virtual camera e for the right eye (or left eye). A right eye (or left eye) mapping function Ψr (l) epi is obtained, and the left eye (or right eye) having position and orientation information at the other position different from either the right eye or the left eye viewing the quadric surface screen Mapping function between the position and orientation of the viewpoint virtual camera e for the left eye (or right eye) and the reference virtual camera vm having the position and orientation by setting the viewpoint virtual camera e evm is obtained, and the left eye (or right eye) mapping function Ψl (r) epi between each projector and the left eye (or right eye) viewpoint virtual camera e is obtained by the mapping function Ψvmpi in the fourth process. The fifth step,
A sixth step of obtaining the inverse function by the right eye and left eye mapping functions ψ repi, ψ lepi of the fifth step;
A mapping function generation method for distortion correction and integration by divided imaging, characterized by comprising: In claim 4, the fourth process, the mapping function Ψvmpi including an error as the initial value, the mapping function generating method for integrating the distortion correction by dividing an imaging characterized by optimized the error to a minimum . A quadratic curved screen, a plurality of n projectors that are divided into the screens and arranged so that adjacent projection portions are continuous including overlapping portions, a computer provided for each projector, and a storage device ,
6. A right-eye and left-eye mapping function Ψrepi for each projector obtained in advance in claim 4 or claim 5 while generating images for right-eye and left-eye for each area projected by each projector by the computers. , Ψlepi inverse function, a first process of generating a distortion-corrected image in the pixel buffer,
A distortion correction and integration method by divided imaging, comprising a second step of projecting right-eye and left-eye images corrected for distortion in the pixel buffer from each projector onto the quadric curved screen. A quadratic curved screen, a plurality of n projectors divided into the screens and arranged so that adjacent projection locations are continuous including an overlapping portion, a computer, and a storage device,
The computer generates an image for each area projected by each projector, and the image obtained by correcting distortion of the generated image by an inverse function of the mapping for each projector previously obtained in claim 1 or 2 is used as a pixel buffer. A first process to generate;
Distortion correction by divided imaging, characterized in that it comprises a second step of projecting the image corrected for distortion in the pixel buffer from each projector onto the quadric curved screen and increasing the brightness of the common overlapping portion. And how to integrate. A quadratic curved screen, a plurality of n projectors divided into the screens and arranged so that adjacent projections are continuous including overlapping portions, and video equipment,
The video generated from the video device is divided and generated so as to be projected onto the plurality of n projectors onto the screen, and the divided and generated video is obtained by using an inverse function of mapping for each projector previously obtained in claim 1 or 2. A first process for generating a distortion-corrected image;
A distortion correction and integration method by divided imaging, comprising a second step of projecting the distortion-corrected video from each projector onto the quadric curved screen. A quadratic curved screen, a plurality of n projectors divided into the screens and arranged so that adjacent projections are continuous including overlapping portions, and video equipment,
The division from the video device is generated by projecting the image onto the screen onto the plurality of n projectors, and the division is performed according to the inverse function of the mapping function for the right eye and the left eye for each projector previously obtained in claim 4. A first step of generating a distortion corrected image of the generated image;
A distortion correction and integration method by divided imaging, comprising a second step of projecting the distortion-corrected right-eye and left-eye images from each projector onto the quadric curved screen. A quadratic curved screen, a plurality of n projectors disposed in front of the screen and arranged to have a common area projected onto the screen and overlapping, and a video device,
The video generated from the video device is divided and generated so as to be projected onto the plurality of n projectors onto the screen, and the divided and generated video is obtained by using an inverse function of mapping for each projector previously obtained in claim 1 or 2. A first process for generating a distortion-corrected image;
Distortion correction and integration method by divided imaging, characterized by comprising a second step of projecting the distortion-corrected image from each projector onto the quadric curved screen and increasing the brightness of the common overlapping portion .   11. The distortion correction and integration method by divided imaging according to claim 8, wherein the video device is a television device, a video playback device, or a personal computer. In the distortion correction and integration method by divided imaging according to claim 3 or claim 6 or claim 7 or claim 8 or claim 9 or claim 10 or claim 11,
A quadratic curved screen, a plurality of n projectors divided into the screens and arranged so that adjacent projections are continuous including overlapping portions, a computer, and a screen located in front of the screen so as to photograph the screen A stereo camera arranged,
A first step of photographing with a stereo camera in a state where none of the projectors projects onto the screen;
A second process in which a black image is projected from the projectors onto the screen and photographed by a stereo camera and repeated n times by a plurality of n projectors;
In the computer, the maximum value is obtained by the sum of the average value of the screen by each projector and the average value based on the difference between the brightness of the image for each projector in the second process and the brightness of the image obtained in the first process. A third step of obtaining an offset value by the difference between the maximum value and the average value;
The computer comprises a fourth process in which the luminance of the image projected at a constant cycle is measured, and the black level is adjusted by calculating and changing the offset value to be raised to the black offset at a constant cycle according to the average value. Distortion correction and integration method by divided imaging characterized by this. A quadratic curved screen, a plurality of n projectors divided into the screens and arranged so that adjacent projections are continuous including overlapping portions, a polarizing plate provided in front of each projector, and each projector A computer provided with a storage device;
The computer generates a video for each area projected by each projector in the pixel buffer, and the distortion of the generated video is corrected by an inverse function of the mapping function for each projector previously obtained in claim 1 or claim 2. A first step of generating an image in a pixel buffer;
A second process of projecting, from each projector, a distortion corrected image in the pixel buffer onto the quadric curved screen;
A distortion correction and integration method by divided imaging, comprising a third process of dimming with the polarizing plate when representing a night scene. A quadratic curved screen, a plurality of n projectors that are divided into the screens and arranged so that adjacent projections are continuous including overlapping portions, and are arranged to photograph the screen in front of the screen A stereo camera and a plurality of reference virtual cameras vm (m (m) arranged to divide the projection surface of the quadratic curved screen and shoot the whole and image the projection surface of the quadric curved screen projected by the projector pi. = 1... M), a viewpoint virtual camera e having position and orientation information arranged at a position to view the quadric curved screen, a computer, and a storage device,
A test point 3D restoration means for performing a 3D restoration by imaging a test point projected from each projector pi on a screen with a stereo camera;
Quadratic surface coefficient matrix (Q) estimation that integrates the restored three-dimensional points of each projector pi into one coordinate system and estimates a quadratic surface coefficient matrix (Q) representing the screen shape from the integrated three-dimensional points. Means,
Projector position / orientation estimation means for estimating a position / orientation matrix (P) of each projector pi from the three-dimensional point of each projector pi after integration and the two-dimensional point of the test point corresponding thereto;
Wherein the estimated quadratic surface coefficient matrix including an error (Q), the plurality of reference position and orientation matrix of the virtual camera vm (m = 1 ... M) , the position and orientation matrix for each projector pi including an error that the estimated ( P) to obtain a mapping function Ψvmpi between the reference virtual camera vm and the projector pi , including the error of the quadratic surface coefficient matrix (Q) and the error of the position and orientation matrix (P) of each projector pi . 1 mapping function calculation means;
A mapping function ψevm between the position and orientation of the viewpoint virtual camera e and the reference virtual camera vm having the position and orientation is obtained, and the mapping function ψvmpi by the first mapping function calculation means is used to determine whether each projector and the viewpoint virtual camera e A second mapping function calculation means for obtaining a mapping function Ψepi between
An inverse function calculating means for obtaining an inverse function by a mapping function Ψepi by the second mapping function calculating means;
A mapping function generating apparatus for distortion correction and integration by divided imaging, comprising a storage device storing mapping functions Ψvmpi, Ψepi and the inverse function for each of the plurality of projectors. In mapping function generator for distortion correction and integration by resolution imaging of claim 14, optimal first mapping function calculation means, a mapping function Ψvmpi including an error as the initial value, to optimize the error minimized A mapping function generation apparatus for distortion correction and integration by divided imaging, characterized by comprising: A quadratic curved screen, a plurality of n projectors that are divided into the screens and arranged so that adjacent projection portions are continuous including overlapping portions, a computer provided for each projector, and a storage device ,
An image for each region projected by each projector is generated in the pixel buffer by each computer, and the generated image is distortion-corrected by an inverse function of the mapping for each projector previously obtained in claim 14 or claim 15. Video generation means for generating a pixel buffer;
An apparatus for distortion correction and integration by divided imaging, comprising: a plurality of n projectors that project each distortion-corrected image from a pixel buffer of each computer onto the quadric curved screen. A quadratic curved screen, a plurality of n projectors that are divided into the screens and arranged so that adjacent projections are continuous including overlapping portions, and are arranged to photograph the screen in front of the screen A stereo camera and a plurality of reference virtual cameras vm (m (m) arranged to divide the projection surface of the quadratic curved screen and shoot the whole and image the projection surface of the quadric curved screen projected by the projector pi. = 1 ... M)
A viewpoint virtual camera er for right eye having position and orientation information arranged at the position of the right eye viewing the quadratic curved screen, a viewpoint virtual camera el for left eye having position and orientation information arranged at the position of the left eye, a computer, A storage device,
A test point 3D restoration means for performing a 3D restoration by imaging a test point projected from each projector pi on a screen with a stereo camera;
Quadratic surface coefficient matrix (Q) estimation that integrates the restored three-dimensional points of each projector pi into one coordinate system and estimates a quadratic surface coefficient matrix (Q) representing the screen shape from the integrated three-dimensional points. Means,
Projector position / orientation estimation means for estimating a position / orientation matrix (P) of each projector pi from the three-dimensional point of each projector pi after integration and the two-dimensional point of the test point corresponding thereto;
Wherein the estimated quadratic surface coefficient matrix including an error (Q), the plurality of reference position and orientation matrix of the virtual camera vm (m = 1 ... M) , the position and orientation matrix for each projector pi including an error that the estimated ( P) to obtain a mapping function Ψvmpi between the reference virtual camera vm and the projector pi , including the error of the quadratic surface coefficient matrix (Q) and the error of the position and orientation matrix (P) of each projector pi . 1 mapping function calculation means;
A mapping function Ψevm between the position and orientation of the viewpoint virtual camera e for the right eye (or left eye) and the reference virtual camera vm having the position and orientation is obtained, and each projector is determined by the mapping function Ψvmpi by the first mapping function calculation means. A right eye (or left eye) mapping function Ψr (l) epi between the left eye (or left eye) viewpoint virtual camera e, and the position and orientation of the left eye (or right eye) viewpoint virtual camera e The mapping function Ψevm between the reference virtual cameras vm having the position and orientation is obtained, and the left eye between each projector and the viewpoint virtual camera e for the left eye (or right eye) is determined by the mapping function Ψvmpi by the first mapping function calculation means. A second mapping function calculating means for obtaining a mapping function Ψl (r) epi for (or the right eye);
Inverse function calculation means for obtaining the inverse function by the right eye and left eye mapping functions Ψ repi, Ψ lepi of the second mapping function calculation means;
A mapping function generating apparatus for distortion correction and integration by divided imaging, comprising a storage device storing mapping functions Ψvmpi, Ψepi and the inverse function for each of the plurality of projectors. In mapping function generator for distortion correction and integration by resolution imaging of claim 17, optimizing the first mapping function calculation means, and a mapping function Ψvmpi including an error as the initial value, and the minimum error A mapping function generation device for distortion correction and integration by divided imaging, comprising: an optimization unit that performs the above-described optimization. A quadratic curved screen, a plurality of n projectors that are divided into the screens and arranged so that adjacent projection portions are continuous including overlapping portions, a computer provided for each projector, and a storage device ,
The right-eye and left-eye images for each area projected by each projector are generated in the pixel buffer by the computers, and the inverse of the right-eye and left-eye mapping for each projector previously obtained in claim 17 or 18. Image generating means for generating, in a pixel buffer, an image obtained by correcting the distortion of the generated image by a function;
An apparatus for distortion correction and integration by divided imaging, comprising: a plurality of n projectors that project right-eye and left-eye images corrected for distortion in the pixel buffer onto the quadric curved screen. A quadratic curved screen, a plurality of n projectors divided into the screens and arranged so that adjacent projection locations are continuous including an overlapping portion, a computer, and a storage device,
An image for each region projected by each projector is generated in the pixel buffer by each computer, and the generated image is distortion-corrected by an inverse function of the mapping for each projector previously obtained in claim 14 or claim 15. Video generation means for generating a pixel buffer;
A plurality of projectors arranged so as to project the distortion-corrected images from the pixel buffer of each computer onto the quadric curved screen to increase the brightness of the common overlapping portion. Distortion correction and integration device by featured divided imaging. A quadratic curved screen, a plurality of n projectors divided into the screens and arranged so that adjacent projections are continuous including overlapping portions, and video equipment,
The video generated from the video device is divided and generated so as to be projected onto the plurality of n projectors onto the screen, and the divided and generated video is obtained by an inverse function of mapping for each projector previously obtained in claim 14 or claim 15. A video generation means for generating a video with distortion corrected,
An apparatus for distortion correction and integration by divided imaging, comprising: a plurality of n projectors that project the distortion-corrected images onto the quadric curved screen. A quadratic curved screen, a plurality of n projectors divided into the screens and arranged so that adjacent projections are continuous including overlapping portions, and video equipment,
The division generation is performed by dividing and generating the image from the video device so as to be projected onto the n projectors on the screen, and by the inverse function of the right-eye and left-eye mapping for each projector previously obtained in claim 17. Image generation means for generating a distortion corrected image,
An apparatus for distortion correction and integration by divided imaging, comprising a plurality of n projectors that project the distortion-corrected right-eye and left-eye images onto the quadric curved screen. A quadratic curved screen, a plurality of n projectors divided into the screens and arranged so that adjacent projections are continuous including overlapping portions, and video equipment,
The video generated from the video device is divided and generated so as to be projected onto the plurality of n projectors onto the screen, and the divided and generated video is obtained by an inverse function of mapping for each projector previously obtained in claim 14 or claim 15. A video generation means for generating a video with distortion corrected,
Distortion correction by divided imaging, comprising: a plurality of n projectors arranged to project the distortion-corrected image on the quadric curved screen and increase the brightness of the common overlapping portion; Integrated device.   24. The distortion correction / integration apparatus according to claim 21, 22 or 23, wherein the video equipment is a television apparatus, a video reproduction apparatus, or a personal computer.   In the distortion correction and integration device by divided imaging according to claim 16 or claim 19 or claim 20 or claim 21 or claim 22 or claim 23 or claim 24,
  A quadratic curved screen, a plurality of n projectors that are divided into the screens and arranged so that adjacent projections are continuous including overlapping portions, and are arranged to photograph the screen in front of the screen A stereo camera,
  A camera that shoots a screen in a first state that is not projected from any of the projectors and that shoots a second state of the screen that projects a black state from each projector;
  The average value of the screen by each projector and the sum of the average values by the difference between the luminance of the video for each projector that has projected the second state photographed by the camera and the luminance of the video that has photographed the first state An offset calculation means for obtaining a maximum value by means of and obtaining an offset value by a difference between the maximum value and the average value;
  A video for generating a video by measuring the luminance of a video projected at a constant cycle by each of the computers, adjusting the black level according to the average value, calculating and changing the offset value to be raised to a black offset at a fixed cycle An apparatus for distortion correction and integration by divided imaging, characterized by comprising: generating means. A quadratic curved screen, a plurality of n projectors divided into the screens and arranged so that adjacent projections are continuous including overlapping portions, a polarizing plate, a computer provided for each projector, and a storage device And
An image for each region projected by each projector is generated in the pixel buffer by each computer, and the generated image is distortion-corrected by an inverse function of the mapping for each projector previously obtained in claim 14 or claim 15. Video generation means for generating a pixel buffer;
A plurality of projectors for projecting each distortion-corrected image from the pixel buffer of each computer onto the quadric curved screen;
A distortion correction and integration apparatus using divided imaging, which is provided in front of each projector and includes a polarizing plate that reduces the amount of light from each projector when representing a night scene.   19. Each of the projectors is installed on a floor surface by a flexible arm having a degree of freedom, and is used for distortion correction and integration by divided imaging according to claim 14 or claim 15 or claim 17 or claim 18. Mapping function generator.   27. The distortion correction and integration device by divided imaging according to claim 16, wherein each projector is installed on a floor surface by a flexible arm having a degree of freedom.

Images