WO2020218028A1 - Image processing device, image processing method, program, and image processing system - Google Patents

Abstract

The present technology relates to an image processing device, an image processing method, a program, and an image processing system which enables re-correction to be easily performed when a mismatch occurs between corresponding points on a projection image projected by a projection device and a captured image captured by an image-capturing device. An image processing device according to an aspect of the present technology detects a mismatch between corresponding points, which occurs after initial correction has been performed. The initial correction includes corresponding point detection in which the corresponding points on the projection image and the captured image are detected, and orientation estimation in which the orientations of the projection device and the image-capturing device are estimated on the basis of the result of the corresponding point detection. In addition, on the basis of corresponding point information obtained by the initial correction, the image processing device generates a re-correction pattern based on the position of the corresponding point on the projection image, detects the mismatch between the corresponding point on the basis of a captured image obtained by capturing an image of the re-correction pattern projected from the projection device, and updates the corresponding point information. The present technology can be applied to an image processing device which controls the projection of a projector.

Description

Image processing equipment, image processing methods, programs, and image processing systems

The present technology relates to an image processing device, an image processing method, a program, and an image processing system, and in particular, a deviation occurs in the corresponding point on each image between the projected image projected by the projection device and the captured image captured by the photographing device. The present invention relates to an image processing apparatus, an image processing method, a program, and an image processing system that enable easy re-correction in the case of the above.

There is a technique for projecting one image using a plurality of projectors on a screen having a predetermined shape such as a flat surface, a curved surface, or a dome shape. In order to project one image using a plurality of projectors, it is necessary to correct the distortion in consideration of the shape of the screen and estimate the overlapping area where the images projected from the projectors overlap.

This can be achieved by calibration using the 3D shape measurement function of the projector / camera system. The projector / camera system is a system in which a projector and a camera are provided, and the posture of each device can be estimated by taking an image of a predetermined pattern projected from the projector with the camera.

By estimating the parameters representing the postures of the projector and the camera and performing geometric correction of the image projected from the projector according to the estimation result, it is possible to project a high-quality image.

Japanese Unexamined Patent Publication No. 2016-14720

After performing initial correction including calibration once, correction deviation may occur due to changes over time of the projector or physical contact with the device. The correction deviation means that a deviation occurs at the corresponding point on each image of the image projected by the projector and the image taken by the camera, which is once detected by the initial correction.

Such correction deviation has a large effect on the quality of the viewing experience, even if it is trivial.

This technology was made in view of such a situation, and re-correction occurs when there is a deviation between the corresponding points on the projected image projected by the projection device and the captured image captured by the photographing device. It makes it easy to do.

The image processing device on one aspect of the present technology determines the corresponding points on each image based on the projected image projected by the projection device and the captured image obtained by photographing the projected projected image by the photographing device. Detects the deviation of the corresponding points that occurs after the initial correction including the detection of the corresponding points to be detected and the attitude estimation that estimates the postures of the projection device and the photographing device based on the result of the corresponding point detection. It is composed of a predetermined pattern based on the position of the corresponding point on the projected image based on the corresponding point information which is the information indicating the relationship between the detection unit and the corresponding point obtained by the initial correction. Based on the generation unit that generates the recorrection pattern and the photographed image obtained by photographing the recorrection pattern projected from the projection device by the photographing apparatus, the corresponding points on the captured image It is provided with an update unit that detects a deviation and updates the corresponding point information.

One aspect of the present technology is to detect a corresponding point on each image based on a projected image projected by the projection device and a captured image obtained by photographing the projected projected image by the photographing device. The deviation of the corresponding points that occurs after the initial correction including the point detection and the posture estimation that estimates the postures of the projection device and the photographing device based on the result of the corresponding point detection is detected. Further, based on the corresponding point information which is the information indicating the relationship of the corresponding points obtained by the initial correction, the re-correction composed of a predetermined pattern based on the position of the corresponding points on the projected image. A pattern is generated, and the deviation of the corresponding point on the captured image is detected based on the captured image obtained by photographing the recorrection pattern projected from the projection device by the photographing device. The corresponding point information is updated.

It is a figure which shows the configuration example of the projector / camera system which concerns on one Embodiment of this technology. It is a figure which shows the other configuration example of a projector / camera system. It is a figure which shows the example of the coding using the Gray code. It is a figure which shows the example of other coding. It is a figure which shows the example of decoding. It is a flowchart explaining the initial correction process. It is a figure which shows the example of the result of correspondence point detection. It is a flowchart explaining the posture estimation process. It is a block diagram which shows the configuration example of the hardware of an image processing apparatus. It is a block diagram which shows the functional structure example of an image processing apparatus. It is a flowchart explaining the recorrection process. It is a figure which shows the relationship of each apparatus before the correction deviation occurs. It is a figure which shows the relationship of each apparatus after the correction deviation occurs. It is a flowchart explaining the correspondence point information update process performed in step S33 of FIG. It is a figure which shows the example of the pattern for re-correction. It is a figure which shows the example of the pattern for re-correction. It is a figure which shows the example of the camera image which shows the pattern for re-correction. It is a figure which shows the shift of a dot schematically. It is a figure which shows the example of the update of correspondence point information. It is a flowchart explaining the parameter estimation value update process performed in step S34 of FIG. It is a figure which shows the other configuration example of a projector / camera system. It is a figure which shows the other configuration example of a projector / camera system.

Hereinafter, modes for implementing the present technology will be described. The explanation will be given in the following order.
1. 1. Projector / camera system 2. Corresponding point detection at the time of initial correction 3. Initial correction processing 4. Configuration of image processing device 5. Operation of image processing device 6. Detection of correction deviation (change of corresponding point) 7. Correspondence point information update processing 8. Parameter estimated value update processing 9. Geometric correction 10. Modification example

<Projector / camera system>
FIG. 1 is a diagram showing a configuration example of a projector / camera system as an image processing system according to an embodiment of the present technology.

The projector / camera system of FIG. 1 is configured by connecting a plurality of projectors, which are projection devices, and a plurality of cameras, which are imaging devices, to an image processing device 1 such as a PC. In the example of FIG. 1, two projectors of projectors 2-1 and 2-2 and two cameras of cameras 3-1 and 3-2 are provided. A screen 11 serving as a projection surface is provided in front of the projectors 2-1 and 2-2.

The projector 2-1 and the projector 2-2 are installed so that the projection range of the projector 2-1 and the projection range of the projector 2-2 partially overlap. An overlapping region is formed on the screen 11 in which the image projected from the projector 2-1 and the image projected from the projector 2-2 overlap.

The camera 3-1 is installed so that the shooting range includes the projection range of the projector 2-1. At least, the projected image of the projector 2-1 projected on the screen 11 is captured in the captured image captured by the camera 3-1.

Similarly, the camera 3-2 is installed so that the shooting range includes the projection range of the projector 2-2. At least, the projected image of the projector 2-2 projected on the screen 11 is captured in the captured image captured by the camera 3-2.

For example, the camera 3-1 is provided near the projector 2-1 and the camera 3-2 is provided near the projector 2-2. The projector 2-1 and the camera 3-1 and the projector 2-2 and the camera 3-2 may each be composed of devices having the same housing. Further, the cameras 3-1 and 3-2 may be provided at positions away from the projectors 2-1 and 2, respectively.

In the example of FIG. 1, two projectors and two cameras are provided, but the number of projectors and cameras is arbitrary.

The projectors 2-1 and 2-2 and the cameras 3-1 and 3-2 have optical systems capable of modeling distortion, such as a general perspective projection optical system and an optical system having a fisheye lens. The shape of the screen 11 may be any shape that can be modeled, such as a flat surface, a curved surface, and a hemispherical surface (dome shape).

The image processing device 1 is provided with a projection control unit 21 that controls projection by projectors 2-1 and 2-2, and a shooting control unit 22 that controls shooting by cameras 3-1 and 3-2. As will be described later, the image processing device 1 is provided with various configurations other than the projection control unit 21 and the imaging control unit 22.

FIG. 2 is a diagram showing another configuration example of the projector / camera system.

As shown in FIG. 2, it is also possible to provide sensors 4-1 and 4-2 composed of an acceleration sensor and a gyro sensor.

The sensor data processing unit 23 of the image processing device 1 processes the sensor data supplied from the sensor 4-1 and the sensor data supplied from the sensor 4-2. Based on the processing result by the sensor data processing unit 23, the correction deviation as described later is detected.

Hereinafter, when it is not necessary to distinguish between projectors 2-1 and 2, they are collectively referred to as projector 2. When it is not necessary to distinguish between cameras 3-1 and 3-2, they are collectively referred to as camera 3. When it is not necessary to distinguish between sensors 4-1 and 4-2, they are collectively referred to as sensor 4.

The image processing device 1 has a three-dimensional measurement function that automatically measures (estimates) the posture of each device and calculates parameters, and a geometry that corrects the image projected from the projector 2 based on the calculated parameters. It has a correction function. By performing geometric correction using the parameters calculated by the posture estimation, the projected image projected on the screen 11 becomes a high-quality one with less distortion and the like.

Here, the parameters calculated by the image processing device 1 by posture estimation include internal parameters and external parameters.

The internal parameters include information such as the focal length and principal point of the projector 2 and the camera 3. Other information, such as the distortion factor of the lens, may be estimated as an internal parameter. The external parameters include information on the postures (rotation / translation) of the projector 2 and the camera 3.

Posture estimation for calculating such parameters is performed by inputting correspondence point information. The corresponding point information is information representing corresponding points on the respective images of the projector image, which is an image projected by the projector 2, and the camera image, which is an image obtained by shooting with the camera 3.

In order to properly calculate the parameters used for geometric correction by posture estimation, it is necessary to accurately detect the corresponding points. Based on the result of the corresponding point detection, the posture estimation for estimating the posture of the projector 2 and the camera 3 is performed, and the parameters are calculated. Further, the shape and position of the screen 11 are estimated based on the parameters calculated by the posture estimation.

In general, in the corresponding point detection, a pattern for sensing is projected from a projector, and the pattern projected on the screen is photographed by a camera to obtain the correspondence between the pixel positions of the projector image and the camera image. Structured Lighting ( It is done using a technique called structured light projection).

In the projector / camera system of FIGS. 1 and 2, initial correction including corresponding point detection and posture estimation is performed in advance at a predetermined timing such as before projecting an image of the content.

<Detection of corresponding points during initial correction>
Typical sensing patterns include binary code and gray code. Correspondence point detection using these codes encodes each pixel of the projector image by sequentially projecting a black-and-white (binary) pattern, captures the state of projection, and decodes each pixel of the projector image. The correspondence between the image and each pixel of the camera image is obtained.

FIG. 3 is a diagram showing an example of coding using a Gray code.

As shown in A of FIG. 3, the coding of each pixel of the projector image is performed by projecting a black and white pattern while switching in time series in each of the horizontal direction and the vertical direction. When performing 8-bit coding, eight patterns are used in each of the horizontal direction and the vertical direction.

For example, the pixels in the upper left corner are projected in the pattern of "black, black, black, ..." when projected in the horizontal direction. Assuming that white is "1" and black is "0", the pixels in the upper left corner are encoded by "000 ...". In coding using a typical Gray code, a striped pattern is projected in the horizontal direction and the vertical direction so that all the pixels of the projector image can be coded.

On the other hand, in decoding, the patterns projected on the screen are sequentially photographed, and as shown in B of FIG. 3, the black and white changes of each pixel on the camera image are set to "1" (white) and "0" (black). It is done as shown. For example, the pixel at the position surrounded by a rectangle in B in FIG. 3 is represented by the reference numeral “00011000”.

In this way, by detecting the unique code of each pixel of the camera image and associating it with each pixel of the projector image represented by the same code, the corresponding points on the respective images of the projector image and the camera image are detected. To.

FIG. 4 is a diagram showing an example of other coding.

The coding method shown in FIG. 4 is a method using a dot pattern in which a pattern representing a gray code is arranged at a position of a specific pixel group.

First, by projecting a checker pattern as shown in A of FIG. 4, the positions of the corner points of each checker are detected using the camera image. The checker pattern is a pattern composed of a black substantially square area and a white approximately square area arranged in a checkered pattern.

After that, as shown in B of FIG. 4, a dot pattern in which dots representing the gray code are arranged is projected at the position of the corner point of the checker, and coding is performed. The projected dot pattern is photographed by a camera, and as shown in FIG. 5, decoding is performed so that the change in white / black of each pixel on the camera image is represented by “1” and “0”.

By using such a dot pattern, it is possible to acquire the corresponding points on the projector image and the camera image with high accuracy at checker intervals.

Hereinafter, the corresponding points on the projector image are appropriately referred to as projector compatible points. Corresponding points on the camera image are camera corresponding points.

<Initial correction processing>
A series of initial correction processes including the above-mentioned correspondence point detection will be described with reference to the flowchart of FIG.

In step S1, the image processing device 1 generates a coding pattern. The coding pattern generated here is a pattern for pixels in the entire image as described with reference to FIG. 3, for example.

In step S2, the image processing device 1 controls the projector 2 and projects a coding pattern. Further, the image processing device 1 controls the camera 3 and captures the coded pattern projected on the screen 11. The projection and shooting of the coding pattern are repeated as many times as (number of coding bits) × (number of projectors) × (number of cameras).

In step S3, the image processing device 1 performs decoding processing as described above based on each camera image, and detects corresponding points on the respective images of the projector image and the camera image.

FIG. 7 is a diagram showing an example of the result of corresponding point detection.

Corresponding point detection is performed by associating pixels on the projector image and pixels on the camera image having the same code as corresponding points.

In the example of FIG. 7, the pixel P1 at the coordinates (X1, Y1) on the projector image and the pixel p1 at the coordinates (x1, y1) on the camera image are detected as corresponding points. Pixel P1 is a projector compatible point, and pixel p1 is a camera compatible point corresponding to pixel P1 as a projector compatible point.

Further, the pixel P2 at the coordinates (X2, Y2) on the projector image and the pixel p2 at the coordinates (x2, y2) on the camera image are detected as corresponding points. Pixels P3 to PN on the projector image and pixels p3 to pn on the camera image are also detected as corresponding points, respectively.

By the corresponding point detection as described above, the corresponding point information which is the information indicating the corresponding point on each image of the projector image and the camera image is generated. The correspondence point information is information representing the correspondence relationship as shown in FIG.

In step S4 of FIG. 6, the image processing device 1 performs the posture estimation process. The posture estimation process is performed based on the correspondence point information obtained by the correspondence point detection. The posture estimation process calculates a parameter estimated value which is a parameter representing the postures of the projector 2 and the camera 3.

The posture estimation process will be described with reference to the flowchart of FIG.

In step S11, the image processing device 1 estimates the camera parameters based on the corresponding point information.

In step S12, the image processing device 1 estimates the projector parameters based on the corresponding point information. Camera parameter estimates are also used to estimate projector parameters.

In step S13, the image processing device 1 optimizes the projector parameter estimated value and the camera parameter estimated value, and calculates the projector / camera parameter estimated value.

The parameter estimation value is optimized so that, for example, when ray tracking is performed for each of the projector corresponding point and the camera corresponding point, each tracking destination becomes one point on the screen 11. And the camera parameter estimates are adjusted respectively.

After the parameter estimated value is optimized, the process returns to step S4 in FIG. 6 and the subsequent processing is performed.

In step S5 of FIG. 6, the image processing device 1 performs geometric correction on the image projected from the projector 2 based on the parameter estimated value calculated by the posture estimation process.

As described above, the initial correction is a process that includes at least corresponding point detection and attitude estimation.

Correspondence point detection performed at the time of initial correction requires a large amount of processing because it is necessary to project a plurality of patterns. For example, when coding in the time direction such as Gray code is used, it is necessary to project a pattern for each projector by the number of coding bits. Specifically, it is necessary to project a pattern by (the number of encoded bits) × (the number of projectors) and shoot as many as the number of cameras.

That is, as the number of projectors increases, the total number of projected images increases by the number of encoded bits. At the same time, since the number of corresponding points to be sensed increases, the time required for posture estimation after detecting the corresponding points also becomes long.

After a series of initial corrections, depending on the actual operating environment, correction deviations may occur due to physical contact with the device. The correction deviation is a deviation between the projector corresponding point and the camera corresponding point detected by the corresponding point detection at the time of initial correction.

In the projector / camera system of FIGS. 1 and 2, when the deviation amount of the correction deviation is small, the re-correction for correcting the correction deviation does not perform the initial correction as described above again. It is performed based on the information obtained by the initial correction.

This makes it possible to easily perform re-correction even if a correction deviation occurs. A series of recorrection processes will be described later.

<Configuration of image processing device>
FIG. 9 is a block diagram showing a configuration example of the hardware of the image processing device 1.

The CPU (Central Processing Unit) 51, ROM (Read Only Memory) 52, and RAM (Random Access Memory) 53 are connected to each other by the bus 54.

An input / output expansion bus 55 is further connected to the bus 54. A GPU (Graphics Processing Unit) 56, a UI (User Interface) I / F 59, a communication I / F 62, and a recording I / F 63 are connected to the input / output expansion bus 55.

The GPU 56 renders the image projected from the projector 2 using the VRAM 57. For example, the GPU 56 generates a pattern to be projected from the projector 2 and outputs it to the display I / F 58.

The display I / F 58 is an interface for outputting projector images and the like. The display I / F 58 is configured as an interface of a predetermined standard such as HDMI (registered trademark) (High-Definition Multimedia Interface). The display I / F 58 outputs the projector image supplied from the GPU 56 to the projector 2 and projects it.

The UI I / F59 is an interface for detecting operations. The UI I / F 59 detects a user's operation performed by using the keyboard 60 or the mouse 61, and outputs information indicating the content of the operation to the CPU 51.

The communication I / F 62 is an interface for communication with an external device. The communication I / F 62 is composed of network interfaces such as wireless LAN and wired LAN. The communication I / F 62 communicates with an external device via a network such as the Internet, and transmits / receives various data. The content played back in the projector / camera system may be provided from the server via the network.

Further, the communication I / F 62 receives the captured image captured by the camera 3 and transmitted from the camera 3. The communication I / F 62 receives various data transmitted from an external device, such as sensor data transmitted from the sensor 4.

The recording I / F 63 is an interface for a recording medium. A recording medium such as an HDD 64 or a removable medium 65 is attached to the recording I / F 63. The recording I / F 63 reads the data recorded on the mounted recording medium and writes the data to the recording medium. In addition to the contents, various data such as programs executed by the CPU 51 are recorded in the HDD 64.

FIG. 10 is a block diagram showing a functional configuration example of the image processing device 1.

As shown in FIG. 10, the control unit 101 is realized in the image processing device 1. In addition to the projection control unit 21 and the imaging control unit 22, the control unit 101 includes a coding pattern generation unit 111, a corresponding point detection unit 112, a posture estimation unit 113, a correction information storage unit 114, a geometric correction unit 115, and a correction deviation detection. It is composed of a unit 116, a pattern generation unit 117 for re-correction, and an update unit 118.

At the time of initial correction, the projection control unit 21 projects a plurality of coded patterns generated by the coded pattern generation unit 111 from each projector 2.

Further, at the time of re-correction, the projection control unit 21 projects the re-correction pattern generated by the re-correction pattern generation unit 117 from the projector 2 that causes the correction deviation. As will be described in detail later, the re-correction pattern is a pattern for re-correcting the correction deviation, and only one pattern is generated for each projector 2 that causes the correction deviation.

The shooting control unit 22 controls the camera 3 at the time of initial correction to shoot the coded pattern projected on the screen 11. By controlling the communication I / F 62, the photographing control unit 22 acquires a photographed image in which the coded pattern is captured and outputs it as a camera image to the corresponding point detection unit 112.

Further, the shooting control unit 22 controls the camera 3 at the time of re-correction, and causes the camera 3 to shoot the re-correction pattern projected on the screen 11. By controlling the communication I / F 62, the shooting control unit 22 acquires a shooting image in which the recorrection pattern is captured and outputs it as a camera image to the updating unit 118.

At the time of initial correction, the coding pattern generation unit 111 generates a plurality of coding patterns as a pattern for sensing as described with reference to FIGS. 3 and 4. The process of step S1 in FIG. 6 is a process performed by the coding pattern generation unit 111. The coded pattern generated by the coded pattern generation unit 111 is supplied to the projection control unit 21 and the corresponding point detection unit 112 as a projector image.

At the time of initial correction, the corresponding point detection unit 112 detects the corresponding point based on the coded pattern generated by the coded pattern generation unit 111 and the camera image in which the coded pattern is captured, which is supplied from the photographing control unit 22. .. The process of step S3 in FIG. 6 is a process performed by the corresponding point detection unit 112.

That is, each pixel of the camera image as described with reference to FIG. 3 and the like is decoded, and the corresponding point is detected. The correspondence point information obtained by the correspondence point detection by the correspondence point detection unit 112 is supplied to the posture estimation unit 113 and the correction information storage unit 114.

At the time of initial correction, the posture estimation unit 113 estimates the posture based on the correspondence point information supplied from the correspondence point detection unit 112, and calculates the parameter estimation value. The process of step S4 in FIG. 6 is a process performed by the posture estimation unit 113. The parameter estimated value calculated by the posture estimation unit 113 is supplied to the correction information storage unit 114 and the geometric correction unit 115.

The correction information storage unit 114 stores the corresponding point information supplied from the corresponding point detecting unit 112 and the parameter estimated value supplied from the posture estimation unit 113. The information stored in the correction information storage unit 114 is read out at the time of re-correction.

Further, the correction information storage unit 114 stores the updated corresponding point information and the parameter estimated value updated by the update unit 118 at the time of re-correction. The geometric correction by the geometric correction unit 115 after the re-correction is performed based on the updated information stored in the correction information storage unit 114.

The geometric correction unit 115 performs geometric correction of the image projected from each projector 2 based on the parameter estimation value supplied from the posture estimation unit 113. The process of step S5 in FIG. 6 is a process performed by the geometric correction unit 115. The corrected image that has been geometrically corrected by the geometric correction unit 115 is supplied to the projection control unit 21 via a path (not shown) and projected.

The correction deviation detection unit 116 detects the correction deviation based on the information input from the outside. For example, when an impact is applied to a predetermined projector 2 and the sensor 4 detects that the posture is displaced, it is detected as a correction deviation has occurred.

When the correction deviation detection unit 116 detects that the correction deviation has occurred, the correction deviation detection unit 116 compares the correction deviation deviation amount with the threshold amount. When the deviation amount of the correction deviation is smaller than the threshold value (small amount), the correction deviation detection unit 116 outputs information indicating that re-correction is performed to the re-correction pattern generation unit 117 and the update unit 118. ..

On the other hand, when the deviation amount of the correction deviation is larger than the threshold value, the correction deviation detection unit 116 outputs information indicating that the same correction as the initial correction is performed again to the coding pattern generation unit 111.

When the correction deviation detection unit 116 selects to perform re-correction, the re-correction pattern generation unit 117 reads out the corresponding point information obtained by the initial correction from the correction information storage unit 114 and acquires it. Further, the re-correction pattern generation unit 117 generates a re-correction pattern based on the corresponding point information before the occurrence of the correction deviation read from the correction information storage unit 114.

In this way, the re-correction pattern is generated based on the corresponding point information obtained at the time of initial correction. The recorrection pattern generated by the recorrection pattern generation unit 117 is supplied to the projection control unit 21 and projected from the projector 2 that causes the correction deviation.

When the correction deviation detection unit 116 selects to perform re-correction, the update unit 118 reads out the corresponding point information and the parameter estimated value obtained by the initial correction from the correction information storage unit 114 and acquires them.

When the camera image in which the re-correction pattern is captured is supplied from the shooting control unit 22, the update unit 118 detects the position of the camera corresponding point after the correction deviation occurs based on the camera image, and detects the position after the correction deviation occurs. The corresponding point information is updated according to.

In addition, the update unit 118 updates the parameter estimated value by referring to the corresponding point information after the update. The corresponding point information and the parameter estimated value updated by the update unit 118 are stored in the correction information storage unit 114 and used for the subsequent geometric correction.

<Operation of image processing device>
Here, the recorrection process of the image processing apparatus 1 having the above configuration will be described with reference to the flowchart of FIG.

The process of FIG. 11 is started when the initial correction process as described with reference to FIG. 6 is performed and the state of the projector / camera system is in the correction state.

In step S31, the correction deviation detection unit 116 detects the correction deviation based on the information input from the outside.

In step S32, the correction deviation detection unit 116 determines whether or not the deviation amount of the correction deviation is very small. Here, when the deviation amount is less than the threshold value, it is determined to be minute.

If the amount of deviation of the correction deviation is small, re-correction is performed using the corresponding point information before the occurrence of the correction deviation and the parameter estimated value obtained by the initial correction. The corresponding point information read from the correction information storage unit 114 is supplied to the re-correction pattern generation unit 117 and the update unit 118, and the parameter estimated value is supplied to the update unit 118.

In step S33, the corresponding point information update process is performed. By the correspondence point information update process, the correspondence point information before the correction deviation occurs is updated, and the updated correspondence point information is generated. The corresponding point information update process will be described later with reference to the flowchart of FIG.

In step S34, the parameter estimated value update process is performed. The parameter estimated value update process updates the parameter estimated value before the correction deviation occurs, and generates the updated parameter estimated value. The parameter estimation value update process will be described later with reference to the flowchart of FIG.

In step S35, the geometric correction unit 115 performs geometric correction based on the updated parameter estimated value. The corrected image with geometric correction is supplied to the projection control unit 21 and projected from the projector 2. As a result, the image after the geometric correction considering the slight correction deviation is projected.

On the other hand, since the deviation amount of the correction deviation is larger than the threshold value, if it is determined in step S32 that the deviation amount of the correction deviation is not small, the process proceeds to step S36 and the initial correction process is performed. Here, the same processing as that described with reference to FIG. 6 is performed, the corresponding point information is obtained by the corresponding point detection using a plurality of coding patterns, and the posture is estimated based on the corresponding point information. Will be done.

After the initial correction process is performed in step S36, the process proceeds to step S35, and the geometric correction unit 115 performs geometric correction using the parameter estimated value obtained by the initial correction process.

Details of each process constituting the above recorrection process will be described in the following order.
-Detection of correction deviation (change in corresponding point) (step S31)
-Correspondence point information update process (step S33)
-Parameter estimation value update process (step S34)
-Geometric correction (step S35)

<Detection of correction deviation (change of corresponding point)>
The correction deviation occurs due to physical contact with the device (projector 2, camera 3, screen 11), aging of the device, adjustment of the zoom / shift function of the projector 2, and the like.

FIG. 12 is a diagram showing the relationship of each device before the occurrence of correction deviation.

In the example of FIG. 12, the shooting ranges of the cameras 3-1 and 3-2 include the projection range of the projector 2-1 and the projection range of the projector 2-2, respectively. The images projected from the projector 2-1 are taken by the cameras 3-1 and 3-2, respectively, and the images projected from the projector 2-2 are taken by the cameras 3-1 and 3-2, respectively.

Since the correction state is in the correction state where no correction deviation has occurred, there is no deviation at the corresponding points between the projector image and the camera image.

In FIG. 12, the pixel at the position p11 on the camera image of the camera 3-1 corresponds to the pixel at the position p1 on the projector image projected by the projector 2-1. On the camera image of 2, the pixel at position p12-1 is the corresponding point. As shown by the broken line passing through each corresponding point, the light rays of each pixel serving as the corresponding point intersect at the position P1 on the screen 11, and there is no deviation in the corresponding point.

Similarly, for the pixel at the position p2 on the projector image projected by the projector 2-2, the pixel at the position p11-2 on the camera image of the camera 3-1 becomes a corresponding point, and the camera 3-2 On the camera image of, the pixel at position p12-2 is the corresponding point. As shown by the broken line passing through each corresponding point, the light rays of each pixel serving as the corresponding point intersect at the position P2 on the screen 11, and there is no deviation in the corresponding point.

FIG. 13 is a diagram showing the relationship between each device after the correction deviation occurs.

For example, when the position of the projector 2-2 shifts as shown by the arrow A1, the projector corresponding point and the camera corresponding point are between the projector 2-2 and the camera 3-1 and between the projector 2-2 and the camera 3-2. In each case, the positions are displaced as shown by the arrow A2.

As described above, the correction deviation is defined as a state in which the position of the corresponding point on the camera image is changed with respect to the corresponding point on the projector image due to any change in the projector 2, the camera 3, or the screen 11.

If a slight correction deviation occurs, re-correction is performed based on the information before the correction deviation occurred obtained by the initial correction. The slight correction deviation means a small deviation such that the projection position from the projector 2 is slightly deviated on the camera image.

Such a slight correction deviation is detected by the correction deviation detection unit 116 in step S31 of FIG.

The correction deviation due to physical contact with the device is detected, for example, based on the measurement result by the sensor 4 provided as shown in FIG. The correction deviation detection unit 116 identifies the device that caused the correction deviation based on the measurement result of the sensor 4 including the acceleration sensor and the gyro sensor supplied from the sensor data processing unit 23, and uses it as a trigger. , Start re-correction.

Further, the correction deviation due to the time-dependent change of the projector 2 and the change of the zoom shift is detected by using, for example, a pattern for detecting the deviation of the corresponding point. In this case, the image in which the pattern is embedded is projected from the projector 2, and the projected image is captured by the camera 3. The projection of the projector 2 and the shooting of the camera 3 are performed in synchronization.

The correction deviation detection unit 116 determines the correction deviation based on the result of the corresponding point detection performed by the corresponding point detecting unit 112, for example, using the pattern embedded in the image projected by the projector 2 and the pattern reflected in the camera image. To detect. The correction deviation detection unit 116 identifies the device that caused the correction deviation, and triggers the device to start re-correction.

The occurrence of correction deviation may be manually input by the user who is the administrator of the projector / camera system. In this case, the user calculates a change in the overlap region where the projected images overlap when a reference image such as a crosshatch or a coding pattern for sensing is projected, or a change in the corresponding point.

The correction deviation detection unit 116 detects the correction deviation based on the calculation result by the user, identifies the device that caused the correction deviation, and uses it as a trigger to start the re-correction. The device that caused the correction deviation may be input by the user.

<Corresponding point information update processing>
The corresponding point information update process performed in step S33 of FIG. 11 will be described with reference to the flowchart of FIG.

In step S51, the re-correction pattern generation unit 117 reads out from the correction information storage unit 114 and acquires the corresponding point information before the occurrence of the correction deviation, which is obtained by the initial correction. Further, the update unit 118 reads out the corresponding point information and the parameter estimated value before the correction deviation occurs from the correction information storage unit 114 and acquires them.

In step S52, the re-correction pattern generation unit 117 identifies the projector 2 and the camera 3 that caused the correction deviation based on the information supplied from the correction deviation detection unit 116 and the like. The projector 2 that causes the correction deviation is set as the target projector. Further, the camera 3 that causes the correction deviation is set as the target camera.

For example, when the correction deviation occurs due to the displacement of the projector 2-2 as described with reference to FIG. 13, the projector 2-2 is set as the target projector. Further, the camera 3-1 and the camera 3-2, which capture the image projected by the projector 2-2 as the target projector, are set as the target cameras, respectively.

In step S53, the re-correction pattern generation unit 117 generates a re-correction pattern for each target projector. Here, for example, one recorrection pattern is generated for each target projector. A plurality of re-correction patterns may be generated for each target projector.

The re-correction pattern is generated by specifying the position of the projector corresponding point based on the corresponding point information before the correction deviation occurs and arranging dots of a predetermined pattern based on the specified position.

FIG. 15 is a diagram showing an example of a recorrection pattern.

As shown in FIG. 15, each dot has a circular shape having a predetermined diameter, and is arranged so that, for example, the position of the projector corresponding point specified based on the corresponding point information is the center of gravity position. Will be done. When the corresponding point information is obtained as described with reference to FIG. 7, the position of the center of gravity of each dot constituting the recorrection pattern is the position of each of the pixels P1 to PN on the projector image.

In this way, when projected from the projector 2, a sparse pattern in which the corresponding points that are feature points can be detected based on the camera image is generated as a recorrection pattern.

Instead of arranging dots over the entire image, as shown on the right side of FIG. 16, only dots included in the range projected on the screen 11 may be arranged. The range surrounded by the ellipse # 1 in FIG. 16 is the range actually projected on the screen 11.

The range projected on the screen 11 in the entire projector image is specified, for example, based on the shape of the screen 11 estimated at the time of initial correction. The shape of the screen 11 is estimated at the time of geometric correction in the initial correction based on the parameter estimated value calculated by the posture estimation.

This makes it possible to use only the dots actually projected on the screen 11 for updating the corresponding point information.

In the initial correction, since the shape and position of the screen 11 are unknown, the range actually projected on the screen 11 in the entire projector image is unknown. Therefore, it is necessary to use a pattern for the entire projector image as the coding pattern for sensing to be projected at the time of initial correction.

This means that the sensing points that are not projected on the screen 11 may be the target of the corresponding point detection, etc., which causes a decrease in the correction accuracy. Further, depending on the shape of the screen 11, the projected pattern is greatly distorted, and high-precision sensing cannot be performed, so that the correction accuracy may decrease. Sensing points that adversely affect the final geometric correction need to be removed as outliers when decoding the coding pattern or estimating the attitude.

By using the re-correction pattern consisting of the dots actually projected on the screen 11, the corresponding point information can be updated using only the effective dots, and it is possible to prevent the correction accuracy from being lowered. In addition, since processing such as outlier removal is not required, efficient processing is possible.

Returning to the description of FIG. 14, in step S54, the projection control unit 21 controls the target projector and projects the recorrection pattern.

In step S55, the shooting control unit 22 controls the target camera to shoot the recorrection pattern projected on the screen 11. The camera image obtained by photographing the recorrection pattern projected on the screen 11 is supplied to the update unit 118.

In step S56, the shooting control unit 22 determines whether or not the recorrection pattern has been shot by all the target cameras, and switches the target cameras until it is determined that the recorrection pattern has been shot by all the target cameras, and the process of step S55. repeat.

If it is determined in step S56 that the pattern for re-correction has been captured by all the target cameras, the process proceeds to step S57.

In step S57, the projection control unit 21 determines whether or not the recorrection pattern has been projected from all the target projectors, and switches the target projectors until it is determined that the projection has been projected from all the target projectors. Repeat the process.

If it is determined in step S57 that the recorrection pattern is projected from all the target projectors, the process proceeds to step S58.

In step S58, the update unit 118 detects the position of each camera corresponding point after the correction deviation occurs, based on the camera image supplied from the shooting control unit 22.

On the camera image obtained by shooting the pattern for re-correction, the position of the camera corresponding point after the correction deviation occurs is represented as the position of the center of gravity of the dots reflected in the camera image. The update unit 118 detects the position of the center of gravity of the dots arranged at the positions corresponding to the respective cameras after the correction deviation occurs.

In step S59, the updating unit 118 updates the position information of the camera corresponding points included in the corresponding point information before the correction deviation occurs, based on the detection result of the position after the correction deviation of each camera corresponding point occurs.

FIG. 17 is a diagram showing an example of a camera image in which a pattern for re-correction is captured.

When the same pattern as the recorrection pattern is projected from the projector 2 before the correction deviation occurs and the camera 3 captures the same pattern, a camera image showing the same pattern as the recorrection pattern as shown on the left side of FIG. 17 appears. can get. Since the re-correction pattern is generated after the correction deviation occurs, the camera image as shown on the left side of FIG. 17 is not actually taken.

Further, after the correction deviation occurs, when the re-correction pattern is projected from the projector 2 which is the cause of the correction deviation and the re-correction pattern is photographed by the camera 3, the camera showing the re-correction pattern as shown on the right side of FIG. An image is obtained.

For example, when paying attention to one dot D at the right end, the position of the dot D after the correction deviation occurs is shifted from the position before the correction deviation occurs in the direction indicated by the white arrow.

FIG. 18 is a diagram schematically showing the deviation of the dot D.

In FIG. 18, the center of the circle indicated by the broken line represents the position of the dot D before the correction deviation occurs, that is, the position of the camera corresponding point before the correction deviation occurs. The circle shown by the solid line represents the position of the dot D after the correction deviation occurs.

For example, the update unit 118 performs the nearest neighbor search based on the position of the camera corresponding point before the correction deviation occurs for the camera image in which the recorrection pattern appears, so that the camera corresponding point after the correction deviation occurs. Identify the dots placed at the position of.

The update unit 118 updates the position information of the camera correspondence point included in the correspondence point information by setting the position (center of gravity position) of each dot after the correction deviation occurs as a new position of the camera correspondence point. ..

In this way, the update of the position information of the camera corresponding point included in the corresponding point information is performed by performing the nearest neighbor search based on the position of the camera corresponding point before the correction deviation occurs, and the position after the correction deviation of the camera corresponding point occurs. It is done so as to track. The slight correction deviation that triggers the re-correction is, for example, a deviation that makes it possible to trace the position of the camera corresponding point before the correction deviation occurs by the nearest neighbor search, that is, the position after the correction deviation of a certain camera corresponding point occurs. However, the deviation of the adjacent camera corresponding points does not exceed the position before the correction deviation occurs.

FIG. 19 is a diagram showing an example of updating the corresponding point information.

The projector correspondence points shown on the left side of FIG. 19 are the same as those described with reference to FIG. 7. In the example of FIG. 19, the camera correspondence point corresponding to the pixel P1 as the projector correspondence point is from the pixel p1 (FIG. 7) at the coordinates (x1, y1) on the camera image to the coordinates (x'1, y'. It has been updated as pixel p'1 in 1).

Further, the camera corresponding point corresponding to the pixel P2 as the projector corresponding point is changed from the pixel p2 at the coordinates (x2, y2) on the camera image to the pixel p'2 at the coordinates (x'2, y'2). It has been updated. Other camera correspondence points are also updated as pixels p'3 to p'n, respectively.

After the corresponding point information is updated as described above, the process returns to step S33 in FIG. 11 and the subsequent processing is performed. The updated correspondence point information is supplied to and stored in the correction information storage unit 114.

In this way, in the re-correction, the uniqueness resolution by the coding / decoding process using the time-direction coding pattern such as the Gray code is completed at the time of the initial correction before the correction deviation occurs. It is not necessary to project the number of patterns according to the number of conversion bits. The uniqueness solution means to identify the correspondence between the feature points on the projector image and the feature points on the camera image.

This makes it possible to reduce the number of patterns projected during re-correction.

<Parameter estimation value update processing>
The parameter estimation value update process performed in step S34 of FIG. 11 will be described with reference to the flowchart of FIG.

The parameter estimated value is updated using the updated corresponding point information and the parameter estimated value before the correction deviation occurs. Based on the updated correspondence point information, it is possible to estimate the posture of each device after the correction deviation occurs.

In step S71, the update unit 118 evaluates the deviation amount of the correction deviation by determining whether or not the amount is less than the threshold value.

When it is determined in step S71 that the deviation amount of the correction deviation is larger than the threshold value, the update unit 118 estimates the posture of the target camera based on the updated corresponding point information in step S72. Here, the posture estimation is performed only for the camera 3 that causes the correction deviation.

In step S73, the update unit 118 estimates the attitude of the target projector based on the updated correspondence point information and the target camera parameter estimated value calculated by the attitude estimation of the target camera. Here, the posture estimation is performed only for the projector 2 that causes the correction deviation.

In this way, it is possible to reduce the processing required for re-correction by estimating the posture only for the device that caused the correction deviation. As for the parameter estimation value of the device that does not cause the correction deviation, the value before the correction deviation occurs is used as it is.

In step S74, the update unit 118 optimizes the target camera parameter estimated value calculated by the posture estimation of the target camera and the target projector parameter estimated value calculated by the posture estimation of the target projector. The optimization of the parameter estimated value is the same as the optimization performed in the attitude estimation at the time of initial correction.

On the other hand, if it is determined in step S71 that the deviation amount of the correction deviation is less than the threshold amount, the posture estimation in steps S72 and S73 is skipped, and in step S74, the parameter estimation value before the correction deviation occurs is optimized. Will be done.

When the deviation amount of the correction deviation is small, it can be said that the parameter estimated value calculated by the initial correction has a certain accuracy even after the correction deviation occurs. The update unit 118 updates the parameter estimated value before the occurrence of the correction deviation by the parameter estimated value obtained by the optimization performed according to the deviation amount of the correction deviation.

In this way, even if the correction deviation occurs, when the parameter estimation value is considered to have a certain accuracy, the posture estimation is not performed, and the optimization, which can be said to be a fine adjustment, is performed before the correction deviation occurs. It is possible to complete the update of the parameter estimates.

After the parameter estimated value is optimized in step S74, the process returns to step S34 in FIG. 11 and the subsequent processing is performed.

<Geometric correction>
After the parameter estimates of the projector 2 and the camera 3 are obtained by posture estimation, the projector-compatible points and the camera-compatible points can be calculated as points in the three-dimensional space by the principle of triangulation. The positions of the projector corresponding points and the camera corresponding points in the three-dimensional space are specified by the parameter estimated values (internal parameters, external parameters) calculated by the parameter estimated value updating process as described above.

That is, it is possible to restore each point on the screen 11 with the same density as the corresponding points. By applying the calculated point cloud to a model of a plane or a quadric surface representing the screen 11, the shape of the entire screen 11 can be estimated.

Since it is possible to generate a geometric correction vector for each pixel indicating which position each pixel of the projector image corresponds to on the screen 11, it is possible to realize geometric correction in consideration of the shape of the screen 11. It will be possible.

By performing the re-correction as described above, the corresponding point information due to the correction deviation caused by physical contact or aging is updated with the sensing pattern generated based on the information before the correction deviation occurs. It becomes possible to perform it efficiently by using a certain recorrection pattern.

Further, by using the re-correction pattern generated based on the information before the correction deviation occurs, it is possible to omit the decoding process of the coding pattern and reduce the total number of projected images and the number of shots for re-correction. It becomes. The number of patterns that need to be projected can be the same as the number of projectors 2 that have caused the correction deviation. Even when the number of projectors 2 increases, it is possible to suppress an increase in the number of projections and the number of shots required for re-correction.

In this way, since it is possible to update only the position of the corresponding point while diverting the uniqueness identified before the occurrence of the correction deviation, the coding / decoding process is omitted, and the corresponding point information can be obtained by simple processing. It will be possible to update.

Furthermore, by using the updated correspondence point information, it is possible to easily and quickly update the parameter estimated value before the correction deviation occurs.

<Modification example>
-Example of pattern modification In the sensing in this technology, "Spatial coding by projecting a pattern such as Gray code in time series" and "The corresponding points to be acquired are not dense and are projected. It is premised that it can be detected as a sparse feature point in the area. "

In the above example, the dot pattern is projected as a pattern for re-correction, but what kind of pattern is used for re-correction as long as it is a sparse pattern that can obtain a corresponding point with sufficient accuracy for posture estimation? Pattern may be used.

For example, when the distortion of the screen is small, it is possible to use a checker pattern as a re-correction pattern that detects the projector corresponding point before the correction deviation occurs as a corner. Further, an equilateral triangle tiling pattern or a ChAruCo marker may be used as a recorrection pattern.

The re-correction pattern may be generated for each combination of the projector that caused the correction deviation and the camera that captures the image projected by the projector.

In a system composed of a plurality of projectors and cameras, it is possible that pixels of a plurality of camera images are associated with pixels of one projector image.

In the above example, when the pattern for re-correction is generated, one pattern is generated for each projector that causes the correction deviation. At this time, the camera image of one camera includes corresponding points corresponding to the pixels of the camera image of another camera. When tracking the position of the corresponding point after the correction deviation occurs, by searching for a point near the corresponding point before the correction deviation occurs for each camera 3, there is a possibility that an erroneous correspondence with such a point is taken. Although it can be eliminated, it is a little redundant as a corresponding point update process.

Correspondence points corresponding to the pixels of the camera image of another camera by using one re-correction pattern for each combination of the projector that caused the correction deviation and the camera that captures the image projected by the projector. (Because it is not necessary to include it as a dot of the recorrection pattern), it is possible to efficiently update the corresponding point. At this time, the number of patterns generated is (number of projectors) × (number of cameras), but there is no change in the total number of shots.

-Modified example of device configuration FIG. 21 is a diagram showing another configuration example of the projector / camera system.

The above re-correction process can also be applied to a system in which three or more projectors 2 are provided as shown in FIG.

In the projector / camera system shown in FIG. 21, four projectors 2 of projectors 2-1 to 2-4 are provided. Further, four cameras 3 of cameras 3-1 to 3-4 are provided.

One projector 2 such that the projector 2-1 and the camera 3-1 are provided in the projection / photographing device 211-1 and the projector 2-2 and the camera 3-2 are provided in the projection / photographing device 201-2. And one camera 3 are provided as a configuration of a device having the same housing.

Similarly, the projector 2-3 and the camera 3-3 are provided in the projection / photographing device 201-3, and the projector 2-4 and the camera 3-4 are provided in the projection / photographing device 201-4.

In this way, three or more projectors 2 may be provided, or the projector 2 and the camera 3 may be provided in the same housing.

The frames # 1 to # 4 shown on the screen 11 represent the projection areas of the projectors 2-1 to 2-4, respectively. The re-correction process can also be applied when one composite image is projected using such four projectors 2. The overlap region may be formed by a combination of projection regions of any projector 2.

FIG. 22 is a diagram showing still another configuration example of the projector / camera system.

As shown in FIG. 22, the projector 2 may be configured by a fisheye projector which is a projector equipped with a fisheye lens, and the camera 3 may be configured by a fisheye camera which is a camera equipped with a fisheye lens.

Further, as shown in FIG. 22, the screen 11 may be a dome-shaped screen whose projection surface is a hemispherical surface.

-About the program The series of processes described above can be executed by hardware or software. When a series of processes are executed by software, the programs constituting the software are installed from the program recording medium on a computer embedded in dedicated hardware or a general-purpose personal computer.

The program to be installed is provided by being recorded on the removable media 65 shown in FIG. 9, which consists of an optical disk (CD-ROM (Compact Disc-Read Only Memory), DVD (Digital Versatile Disc), etc.) or a semiconductor memory. It may also be provided via a wired or wireless transmission medium such as a local area network, the Internet, or digital broadcasting. The program can be installed in ROM 52 or HDD 64 in advance.

The program executed by the computer may be a program in which processing is performed in chronological order in the order described in this specification, or processing is performed in parallel or at a necessary timing such as when a call is made. It may be a program to be performed.

In the present specification, the system means a set of a plurality of components (devices, modules (parts), etc.), and it does not matter whether all the components are in the same housing. Therefore, a plurality of devices housed in separate housings and connected via a network, and a device in which a plurality of modules are housed in one housing are both systems. ..

The effects described in this specification are merely examples and are not limited, and other effects may be obtained.

The embodiment of the present technology is not limited to the above-described embodiment, and various changes can be made without departing from the gist of the present technology.

For example, this technology can have a cloud computing configuration in which one function is shared by a plurality of devices via a network and processed jointly.

Further, each step described in the above flowchart can be executed by one device or can be shared and executed by a plurality of devices.

Further, when one step includes a plurality of processes, the plurality of processes included in the one step can be executed by one device or shared by a plurality of devices.

<Example of configuration combination>
The present technology can also have the following configurations.

(1)
Correspondence point detection that detects corresponding points on each image based on the projected image projected by the projection device and the captured image obtained by photographing the projected projected image by the photographing device, and the projection device A detection unit that detects the deviation of the corresponding points that occurs after the initial correction including the posture estimation that estimates the posture of the photographing device based on the result of the corresponding point detection.
A re-correction pattern composed of a predetermined pattern based on the position of the corresponding point on the projected image based on the corresponding point information which is the information indicating the relationship of the corresponding point obtained by the initial correction. And the generator that generates
Based on the photographed image obtained by photographing the recorrection pattern projected from the projection device by the photographing device, the deviation of the corresponding point on the photographed image is detected, and the corresponding point information is updated. An image processing device including an update unit.
(2)
The image processing apparatus according to (1), wherein the generation unit generates the recorrection pattern in which dots are arranged at the positions of the corresponding points on the projected image.
(3)
The image according to (1) above, wherein the generation unit is a position of the corresponding point on the projected image, and generates the recorrection pattern in which dots are arranged at positions within a range projected on the projection surface. Processing equipment.
(4)
The image processing device according to any one of (1) to (3), wherein the generation unit generates one recorrection pattern for each projection device that causes the deviation of the corresponding points.
(5)
The image processing device according to any one of (1) to (4), wherein the updating unit updates the corresponding point information based on the deviation of the corresponding points on the captured image.
(6)
The image processing according to any one of (1) to (5) above, wherein the updating unit updates the parameter estimated value obtained by the posture estimation at the time of the initial correction based on the updated corresponding point information. apparatus.
(7)
When the amount of deviation of the corresponding points is larger than the amount that becomes the threshold value, the updating unit updates the parameter estimated value by estimating the postures of the projection device and the photographing device that caused the deviation of the corresponding points. The image processing apparatus according to (6) above.
(8)
When the amount of deviation of the corresponding point is less than the amount that becomes the threshold value, the updating unit estimates the parameter by adjusting the estimated value of the parameter before the deviation of the corresponding point occurs according to the deviation of the corresponding point. The image processing apparatus according to (6) above, wherein the value is updated.
(9)
A projection control unit that controls projection by the projection device,
The image processing device according to any one of (1) to (8) above, further comprising a shooting control unit that controls shooting by the shooting device.
(10)
The projection control unit controls projection by the plurality of projection devices, and controls projection.
The image processing device according to (9) above, wherein the imaging control unit controls imaging by a plurality of the imaging devices.
(11)
The projection control unit projects the recorrection pattern from the projection device that causes the deviation of the corresponding points.
The imaging control unit causes the projected recorrection pattern to be photographed by the imaging device that captures the projected image projected by the projection device that projects the recorrection pattern (9) or (10). ). The image processing apparatus.
(12)
The image processing apparatus according to any one of (1) to (11) above, wherein the projection surface of the projected image is a flat screen, a curved screen, or a spherical screen.
(13)
The corresponding point detection at the time of the initial correction uses the projected image representing a plurality of predetermined patterns and the plurality of captured images obtained by photographing each of the projected predetermined patterns. The image processing apparatus according to any one of (1) to (12) above.
(14)
When the deviation amount of the corresponding point is smaller than the threshold value, the corresponding point information is updated using the recorrection pattern, and when the deviation amount of the corresponding point is larger than the threshold value, the initial stage The image processing apparatus according to any one of (1) to (13) above, further comprising a control unit that performs the same processing as the correction.
(15)
The image processing device
Correspondence point detection that detects corresponding points on each image based on the projected image projected by the projection device and the captured image obtained by photographing the projected projected image by the photographing device, and the projection device The deviation of the corresponding points generated after the initial correction including the posture estimation for estimating the posture of the photographing device based on the result of the corresponding point detection is detected.
A re-correction pattern composed of a predetermined pattern based on the position of the corresponding point on the projected image based on the corresponding point information which is the information indicating the relationship of the corresponding point obtained by the initial correction. To generate
Based on the photographed image obtained by photographing the recorrection pattern projected from the projection device by the photographing device, the deviation of the corresponding point on the photographed image is detected, and the corresponding point information is updated. Image processing method to be performed.
(16)
On the computer
Correspondence point detection that detects corresponding points on each image based on the projected image projected by the projection device and the captured image obtained by photographing the projected projected image by the photographing device, and the projection device The deviation of the corresponding points generated after the initial correction including the posture estimation for estimating the posture of the photographing device based on the result of the corresponding point detection is detected.
A re-correction pattern composed of a predetermined pattern based on the position of the corresponding point on the projected image based on the corresponding point information which is the information indicating the relationship of the corresponding point obtained by the initial correction. To generate
Based on the photographed image obtained by photographing the recorrection pattern projected from the projection device by the photographing device, the deviation of the corresponding point on the photographed image is detected, and the corresponding point information is updated. A program for executing the processing to be performed.
(17)
Correspondence point detection that detects corresponding points on each image based on the projected image projected by the projection device and the captured image obtained by photographing the projected projected image by the photographing device, and the projection device A detection unit that detects the deviation of the corresponding points that occurs after the initial correction including the posture estimation that estimates the posture of the photographing device based on the result of the corresponding point detection.
A re-correction pattern composed of a predetermined pattern based on the position of the corresponding point on the projected image based on the corresponding point information which is the information indicating the relationship of the corresponding point obtained by the initial correction. And the generator that generates
Based on the photographed image obtained by photographing the recorrection pattern projected from the projection device by the photographing device, the deviation of the corresponding point on the photographed image is detected, and the corresponding point information is updated. An image processing device equipped with an update unit
With the plurality of the projection devices
An image processing system configured to include a plurality of the photographing devices.

1 Image processing device, 2-1 and 2-2 projector, 3-1, 3-2 camera, 21 projection control unit, 22 shooting control unit, 101 control unit, 111 coding pattern generation unit, 112 corresponding point detection unit, 113 Posture estimation unit, 114 Correction information storage unit, 115 Geometric correction unit, 116 Correction deviation detection unit, 117 Recorrection pattern generation unit, 118 Update unit

Claims (17)

1. Correspondence point detection that detects corresponding points on each image based on the projected image projected by the projection device and the captured image obtained by photographing the projected projected image by the photographing device, and the projection device A detection unit that detects the deviation of the corresponding points that occurs after the initial correction including the posture estimation that estimates the posture of the photographing device based on the result of the corresponding point detection.
   A re-correction pattern composed of a predetermined pattern based on the position of the corresponding point on the projected image based on the corresponding point information which is the information indicating the relationship of the corresponding point obtained by the initial correction. And the generator that generates
   Based on the photographed image obtained by photographing the recorrection pattern projected from the projection device by the photographing device, the deviation of the corresponding point on the photographed image is detected, and the corresponding point information is updated. An image processing device including an update unit.
2. The image processing apparatus according to claim 1, wherein the generation unit generates the recorrection pattern in which dots are arranged at the positions of the corresponding points on the projected image.
3. The image processing according to claim 1, wherein the generation unit is a position of the corresponding point on the projected image, and generates the recorrection pattern in which dots are arranged at positions within a range projected on the projection surface. apparatus.
4. The image processing device according to claim 1, wherein the generation unit generates one recorrection pattern for each projection device that causes the deviation of the corresponding points.
5. The image processing device according to claim 1, wherein the updating unit updates the corresponding point information based on the deviation of the corresponding points on the captured image.
6. The image processing device according to claim 1, wherein the updating unit updates the parameter estimated value obtained by the posture estimation at the time of the initial correction based on the updated corresponding point information.
7. When the amount of deviation of the corresponding points is larger than the amount that becomes the threshold value, the updating unit updates the parameter estimated value by estimating the postures of the projection device and the photographing device that caused the deviation of the corresponding points. The image processing apparatus according to claim 6.
8. When the amount of deviation of the corresponding point is less than the amount that becomes the threshold value, the updating unit estimates the parameter by adjusting the estimated value of the parameter before the deviation of the corresponding point occurs according to the deviation of the corresponding point. The image processing apparatus according to claim 6, wherein the value is updated.
9. A projection control unit that controls projection by the projection device,
   The image processing device according to claim 1, further comprising a shooting control unit that controls shooting by the shooting device.
10. The projection control unit controls projection by the plurality of projection devices, and controls projection.
    The image processing device according to claim 9, wherein the imaging control unit controls imaging by the plurality of imaging devices.
11. The projection control unit projects the recorrection pattern from the projection device that causes the deviation of the corresponding points.
    The image according to claim 9, wherein the imaging control unit captures the projected recorrection pattern by the imaging device that captures the projected image projected by the projection device that projects the recorrection pattern. Processing equipment.
12. The image processing apparatus according to claim 1, wherein the projection surface of the projected image is a flat screen, a curved screen, or a spherical screen.
13. The corresponding point detection at the time of the initial correction uses the projected image representing a plurality of predetermined patterns and the plurality of captured images obtained by photographing each of the projected predetermined patterns. The image processing apparatus according to claim 1.
14. When the deviation amount of the corresponding point is smaller than the threshold value, the corresponding point information is updated using the recorrection pattern, and when the deviation amount of the corresponding point is larger than the threshold value, the initial stage The image processing apparatus according to claim 1, further comprising a control unit that performs the same processing as the correction.
15. The image processing device
    Correspondence point detection that detects corresponding points on each image based on the projected image projected by the projection device and the captured image obtained by photographing the projected projected image by the photographing device, and the projection device The deviation of the corresponding points generated after the initial correction including the posture estimation for estimating the posture of the photographing device based on the result of the corresponding point detection is detected.
    A re-correction pattern composed of a predetermined pattern based on the position of the corresponding point on the projected image based on the corresponding point information which is the information indicating the relationship of the corresponding point obtained by the initial correction. To generate
    Based on the photographed image obtained by photographing the recorrection pattern projected from the projection device by the photographing device, the deviation of the corresponding point on the photographed image is detected, and the corresponding point information is updated. Image processing method to be performed.
16. On the computer
    Correspondence point detection that detects corresponding points on each image based on the projected image projected by the projection device and the captured image obtained by photographing the projected projected image by the photographing device, and the projection device The deviation of the corresponding points generated after the initial correction including the posture estimation for estimating the posture of the photographing device based on the result of the corresponding point detection is detected.
    A re-correction pattern composed of a predetermined pattern based on the position of the corresponding point on the projected image based on the corresponding point information which is the information indicating the relationship of the corresponding point obtained by the initial correction. To generate
    Based on the photographed image obtained by photographing the recorrection pattern projected from the projection device by the photographing device, the deviation of the corresponding point on the photographed image is detected, and the corresponding point information is updated. A program for executing the processing to be performed.
17. Correspondence point detection that detects corresponding points on each image based on the projected image projected by the projection device and the captured image obtained by photographing the projected projected image by the photographing device, and the projection device A detection unit that detects the deviation of the corresponding points that occurs after the initial correction including the posture estimation that estimates the posture of the photographing device based on the result of the corresponding point detection.
    A re-correction pattern composed of a predetermined pattern based on the position of the corresponding point on the projected image based on the corresponding point information which is the information indicating the relationship of the corresponding point obtained by the initial correction. And the generator that generates
    Based on the photographed image obtained by photographing the recorrection pattern projected from the projection device by the photographing device, the deviation of the corresponding point on the photographed image is detected, and the corresponding point information is updated. An image processing device equipped with an update unit
    With the plurality of the projection devices
    An image processing system configured to include a plurality of the photographing devices.

Images