WO2022044806A1

WO2022044806A1 - Information processing device and method

Info

Publication number: WO2022044806A1
Application number: PCT/JP2021/029625
Authority: WO
Inventors: 直樹小林; 清登染谷
Original assignee: ソニーグループ株式会社
Priority date: 2020-08-24
Filing date: 2021-08-11
Publication date: 2022-03-03

Abstract

The present disclosure relates to an information processing device and method which make it possible to suppress a reduction in projection correction accuracy. A three-dimensional point position, which is projection position, in a three-dimensional space, corresponding to a pixel of interest of a projecting unit that projects an input image, is projected onto a light receiving region of a virtual imaging unit, which is a virtual imaging unit for imaging a projected image projected by the projecting unit, using a position and attitude such that the three-dimensional point position is the viewpoint of the projected image, and a correction vector corresponding to the pixel of interest is derived by converting a projection point, indicating the position at which the three-dimensional point position has been projected onto the light receiving region, into the coordinate system of the input image. The present disclosure is applicable, for example, to information processing devices, projecting devices, imaging devices, projection/imaging devices, projection/imaging control devices, and image projection/imaging systems.

Description

Information processing equipment and methods

The present disclosure relates to an information processing device and a method, and more particularly to an information processing device and a method capable of suppressing a decrease in the accuracy of projection correction.

Conventionally, there was a system that projects an image using multiple projectors. In such a system, one image can be projected by appropriately connecting the projected images of the respective projectors. Therefore, the projected image is sensed by a camera or the like, and the image projected by each projector is corrected by using the sensing result (see, for example, Patent Document 1).

Japanese Unexamined Patent Publication No. 2016-14720

However, in this method, the projector, camera, and screen are modeled and estimated, so if the accuracy of the model estimation is low, the accuracy of the final corrected image may decrease.

This disclosure is made in view of such a situation, and makes it possible to suppress a decrease in the accuracy of projection correction.

The information processing apparatus on one aspect of the present technology sets the three-dimensional point position, which is the projection position on the three-dimensional space corresponding to the pixel of interest of the projection unit that projects the input image, to the viewpoint of the projection image projected by the projection unit. The input points are projected onto the light receiving area of the virtual imaging unit, which is a virtual imaging unit that captures the projected image at the position and orientation, and the projection points indicating the positions where the three-dimensional point positions in the light receiving region are projected are input. It is an information processing apparatus including a correction vector derivation unit that derives a correction vector corresponding to the pixel of interest by converting it into an image coordinate system.

In the information processing method of one aspect of the present technology, the three-dimensional point position, which is the projection position in the three-dimensional space corresponding to the pixel of interest of the projection unit that projects the input image, is the viewpoint of the projection image projected by the projection unit. The input points are projected onto the light receiving area of the virtual imaging unit, which is a virtual imaging unit that captures the projected image at the position and orientation, and the projection points indicating the positions where the three-dimensional point positions in the light receiving region are projected are input. This is an information processing method for deriving a correction vector corresponding to the pixel of interest by converting to the coordinate system of an image.

In the information processing apparatus and method of one aspect of the present technology, the 3D point position, which is the projection position in the 3D space corresponding to the pixel of interest of the projection unit that projects the input image, is the projection projected by the projection unit. A projection point that is projected onto the light receiving area of the virtual imaging unit, which is a virtual imaging unit that captures the projected image at the position and orientation as the viewpoint of the image, and the three-dimensional point position in the light receiving region is projected. , The correction vector corresponding to the pixel of interest is derived by converting to the coordinate system of the input image.

It is a figure explaining the example of the state of the projection correction. It is a block diagram which shows the main configuration example of a projection imaging system. It is a block diagram which shows the main configuration example of a portable terminal apparatus. It is a functional block diagram which shows the example of the main function realized by an information processing unit. It is a block diagram which shows the main configuration example of a projector. It is a functional block diagram which shows the example of the main function realized by an information processing unit. It is a flowchart explaining the example of the flow of a projection correction process. It is a figure explaining the correspondence point detection. It is a figure explaining the correspondence point detection. It is a figure explaining the camera posture estimation. It is a figure explaining the camera posture estimation. It is a figure explaining the camera posture estimation. It is a figure explaining the reconstruction of a flat screen. It is a figure explaining the reconstruction of a flat screen. It is a figure explaining the correction vector derivation. It is a figure explaining the correction vector derivation. It is a figure explaining the correction vector derivation. It is a figure explaining the correction vector derivation. It is a figure explaining the projection area adjustment. It is a figure explaining the projection area adjustment. It is a figure explaining the reconstruction of a cylindrical surface screen. It is a figure explaining the reconstruction of a cylindrical surface screen. It is a figure explaining the correction vector derivation. It is a figure explaining the correction vector derivation. It is a figure explaining the correction vector derivation. It is a figure explaining the reconstruction of a spherical screen. It is a figure explaining the reconstruction of a spherical screen. It is a figure explaining the correction vector derivation. It is a figure explaining the correction vector derivation. It is a figure explaining the correction vector derivation.

Hereinafter, embodiments for carrying out the present disclosure (hereinafter referred to as embodiments) will be described. The explanation will be given in the following order.
1. 1. Projection correction 2. First Embodiment (in the case of a flat screen)
3. 3. Second embodiment (in the case of a cylindrical screen)
4. Third embodiment (in the case of a spherical screen)
5. Addendum

<1. Projection correction>
<Correction accuracy>
Conventional general projection imaging systems (for example, Flexible Display Technologies) do not perform three-dimensional position / orientation / shape estimation, and in order to automatically correct the projection, a camera or a camera is placed at a position facing the screen. The projector had to be fixedly placed. Further, when adjusting the size, position, inclination, etc. of the image after projection correction, it is necessary for the coordinator to correct each of the positions of the four corners of the projected image as shown in A of FIG. Since the stone adjustment is also performed visually at the same time, not only complicated work is required, but also the correction accuracy may depend on the ability of the coordinator.

Further, in the case of the method described in Patent Document 1, since the projector, the camera, and the screen are modeled and estimated, the correction accuracy may depend on the accuracy of the model. Therefore, if the accuracy of the model estimation is low, the accuracy of the final corrected image may be reduced.

<Derivation of correction vector using 3D point position>
Therefore, the projected image is the position and orientation of the three-dimensional point position, which is the projection position on the three-dimensional space corresponding to the pixel of interest of the projection unit that projects the input image, as the viewpoint of the projection image projected by the projection unit. By projecting onto the light receiving area of the virtual imaging unit, which is a virtual imaging unit that captures images, and converting the projection point indicating the position where the three-dimensional point position in the light receiving area is projected into the coordinate system of the input image. , A correction vector corresponding to the target pixel is derived.

For example, in an information processing apparatus, a position in which a three-dimensional point position, which is a projection position in a three-dimensional space corresponding to a pixel of interest of a projection unit that projects an input image, is used as a viewpoint of a projection image projected by the projection unit and a position The coordinate system of the input image is a projection point that is projected onto the light receiving area of the virtual imaging unit, which is a virtual imaging unit that captures the projected image in the posture, and the projection point indicating the position where the three-dimensional point position in the light receiving region is projected. By converting to, a correction vector derivation unit for deriving a correction vector corresponding to the pixel of interest is provided.

By doing so, projection correction becomes possible without the need for modeling the projection unit. That is, it is possible to perform projection correction with an accuracy that does not depend on the accuracy of the model. That is, it is possible to suppress a decrease in the accuracy of projection correction. Further, for example, as shown in B of FIG. 1, adjustment can be made only by the zoom shift roll while holding the corrected image. That is, the user does not need to consider and adjust the keystone correction. That is, it is possible to suppress a decrease in the accuracy of projection correction.

<2. First Embodiment>
<Projection imaging system>
FIG. 2 is a block diagram showing a main configuration example of a projection imaging system which is an embodiment of an information processing system to which the present technology is applied. In FIG. 2, the projection imaging system 100 includes a portable terminal device 101, a projector 102-1, and a projector 102-2, and is a system that projects an image on a screen 120 or images a screen 120.

The portable terminal device 101, the projector 102-1, and the projector 102-2 are connected to each other so as to be able to communicate with each other via the communication path 110. The communication path 110 is arbitrary and may be wired or wireless. For example, the portable terminal device 101, the projector 102-1, and the projector 102-2 can send and receive control signals, image data, and the like via the communication path 110.

The portable terminal device 101 is a user-portable device such as a smartphone, a tablet terminal, a notebook personal computer, or the like. The portable terminal device 101 has a communication function, an information processing function, and an image pickup function. For example, the portable terminal device 101 can control the image projection by the projector 102-1 and the projector 102-2. Further, the portable terminal device 101 can perform projection correction of the projector 102-1 and the projector 102-2. Further, the portable terminal device 101 can capture a projected image projected on the screen 120 by the projector 102-1 or the projector 102-2.

Projector 102-1 and projector 102-2 are projection devices that project images. The projector 102-1 and the projector 102-2 are similar devices to each other. In the following, when it is not necessary to distinguish the projector 102-1 and the projector 102-2 from each other, the projector 102 is referred to as a projector 102. For example, the projector 102 can project an input image onto the screen 120 under the control of the portable terminal device 101.

Projector 102-1 and projector 102-2 can project images in cooperation with each other. For example, the projector 102-1 and the projector 102-2 can project an image at the same position with each other to realize high brightness of the projected image. Further, the projector 102-1 and the projector 102-2 project an image so that the projected images of each other are arranged side by side, and the two projected images form one image, and the projected image is enlarged (high resolution). Can be realized. Further, the projector 102-1 and the projector 102-2 can project an image so as to superimpose a part of each other's projected images or to include the other projected image in one projected image. By collaborating in this way to project, the projector 102-1 and projector 102-2 not only have higher brightness and larger screens, but also have, for example, higher dynamic range and higher frame rate of the projected image. Etc. can also be realized.

In such image projection, the projector 102 can geometrically correct the projected image under the control of the portable terminal device 101 so that the projected image is superimposed at the correct position.

For example, as shown in FIG. 2, the projector 102-1 projects the input image in the pixel region 121-1 with geometric correction like the corrected image 122-1. Further, the projector 102-2 projects the input image in the pixel region 121-2 with geometric correction like the corrected image 122-2.

On the screen 120, the image of the pixel area 121-1 is projected by the projector 102-1 like the projected image 123-1. Further, the image of the pixel area 121-2 is projected by the projector 102-2 like the projected image 123-2. In the portion where the projected image 123-1 and the projected image 123-2 overlap, the corrected image 122-1 and the corrected image 122-2 are not distorted at the same position as the projected image 124 (as in the projected image 124). It is projected (in a rectangular shape).

The screen 120 is, for example, a flat screen in which the projection surface is formed by a flat surface.

In such a projection imaging system 100, the portable terminal device 101 can perform projection correction of the projector 102 three-dimensionally.

Note that, in FIG. 2, the projection imaging system 100 is composed of one portable terminal device 101 and two projectors 102, but the number of each device is arbitrary and is not limited to this example. For example, the projection imaging system 100 may have a plurality of portable terminal devices 101, or may have three or more projectors 102. Further, the portable terminal device 101 may be integrally configured with any of the projectors 102.

<Portable terminal device>
FIG. 3 is a diagram showing a main configuration example of a portable terminal device 101, which is an embodiment of an information processing device to which the present technology is applied. As shown in FIG. 3, the portable terminal device 101 includes an information processing unit 151, an image pickup unit 152, an input unit 161, an output unit 162, a storage unit 163, a communication unit 164, and a drive 165.

The information processing unit 151 has, for example, a CPU (Central Processing Unit), a ROM (Read Only Memory), a RAM (Random Access Memory), and the like, and can be used to execute various application programs (software). It is a computer that can realize the function. For example, the information processing unit 151 can install and execute an application program (software) that performs processing related to projection correction. Here, the computer includes a computer embedded in dedicated hardware and, for example, a general-purpose personal computer capable of executing various functions by installing various programs.

The image pickup unit 152 has an optical system, an image sensor, and the like, and can capture an image of a subject to generate an image. The image pickup unit 152 can supply the generated captured image to the information processing unit 151.

The input unit 161 has, for example, input devices such as a keyboard, a mouse, a microphone, a touch panel, and an input terminal, and can supply information input via those input devices to the information processing unit 151.

The output unit 162 has, for example, an output device such as a display (display unit), a speaker (audio output unit), and an output terminal, and outputs information supplied from the information processing unit 151 via those output devices. Can be done.

The storage unit 163 has, for example, a storage medium such as a hard disk, a RAM disk, or a non-volatile memory, and can store the information supplied from the information processing unit 151 in the storage medium. The storage unit 163 can read out the information stored in the storage medium and supply it to the information processing unit 151.

The communication unit 164 has, for example, a network interface, can receive information transmitted from another device, and can supply the received information to the information processing unit 151. The communication unit 164 can transmit the information supplied from the information processing unit 151 to another device.

The drive 165 has an interface of a removable recording medium 171 such as a magnetic disk, an optical disk, a magneto-optical disk, or a semiconductor memory, reads information recorded on the removable recording medium 171 mounted on the drive 165, and reads information from the information processing unit 151. Can be supplied to. The drive 165 can record the information supplied from the information processing unit 151 on the writable removable recording medium 171 attached to the drive 165.

The information processing unit 151 loads and executes, for example, the application program stored in the storage unit 163. At that time, the information processing unit 151 can appropriately store data and the like necessary for executing various processes. The application program, data, and the like can be recorded and provided on a removable recording medium 171 as a package media or the like, for example. In that case, the application program, data, and the like are read out by the drive 165 equipped with the removable recording medium 171 and installed in the storage unit 163 via the information processing unit 151. The application program can also be provided via a wired or wireless transmission medium such as a local area network, the Internet, or digital satellite broadcasting. In that case, the application program, data, and the like are received by the communication unit 164 and installed in the storage unit 163 via the information processing unit 151. Further, the application program, data, and the like can be installed in advance in the ROM and the storage unit 163 in the information processing unit 151.

<Functional block of portable terminal device>
FIG. 4 shows a function realized by the information processing unit 151 executing an application program as a functional block. As shown in FIG. 4, the information processing unit 151 executes the application program to execute the corresponding point detection unit 181 and the camera posture estimation unit 182, the screen reconstruction unit 183, and the correction vector derivation unit 184 as functional blocks. It can have a projection control unit 185 and a projection area adjustment unit 186.

The corresponding point detection unit 181 detects the corresponding point for each captured image based on the captured image of the projected image projected on the screen 120. The corresponding point detection unit 181 supplies the corresponding point information indicating the detected corresponding point to the camera posture estimation unit 182.

The camera posture estimation unit 182 estimates the position and posture of the camera corresponding to the captured image (that is, the position and posture taken by the portable terminal device 101 (imaging unit 152)) based on the corresponding point information. do. Further, the camera posture estimation unit 182 derives a three-dimensional point position, which is a projection position in the three-dimensional space corresponding to the pixel of interest of the projector 102, based on the estimated position and posture of the camera. The camera posture estimation unit 182 supplies the camera posture information indicating the estimated position and posture, the three-dimensional point position, and the like to the screen reconstruction unit 183 together with the corresponding point information.

The screen reconstruction unit 183 sets a projection surface (virtual screen) on which the projector 102 projects an image based on the corresponding point information and the camera posture information. In this case, since the screen 120 is a flat screen, the screen reconstruction unit 183 sets a flat projection plane. Further, the screen reconstruction unit 183 sets a viewpoint for viewing the projected image projected on the projection surface (that is, a viewpoint corresponding to the projection surface), and a virtual view showing the view when the projected image is viewed at that viewpoint. Set the viewpoint camera (that is, the viewpoint camera corresponding to the projection plane). The screen reconstruction unit 183 generates temporary screen information including information about the projection surface set in this way and viewpoint camera information which is information about the viewpoint camera set in this way, and together with camera attitude information and the like, a correction vector. It is supplied to the out-licensing unit 184.

The correction vector derivation unit 184 derives a correction vector indicating how to correct each pixel of the input image based on the information and the like. That is, the correction vector derivation unit 184 sets the three-dimensional point position, which is the projection position on the three-dimensional space corresponding to the pixel of interest of the projection unit that projects the input image, as the viewpoint of the projection image projected by the projection unit. The projection point of the input image is projected onto the light receiving area of the virtual imaging unit, which is a virtual imaging unit that captures the projected image at the position and orientation, and the projection point indicating the position where the three-dimensional point position in the light receiving region is projected. By converting to a coordinate system, a correction vector corresponding to the pixel of interest is derived.

Further, the correction vector derivation unit 184 can also acquire the projection correction information prepared in advance indicating the method of projection correction, and derive the correction vector based on the projection correction information. The correction vector derivation unit 184 is a projection position parameter which is a parameter related to the control of the projection area (position, size, inclination, etc.) which is the area where the projection image is projected on the projection surface, which is supplied from the projection area adjustment unit 186. Can also be obtained and a correction vector can be derived based on the projection position parameter. The correction vector derivation unit 184 supplies the derived correction vector to the projection control unit 185.

The projection control unit 185 supplies the correction vector to the projector 102 to be controlled. Further, the projection control unit 185 supplies a projection instruction of the corrected image to the projector 102 to project the corrected image.

The projection area adjustment unit 186 generates, for example, a user interface (UI (User Interface)) image, supplies it to the output unit 162, and displays the UI image on the monitor. Further, the projection area adjustment unit 186 acquires the user instruction for the UI image received by the input unit 161. The projection area adjustment unit 186 controls the position, size, inclination, and the like of the projection area, which is the area on the projection surface on which the projection image is projected, based on the user's instruction. For example, the projection area adjustment unit 186 displays a UI image that accepts instructions such as shift, zoom, and roll of the projection area on the monitor, and acquires the input user instructions regarding shift, zoom, roll, and the like of the projection area. The projection area adjusting unit 186 supplies the correction vector deriving unit 184 with projection position parameters corresponding to the user's instructions, that is, projection position parameters for shifting, zooming, and rolling the projection area as instructed.

<Projector>
FIG. 5 is a diagram showing a main configuration example of the projector 102, which is an embodiment of an information processing apparatus to which the present technology is applied. As shown in FIG. 5, the projector 102 has an information processing unit 201, a projection unit 202, an input unit 211, an output unit 212, a storage unit 213, a communication unit 214, and a drive 215.

The information processing unit 201 is a computer that has, for example, a CPU, ROM, RAM, etc., and can realize various functions by executing an application program (software) using them. For example, the information processing unit 201 may install and execute an application program (software) that performs processing related to image projection. Here, the computer includes a computer embedded in dedicated hardware and, for example, a general-purpose personal computer capable of executing various functions by installing various programs.

The projection unit 202 has an optical device, a light source, and the like, and can be controlled by the information processing unit 201 to project a desired image. For example, the projection unit 202 can project an image supplied from the information processing unit 201.

The input unit 211 has, for example, input devices such as a keyboard, mouse, microphone, touch panel, and input terminal, and can supply information input via those input devices to the information processing unit 201.

The output unit 212 has, for example, an output device such as a display (display unit), a speaker (audio output unit), and an output terminal, and outputs information supplied from the information processing unit 201 via those output devices. Can be done.

The storage unit 213 has a storage medium such as a hard disk, a RAM disk, or a non-volatile memory, and can store the information supplied from the information processing unit 201 in the storage medium. The storage unit 213 can read out the information stored in the storage medium and supply it to the information processing unit 201.

The communication unit 214 has, for example, a network interface, can receive information transmitted from another device, and can supply the received information to the information processing unit 201. The communication unit 214 may transmit the information supplied from the information processing unit 201 to another device.

The drive 215 has an interface of a removable recording medium 221 such as a magnetic disk, an optical disk, a magneto-optical disk, or a semiconductor memory, reads information recorded on the removable recording medium 221 mounted on the drive 215, and reads out information processing unit 201. Can be supplied to. The drive 215 can record the information supplied from the information processing unit 201 on the writable removable recording medium 221 attached to the drive 215.

The information processing unit 201 loads and executes, for example, the application program stored in the storage unit 213. At that time, the information processing unit 201 can appropriately store data and the like necessary for executing various processes. The application program, data, and the like can be recorded and provided on a removable recording medium 221 as a package media or the like, for example. In that case, the application program, data, and the like are read out by the drive 215 equipped with the removable recording medium 221 and installed in the storage unit 213 via the information processing unit 201. The application program can also be provided via a wired or wireless transmission medium such as a local area network, the Internet, or digital satellite broadcasting. In that case, the application program, data, and the like are received by the communication unit 214 and installed in the storage unit 213 via the information processing unit 201. Further, the application program, data, and the like can be installed in advance in the ROM or the storage unit 213 in the information processing unit 201.

<Projector function block>
FIG. 6 shows a function realized by the information processing unit 201 executing an application program as a functional block. As shown in FIG. 6, the information processing unit 201 can have a correction vector acquisition unit 231, an image acquisition unit 232, and a correction image generation unit 233 as functional blocks by executing an application program.

The correction vector acquisition unit 231 acquires the correction vector supplied from the portable terminal device 101 and supplies it to the correction image generation unit 233.

The image acquisition unit 232 acquires the input image and supplies it to the correction image generation unit 233.

The corrected image generation unit 233 corrects the input image using the correction vector and generates the corrected image. The corrected image generation unit 233 supplies the corrected image to the projection unit 202 and projects the corrected image.

As described above, the portable terminal device 101 can perform projection correction without modeling the projector 102 (projection unit 202). That is, the portable terminal device 101 can perform projection correction with an accuracy that does not depend on the accuracy of the model. That is, the portable terminal device 101 can suppress a decrease in the accuracy of projection correction. Further, the portable terminal device 101 can adjust the position, size, tilt, and the like of the projection area only by the zoom shift roll while holding the corrected image. That is, the user does not need to consider the keystone correction in this correction. That is, the portable terminal device 101 can suppress a decrease in the accuracy of projection correction. In other words, the projection imaging system 100 can suppress a decrease in the accuracy of projection correction.

<Flow of projection correction processing>
An example of the flow of the projection correction process executed by the information processing unit 151 of the portable terminal device 101 will be described with reference to the flowchart of FIG. 7.

When the projection correction process is started, the corresponding point detection unit 181 detects the corresponding point in step S101. The internal variables of the camera of the image pickup unit 152 (for example, focal length, principal point position, lens distortion, etc.) are calibrated.

For example, as shown in FIG. 8, the projector 102-1 and the projector 102-2 project a sensing pattern which is a predetermined pattern image. A handheld camera (imaging unit 152 of the portable terminal device 101) captures the projected image from three places. The portable terminal device 101 decodes the captured images to acquire the corresponding points of the projector pixels on the captured images.

The pattern image used for this sensing is composed of the same Structured Light patterns of different colors. For example, the pattern image 301 projected by the projector 102-1 has a pattern in which black dots are arranged at equal intervals on a red background, and the pattern image 302 projected by the projector 102-2 has black dots arranged at equal intervals on a blue background. It may have a pattern arranged in.

The projector 102-1 and the projector 102-2 simultaneously project these pattern images on the screen 120. At that time, the projected images of the pattern images may be superimposed on each other.

The handheld camera (imaging unit 152 of the portable terminal device 101) is moved by the user to capture the projected image from three places. For example, the image pickup unit 152, whose position is controlled by the user, takes images from three locations from the left direction, the front direction, and the right direction with respect to the projection area on the screen 120 (left viewpoint camera 311 in FIG. 8, front view). Camera 312, right viewpoint camera 313). It should be noted that this imaging may be performed at a plurality of positions different from each other. Therefore, the number of shooting positions is arbitrary as long as it is 2 or more. In general, the larger the number of shooting positions (number of shootings), the more accurately the screen can be reconstructed. On the contrary, as the number of shooting positions (number of shootings) is smaller, the increase in processing load can be suppressed. Moreover, each shooting position is arbitrary. In general, the screen can be reconstructed more accurately when the shooting position is wider (the position where the difference in the shooting orientation is larger than each other), for example, as in the above example. The position of the front camera 312 in the above example may be such that the position of the front camera 312 is between the left viewpoint camera 311 and the right viewpoint camera 312, and is directly in front of the screen 120 (position facing the screen 120). It may be there, but it does not have to be directly in front of it. For example, the position of the front camera 312 may be substantially in front of (near directly in front of) the screen 120 manually positioned by the user.

Therefore, these captured images may include both the projected image projected by the projector 102-1 and the projected image projected by the projector 102-2.

The corresponding point detection unit 181 decodes such a captured image and separates a plurality of pattern images included in the captured image and projected by different projectors 102. The method of separating this pattern is arbitrary. For example, the relationship between the color information of the captured image obtained by capturing the mixed image of the projected images projected by giving different color information from the plurality of projectors 102, and the color information of the captured image and the color information of the projected image and the background. A separated image for each color information may be generated based on the color model shown.

The color model includes color information of the projected image changed according to the spectral characteristics of the projection unit 202 and the image pickup unit 152 that acquires the image capture image, a attenuation coefficient indicating the attenuation that occurs in the mixed image captured by the image pickup unit 152, and a background. The color information of is used as a parameter. Therefore, a separated image for each color information is generated based on the color model by using a parameter that minimizes the difference between the color information of the captured image and the color information estimated by the color model.

The Structured Light pattern used here may be any pattern as long as it is capable of color separation and decoding in one imaging. Further, when the camera is fixedly installed on a tripod or the like instead of being held by hand, a pattern such as Gray Code that decodes using information of a plurality of patterns in the time direction may be used. When the camera is fixed, the color separation process is not required, and the projector 102-1 and the projector 102-2 may project the pattern image at different timings in time.

The corresponding point detection unit 181 detects the corresponding point based on the plurality of captured images as described above. For example, in FIG. 9, the projected image 321 of the screen 120 is a projected image of the pattern image 301 projected by the projector 102-1. Further, the projected image 322 is a projected image of the pattern image 302 projected by the projector 102-2. Further, the captured image 331 is an captured image generated by the left viewpoint camera 311 capturing the screen 120 on which the projected image 321 and the projected image 322 are projected. The captured image 332 is an captured image generated by the front camera 312 capturing the projected image 321 and the screen 120 on which the projected image 322 is projected. The captured image 333 is an captured image generated by the right viewpoint camera 313 taking an image of the screen 120 on which the projected image 321 and the projected image 322 are projected.

The corresponding point detection unit 181 displays a pixel included in each of the captured image 331, the captured image 332, and the captured image 333, which displays points corresponding to each other, that is, a predetermined position of the pattern image 301 or the pattern image 302. Detect as a corresponding point (for example, white circle in the figure).

In step S102, the camera posture estimation unit 182 is based on the two-dimensional correspondence point information between the three captured images obtained by the above-mentioned correspondence point detection process, and the left viewpoints at three three-dimensionally matching points. The positions and orientations of the cameras 311 to the right viewpoint camera 313 are estimated.

First, we focus on the corresponding point information of the two captured images. For example, when focusing on the captured image 331 generated by the left viewpoint camera 311 and the captured image 332 generated by the front camera 312, as shown in FIG. 10A, the front viewpoint is captured from the corresponding point of the left viewpoint captured image 331. The homography matrix (H ₁₂ ) that transforms into the corresponding points of the image 332 is obtained. This homography is obtained by RANSAC (Random Sample Consensus), which is a robust estimation algorithm, so that even if there are outliers at the corresponding points, they are not significantly affected.

By RT decomposition of this homography matrix, the relative position and relative posture of the front camera 312 with respect to the left viewpoint camera 311 are derived. For the RT decomposition method, for example, the method described in "Journal of the Society of Image Electronics and Electronics / Vol. 40 (2011) No. 3, p.421-427" is used. At this time, the scale is indefinite, so the scale is determined by some rule.

As shown in B of FIG. 10, the position and orientation of the front camera 312 with respect to the left viewpoint camera 311 obtained here and the corresponding point information thereof are used for triangulation to obtain the corresponding points. Find the 3D point. Here, when finding a three-dimensional point by triangulation, the corresponding rays may not intersect each other. In that case, the midpoint of the line segment connecting the points where the corresponding rays are closest to each other is the three-dimensional point. And.

Next, as shown in A of FIG. 11, the front camera 312 and the right viewpoint camera 313 are focused on, and the same processing is performed to obtain the relative position and the relative posture of the right viewpoint camera 313 with respect to the front camera 312. At this time as well, the scales of the front camera 312 and the right viewpoint camera 313 are indefinite, so the scale is determined by some rule. Further, the three-dimensional points of the corresponding points are obtained by performing triangulation using the positions and postures of the front camera 312 and the right viewpoint camera 313 and their corresponding point information.

Next, as shown in B of FIG. 11, the average distance from the camera (left-view camera 311 or front camera 312) of the three-dimensional points of the corresponding points obtained from the left-view camera 311 and the front camera 312, and the front. Corrected the scales of the front camera 312 and the right viewpoint camera 313 so that the average distances from the camera (front camera 312 or right viewpoint camera 313) of the three-dimensional points of the corresponding points obtained from the camera 312 and the right viewpoint camera 313 match. do. The scale is modified by changing the lengths of the translational component vectors of the front camera 312 and the right viewpoint camera 313.

Finally, as shown in FIG. 12, the left viewpoint camera 311 is fixed as a reference, and the positions and orientations of the front camera 312 and the right viewpoint camera 313 are optimized for the internal parameters, the external parameters, and the world coordinate point cloud. Optimize by Bundle Adjustment. At this time, the evaluation value is the sum of squares of the distances from the three-dimensional points of the corresponding points to the corresponding three light rays, and optimization is performed so that this is the smallest. The three-dimensional corresponding points of the three rays are the triangular survey points of the corresponding rays of the left viewpoint camera 311 and the front camera 312, and the triangular survey points of the corresponding rays of the front camera 312 and the right viewpoint camera 313. The position of the center of gravity between the right-viewpoint camera 313 and the triangular survey point of the corresponding light beam of the left-viewpoint camera 311. As a result, the positions and orientations of the three cameras (that is, the left viewpoint camera 311 and the front camera 312, and the right viewpoint camera 313) are estimated.

In step S103, the screen reconstructing unit 183 reconstructs the screen based on the position and orientation of each camera estimated as described above.

First, the position and orientation of each camera (left viewpoint camera 311, front camera 312, and right viewpoint camera 313) estimated by the process of step S102 as shown in A of FIG. 13 and acquired by the process of step S101. For the three-dimensional point cloud (set of three-dimensional points 341) obtained from the corresponding point information obtained, the most matching plane as shown in B of FIG. 13 is obtained, and this is referred to as a temporary plane screen 351. .. Here, when finding a plane, the RANSAC method is used in order to suppress the influence of outliers mixed in the three-dimensional point cloud.

Next, as shown in A of FIG. 14 and B of FIG. 14, a virtual viewpoint is located at a certain distance in the direction facing the temporary flat screen 351 (normal direction of the temporary flat screen 351). Set the viewpoint camera 361. The viewpoint is a position set as a model of a position for viewing the projected image projected on the temporary flat screen 351. The viewpoint camera 361 is a virtual camera for obtaining a captured image showing the state of the projected image when viewed from the viewpoint. The viewpoint camera 361 is used as a reference for obtaining a correction vector so that the corrected image expected from the viewpoint camera 361 can be seen in the correction vector calculation process described later.

Further, as shown in B of FIG. 14, the vertical angle of the viewpoint camera 361 is a straight line group when the three-dimensional corresponding point group in the same row in the image projected from the projector 102 is linearly approximated. In the direction of the average of. When the projector 102 is installed on a horizontal plane, the vertical direction of the viewpoint camera 361 obtained here matches the vertical direction of the world where the actual screen 120 is installed, and the roll direction of the corrected image is automatically adjusted. It can be carried out. The temporary flat screen 351 is used only for determining the position of the viewpoint camera 361.

In step S104, the correction vector derivation unit 184 has the position and orientation of the camera estimated in the process of step S102, the corresponding point information acquired in the process of step S101, and the position of the viewpoint camera 361 obtained in the process of step S103. And the correction vector of each projector 102 is derived based on the posture.

The correction vector derivation unit 184 interpolates the missing points when the sensing points are missing, for example, as shown in FIG. Homography of the sensing point group 373 in the projector pixels around the missing sensing point and the two-dimensional point group obtained by projecting those three-dimensional points on the plane 374 that is a plane approximation of the corresponding three-dimensional point group is obtained. By converting the missing sensing point 371 of interest in the projector pixel by its homography, the corresponding three-dimensional point 372 is obtained, and this is used as the interpolation point of the missing three-dimensional point.

Next, as shown in FIG. 16, it is estimated how the projection by the projector 102 is projected on the captured image 390 generated by the viewpoint camera 361. The correction vector derivation unit 184 uses the sensing points in the vicinity of the pixels corresponding to the outer peripheral positions of the pattern image 301 and the pattern image 302 projected by the projector 102, and performs the same processing as the interpolation of the missing points described above to perform three-dimensional points. Find the position of. Then, the correction vector derivation unit 184 projects each of the three-dimensional points onto the light receiving region of the viewpoint camera 361 (that is, the captured image generated by the viewpoint camera 361) on the captured image generated by the viewpoint camera 361. The outer peripheral region of the projector 102 (the range projected by each projector 102) is estimated.

The correction vector derivation unit 184 performs such processing for each of the projector 102-1 and the projector 102-2. As a result, the correction vector derivation unit 184 can estimate how the projected images projected by the two projectors 102 appear on the captured image generated by the viewpoint camera 361, that is, how they appear on the screen 120. can.

Note that this process is performed when the projector 102 is not modeled or cannot be modeled. When the projector 102 can be modeled, the projection range (outer peripheral position) of each projector 102 on the captured image generated by the viewpoint camera 361 can be obtained by using its internal variables and external variables. Regarding the internal variable estimation for modeling the projector 102, a method of calibrating in advance and an internal variable corresponding to the zoom / shift can be acquired online in a projector having an optical zoom / shift adjustment. A method of introducing a mechanism in advance, a method of estimating the internal variable itself online, etc. can be considered.

Next, the correction vector derivation unit 184 determines where on the captured image of the viewpoint camera 361 (that is, on the screen 120) the input image is presented, as shown in FIG. The correction vector derivation unit 184 sets a rectangular region having the same aspect ratio as the input image in a region simultaneously included in the outer periphery of the images projected by the two projectors 102 (that is, a region in which both images are superimposed). For example, when the aspect ratio of the input image 402 is 16: 9, the correction vector derivation unit 184 has a projection range 391 of the projector 102-1 and a projection range 392 of the projector 102-2 in the captured image 390 generated by the viewpoint camera 361. A maximum rectangular area of 16: 9 is set in the area where the images overlap each other, and this is set as the image presentation position 401.

The initial values are the projection position parameters (zoom, shift, roll) that determine the image presentation position (for example, zoom: 1.0, shift: (0.0, 0.0), roll: 0.0). .. By changing this projection position parameter, the size, position, and tilt of the final corrected image on the screen also change accordingly. This projection position parameter can be changed by the projection area adjustment process described later.

Next, the correction vector derivation unit 184 obtains the correction vector corresponding to each pixel of the projector as shown in FIG. First, the correction vector derivation unit 184 obtains a three-dimensional point position for a certain pixel of interest of the projector 102 of interest. Here, each of the three-dimensional point coordinate values (X, Y, Z) of the sensing points of the 4x4 around the pixel of interest for which the three-dimensional points have already been obtained is obtained by Bicubic interpolation. Note that this interpolation method is an example and is not limited to this example. For example, Bilinear interpolation may be applied.

Next, the correction vector derivation unit 184 projects the obtained three-dimensional point of the pixel of interest onto the light receiving region (pixel region) of the viewpoint camera 361. The position of the projection point in the light receiving region (captured image 390) of the viewpoint camera 361 in the image presentation position 401 is obtained and converted into the coordinate system of the input image 402. The coordinate values (u, v) obtained here become the correction vector in the pixel of interest of the projector 102. That is, the expected correction image can be generated by storing the input pixel value of the pixel position of the obtained correction vector in the pixel of interest of the projector 102. The correction vector derivation unit 184 performs this processing for all the pixels of the two projectors 102 to obtain the correction vectors for each of the projectors 102-1 and the projector 102-2.

In step S105, the projection control unit 185 supplies the correction vector to the projector 102-1 and the projector 102-2, and projects the corrected image generated by using the correction vector.

The corrected image generation unit 233 of the projector 102 corrects the input image acquired by the image acquisition unit 232 using this correction vector, and generates a corrected image. By generating the corrected image in this way in each projector 102 and projecting it from the projection unit 202 onto the screen 120, the corrected projected image (projected image of the corrected image) is presented on the screen 120.

For example, the correction image generation unit 233 of the projector 102-1 uses the correction vector for the projector 102-1 (supplied from the portable terminal device 101) acquired by the correction vector acquisition unit 231 to obtain an image. A corrected image corresponding to the input image acquired by 232 is generated. The correction image generation unit 233 of the projector 102-2 uses the correction vector for the projector 102-1 (supplied from the portable terminal device 101) acquired by the correction vector acquisition unit 231 by the image acquisition unit 232. Generates a corrected image corresponding to the acquired input image. It should be noted that these processes can be executed in parallel with each other.

The projection unit 202 of the projector 102-1 projects the corrected image thus generated on the screen 120. The projection unit 202 of the projector 102-2 projects the corrected image thus generated on the screen 120. It should be noted that these processes can be executed in parallel with each other.

In step S106, the projection area adjusting unit 186 determines whether or not to adjust the position, size, inclination, etc. of the projection area (image projection position on the screen 120). For example, the user visually observes the projected image of the corrected image projected on the screen 120, determines whether or not the projected image has the expected position, size, and inclination, and gives a user instruction based on the determination result. Is input to the input unit 161. The projection area adjustment unit 186 determines whether to adjust the position, size, inclination, etc. of the projection area based on the received user instruction.

If it is determined to adjust the position, size, tilt, etc. of the projection area, the process proceeds to step S107. In step S107, the projection area adjusting unit 186 controls the output unit 162 to display the UI image, and controls the input unit 161 to accept the user instruction.

For example, the projection area adjustment unit 186 has a key corresponding to enlargement / reduction of the zoom parameter, a key corresponding to the up / down / left / right of the shift parameter, a key corresponding to the tilt angle of the roll parameter, and the like, as shown in A of FIG. The UI image 501 in which various keys 502 are arranged is displayed on the monitor so that the user can operate it. Further, the projection area adjustment unit 186 controls the input unit 161 to receive an operation on the UI image 501 by the user. The projection area adjustment unit 186 supplies the projection position parameter corresponding to the received user instruction to the correction vector derivation unit 184, generates a correction vector, generates a correction image using the correction vector, and projects the correction image. Let me. In this way, the corrected image on the screen 120 is enlarged / reduced, moved up / down / left / right, and rotated left / right in conjunction with the instruction input by the user. At this time, since the three-dimensional estimation is performed, adjustment can be easily performed while guaranteeing that the corrected image itself is rectangular (keystone correction).

Further, as shown in FIG. 19B, an area 512 imitating the screen 120 and an image 513 imitating the projected image displayed in the area 512 are displayed, and the user operates the image 513 with a finger or the like. Then, a UI image 511 capable of shifting, zooming, rolling, etc. may be displayed on the monitor. With such a UI image, the projection size, projection position, and projection angle can be adjusted more intuitively.

In step S108, the projection area adjustment unit 186 adjusts the projection position parameter based on the received user instruction, and supplies it to the correction vector derivation unit 184. For example, when the zoom parameter is updated according to the user's instruction, the projection area adjustment unit 186 may reduce the projection area from the projection image 523 to the projection image 524 or reduce the projection area as shown in FIG. 20A. The projection area is enlarged from 524 like the projection image 523. That is, the projection area adjustment unit 186 sets the projection position parameter so that the projection area has a size according to the user's instruction, and supplies the projection area parameter to the correction vector derivation unit 184. Further, by updating the shift parameter according to the user's instruction, the projection area adjusting unit 186 moves the projection area from the projection image 525 to the projection image 526 or vice versa, as shown in FIG. 20B. That is, the projection area adjustment unit 186 sets the projection position parameter so that the projection area is at a position according to the user's instruction, and supplies the projection area parameter to the correction vector derivation unit 184. Further, by updating the roll parameter according to the user's instruction, the projection area adjusting unit 186 rotates the projection area from the projection image 527 to the projection image 528 or vice versa, as shown in FIG. 20C. That is, the projection area adjustment unit 186 sets the projection position parameter so that the projection area has an inclination according to the user's instruction, and supplies it to the correction vector derivation unit 184.

When the process of step S108 is completed, the process returns to step S104. In step S104, the correction vector derivation unit 184 derives the correction vector using the updated projection position parameter. Assuming that the projector is installed horizontally, the roll direction is automatically adjusted by the process of step S103, so that the roll direction cannot be manually adjusted.

In step S106, when the user's desired projection area is set and it is determined that the position, size, inclination, etc. of the projection area are not adjusted, the projection correction process ends.

By performing the projection correction in this way, the projection correction can be performed without modeling the projector 102 (projection unit 202), so that the portable terminal device 101 (projection imaging system 100) can perform the projection correction. It is possible to suppress a decrease in accuracy. Further, since the user does not need to consider the keystone correction in the adjustment of the projection area, the portable terminal device 101 (projection imaging system 100) can suppress the decrease in the accuracy of the projection correction.

<3. Second Embodiment>
<Cylindrical screen>
The projection surface may be a curved surface and is not limited to a flat surface. For example, it may be a cylindrical surface. That is, the screen 120 may be a cylindrical screen instead of a flat screen.

Also in this case, the corresponding point detection by the corresponding point detection unit 181 and the camera posture estimation by the camera posture estimation unit 182 are the same as in the case of the flat screen described in the first embodiment. In the following, the aspect ratio of the presented image is 16: 9, that is, the format of the image is the same as that of the flat screen.

<Screen reconstruction>
In step S103, the screen reconstruction unit 183 is a three-dimensional point obtained from the position and orientation of the camera estimated in step S102 and the corresponding point information detected in step S101, as shown in A of FIG. 21. For the group, the most matching cylindrical surface as shown in B of FIG. 21 is obtained, and this is referred to as a temporary cylindrical surface screen 601. When determining this cylindrical surface, the RANSAC method is applied in order to suppress the influence of outliers mixed in the three-dimensional point cloud.

Next, the screen reconstruction unit 183 views the position of the cylindrical surface screen radius on the perpendicular extension line from the estimated center of gravity of the three-dimensional point cloud to the temporary cylindrical surface screen 601 as shown in A of FIG. Then, the viewpoint camera 611 is set at that viewpoint. The direction of the viewpoint camera 611 is the direction of the center of gravity of the estimated three-dimensional point cloud. Further, as shown in B of FIG. 22, the roll direction is determined by using the same method as in the case of the flat screen. In this way, the viewpoint camera 611 corresponding to the temporary cylindrical screen 601 is set.

<Derivation of correction vector>
In step S104, the correction vector derivation unit 184 has the position and orientation of the camera estimated in the process of step S102, the corresponding point information detected in the process of step S101, and the temporary cylindrical surface obtained in the process of step S103. The correction vector of each projector 102 is obtained based on the position and orientation of the screen 601 and the viewpoint camera 611.

First, the correction vector derivation unit 184 estimates how the projection of the projector 102 is projected on the cylindrical coordinate system of the temporary cylindrical surface screen 601 as shown in FIG. 23. Here, the pixels corresponding to the outer peripheral position of the projector 102 are obtained by using the sensing points in the vicinity, and the positions of the three-dimensional points are obtained by the same processing as the interpolation of the missing points in the correction vector calculation of the flat screen, and then the third. By converting the dimensional points into a cylindrical coordinate system based on the position of the viewpoint camera 611, the outer peripheral region of the projector 102 on the cylindrical coordinate system (the outer peripheral region 631 and the outer peripheral region 632 in the captured image 630 of the viewpoint camera 611). To estimate. In this cylindrical coordinate system, the coordinate system is expressed in the unit of the distance of [mm] in both the vertical and horizontal directions.

The correction vector derivation unit 184 performs this processing for each projector 102. This makes it possible to estimate how it will appear on the cylindrical screen. It should be noted that this process is performed when the projector 102 is not modeled or cannot be modeled. If the model can be modeled, the outer peripheral position of the projector 102 on the captured image of the viewpoint camera 611 can be obtained by using the internal variables and the external variables, as in the case of the flat screen.

Next, the correction vector derivation unit 184 determines where on the cylindrical coordinate system (= on the cylindrical surface screen) the input image 642 is presented, as shown in FIG. 24. The correction vector derivation unit 184 sets a rectangular region having the same aspect ratio as the input image in the region simultaneously included in the outer circumferences of the two projectors previously obtained in the cylindrical coordinate system. For example, in the correction vector derivation unit 184, when the aspect ratio of the input image is 16: 9, the projection range of the projector 102-1 and the projection range of the projector 102-2 are superimposed on the captured image of the viewpoint camera 611. A maximum rectangular area of 16: 9 is set in the existing area, and this is set as the image presentation position 641.

Next, the correction vector derivation unit 184 obtains the correction vector corresponding to each pixel of the projector 102 as shown in FIG. 25. First, the correction vector derivation unit 184 obtains a three-dimensional point position for a certain pixel of interest of the projector 102 of interest. Here, as in the case of the flat screen, the three-dimensional point coordinate values X, Y, and Z of the sensing points around the pixel of interest for which the three-dimensional points have already been obtained are obtained by Bicubic interpolation.

Next, the correction vector derivation unit 184 converts the three-dimensional point of the pixel of interest obtained here into a cylindrical coordinate system centered on the viewpoint camera 611, and the conversion point on the cylindrical coordinate system is the image presentation position. Find where it is located inside and convert it to the coordinate system of the input image. The coordinate values obtained here are the correction vectors (u, v) in the pixel of interest of the projector. By performing this process on all the pixels of the two projectors 102, the correction vector on the cylindrical screen of each projector 102 is obtained.

By the above processing, the portable terminal device 101 can perform automatic projective geometry correction on the cylindrical surface screen. The process of adjusting the projection area can be realized by adjusting the image presentation position in the same manner as in the case of a flat screen. Therefore, also in the case of this cylindrical screen, the portable terminal device 101 (projection imaging system 100) can perform projection correction without modeling the projection unit, as in the case of the above-mentioned flat screen. , It is possible to suppress a decrease in the accuracy of projection correction. Further, since the user does not need to consider the keystone correction in this correction, the portable terminal device 101 (projection imaging system 100) can suppress a decrease in the accuracy of the projection correction.

<4. Third Embodiment>
<Spherical screen>
Further, the projection surface may be a spherical surface. That is, the screen 120 may be a spherical screen instead of a flat screen.

Also in this case, the corresponding point detection by the corresponding point detection unit 181 and the camera posture estimation by the camera posture estimation unit 182 are the same as in the case of the flat screen described in the first embodiment. In the following, the image to be presented will be in an equirectangular format that is supposed to be projected on a spherical screen or the like.

<Screen reconstruction>
In step S103, the screen reconstruction unit 183 is a three-dimensional point obtained from the position and orientation of the camera estimated in step S102 and the corresponding point information detected in step S101, as shown in A of FIG. 26. For the group, the most matching spherical surface as shown in B of FIG. 26 is obtained, and this is referred to as a pseudo-spherical surface screen 701. When obtaining this sphere, the RANSAC method is applied in order to suppress the influence of outliers mixed in the three-dimensional point cloud.

Next, as shown in A of FIG. 27, the screen reconstruction unit 183 sets the center of the viewpoint camera 711 at the center of this spherical surface, and looks at the direction toward the center of gravity of the three-dimensional point cloud estimated from the center. The orientation of the camera 711. Further, as shown in B of FIG. 27, the screen reconstruction unit 183 determines the roll direction by using the same method as in the case of the flat screen. However, unlike the case of a flat screen or a cylindrical screen, there is no guarantee that the roll adjustment can be performed automatically even if the projector is installed on a horizontal plane. It is used only as the initial value in the roll direction. Alternatively, the vertical direction of the posture of the front camera can be set as the initial roll value here. In this way, the viewpoint camera 711 corresponding to the temporary spherical screen 701 is set.

<Derivation of correction vector>
In step S104, the correction vector derivation unit 184 has the position and orientation of the camera estimated in the process of step S102, the corresponding point information detected in the process of step S101, and the pseudospherical screen obtained in the process of step S103. The correction vector of each projector 102 is obtained based on the position and orientation of the viewpoint camera 711 and the viewpoint camera 711.

First, as shown in FIG. 28, the correction vector derivation unit 184 estimates how the projection of the projector is projected on the regular distance cylindrical coordinate system of the pseudospherical screen 701. Here, the pixels corresponding to the outer peripheral position of the projector 102 are obtained by using the sensing points in the vicinity, and the positions of the three-dimensional points are obtained by the same processing as the interpolation of the missing points in the correction vector calculation of the flat screen, and then the third. By converting the 3D points into a regular-distance cylindrical coordinate system based on the viewpoint camera position, the outer peripheral region of the projector on the regular-distance cylindrical coordinate system is estimated. In this regular-distance cylindrical coordinate system, the coordinates are expressed in units of angle radians in both the vertical and horizontal directions.

The correction vector derivation unit 184 performs this processing for each projector 102. This makes it possible to estimate how it will appear on the spherical screen. It should be noted that this process is performed when the projector 102 is not modeled or cannot be modeled. If the model can be modeled, the outer peripheral position of the projector 102 on the viewpoint camera image can be obtained by using the internal variables and the external variables, as in the case of the flat screen.

Next, the correction vector derivation unit 184 determines where on the regular distance cylindrical coordinate system (= on the spherical screen) the input image is presented, as shown in FIG. 29. The correction vector derivation unit 184 sets a region included in the outer periphery of the two projectors 102 previously obtained in the regular distance cylindrical coordinate system as the presentation range of the input image. Unlike when the input image is in the 16: 9 planar image format, which region in the equirectangular format is presented depends on the projection range on the spherical screen. Of course, it is also possible to intentionally make a projection that ignores the physical meaning of the equirectangular format. It is also possible to project the area including the area where the two projectors are not superimposed, and even if a 16: 9 planar image format is input, it is possible to project the corrected image if some rules are determined.

Next, the correction vector derivation unit 184 obtains the correction vector corresponding to each pixel of the projector as shown in FIG. First, the correction vector derivation unit 184 obtains a three-dimensional point position for a certain pixel of interest of the projector of interest. Here, as in the case of the flat screen, the three-dimensional point coordinate values X, Y, and Z of the sensing points around the pixel of interest for which the three-dimensional points have already been obtained are obtained by Bicubic interpolation.

Next, the correction vector derivation unit 184 converts the three-dimensional point of the pixel of interest obtained here into a regular-distance cylindrical coordinate system centered on the viewpoint camera 711, and the conversion point on the regular-distance cylindrical coordinate system is obtained. Find where it is located and convert it to the coordinate system of the input image. The coordinate values obtained here are the correction vectors (u, v) in the pixel of interest of the projector. By performing this process on all the pixels of the two projectors 102, the correction vector on the spherical screen of each projector 102 is obtained.

By the above processing, automatic projective geometry correction on a spherical screen can be performed. The process of adjusting the projection area can be realized by adjusting the image presentation position in the same manner as in the case of a flat screen. Therefore, also in the case of this spherical screen, the portable terminal device 101 (projection imaging system 100) can perform projection correction without modeling the projection unit, as in the case of the above-mentioned flat screen. It is possible to suppress a decrease in the accuracy of projection correction. Further, since the user does not need to consider the keystone correction in this correction, the portable terminal device 101 (projection imaging system 100) can suppress a decrease in the accuracy of the projection correction.

<5. Addendum>
<Hardware>
The series of processes described above can be executed by software (application program) or by hardware.

<Applicable target of this technology>
In addition, the present technology includes any configuration or a module using a processor (for example, a video processor) as a system LSI (Large Scale Integration), a module using a plurality of processors, or the like (for example, video) mounted on an arbitrary device or a device constituting the system. It can also be implemented as a module), a unit using a plurality of modules (for example, a video unit), a set in which other functions are added to the unit (for example, a video set), or the like (that is, a partial configuration of a device).

Furthermore, this technology can also be applied to a network system composed of a plurality of devices. For example, it can be applied to cloud services that provide services related to images (moving images) to arbitrary terminals such as computers, AV (AudioVisual) devices, portable information processing terminals, and IoT (Internet of Things) devices. can.

Systems, equipment, processing departments, etc. to which this technology is applied should be used in any field such as transportation, medical care, crime prevention, agriculture, livestock industry, mining, beauty, factories, home appliances, weather, nature monitoring, etc. Can be done. The use is also arbitrary.

For example, this technology can be applied to systems and devices used for providing ornamental contents and the like. Further, for example, the present technology can be applied to systems and devices used for traffic such as traffic condition supervision and automatic driving control. Further, for example, the present technology can be applied to systems and devices used for security purposes. Further, for example, the present technology can be applied to a system or device used for automatic control of a machine or the like. Further, for example, the present technology can be applied to systems and devices used for agriculture and livestock industry. The present technology can also be applied to systems and devices for monitoring natural conditions such as volcanoes, forests and oceans, and wildlife. Further, for example, the present technology can be applied to systems and devices used for sports.

<Others>
The embodiment of the present technology is not limited to the above-described embodiment, and various changes can be made without departing from the gist of the present technology.

For example, the present technology includes any configuration that constitutes a device or system, for example, a processor (for example, a video processor) as a system LSI (Large Scale Integration), a module that uses a plurality of processors (for example, a video module), and a plurality of modules. It can also be implemented as a unit using the above (for example, a video unit), a set (for example, a video set) in which other functions are added to the unit, or the like (that is, a partial configuration of the device).

In the present specification, the system means a set of a plurality of components (devices, modules (parts), etc.), and it does not matter whether all the components are in the same housing. Therefore, a plurality of devices housed in separate housings and connected via a network, and a device in which a plurality of modules are housed in one housing are both systems. ..

Further, for example, the configuration described as one device (or processing unit) may be divided and configured as a plurality of devices (or processing units). On the contrary, the configurations described above as a plurality of devices (or processing units) may be collectively configured as one device (or processing unit). Further, of course, a configuration other than the above may be added to the configuration of each device (or each processing unit). Further, if the configuration and operation of the entire system are substantially the same, a part of the configuration of one device (or processing unit) may be included in the configuration of another device (or other processing unit). ..

Further, for example, this technology can have a cloud computing configuration in which one function is shared by a plurality of devices via a network and processed jointly.

Further, for example, the above-mentioned program can be executed in any device. In that case, the device may have necessary functions (functional blocks, etc.) so that necessary information can be obtained.

Further, for example, each step described in the above flowchart can be executed by one device or can be shared and executed by a plurality of devices. Further, when a plurality of processes are included in one step, the plurality of processes included in the one step can be executed by one device or shared by a plurality of devices. In other words, a plurality of processes included in one step can be executed as processes of a plurality of steps. On the contrary, the processes described as a plurality of steps can be collectively executed as one step.

In the program executed by the computer, the processes of the steps for describing the program may be executed in chronological order in the order described in the present specification, or may be called in parallel or called. It may be executed individually at the required timing such as when. That is, as long as there is no contradiction, the processes of each step may be executed in an order different from the above-mentioned order. Further, the processing of the step for describing this program may be executed in parallel with the processing of another program, or may be executed in combination with the processing of another program.

It should be noted that the present techniques described above in this specification can be independently implemented independently as long as there is no contradiction. Of course, any plurality of the present technologies can be used in combination. For example, some or all of the techniques described in any of the embodiments may be combined with some or all of the techniques described in other embodiments. In addition, a part or all of any of the above-mentioned techniques may be carried out in combination with other techniques not described above.

It should be noted that the effects described in the present specification are merely examples and are not limited, and other effects may be obtained.

The present technology can also have the following configurations.
(1) The projection of the three-dimensional point position, which is the projection position in the three-dimensional space corresponding to the pixel of interest of the projection unit that projects the input image, at the position and orientation as the viewpoint of the projection image projected by the projection unit. Projecting onto the light receiving area of the virtual imaging unit, which is a virtual imaging unit that captures an image, and converting the projection point indicating the position where the three-dimensional point position in the light receiving region is projected into the coordinate system of the input image. An information processing device including a correction vector derivation unit that derives a correction vector corresponding to the pixel of interest.
(2) The information processing apparatus according to (1), wherein the correction vector derivation unit obtains the three-dimensional point position of the pixel of interest based on the known three-dimensional point position corresponding to the peripheral pixels of the pixel of interest. ..
(3) The information processing apparatus according to (1) or (2), wherein the correction vector derivation unit derives the correction vector for all the pixels of the projection unit.
(4) The correction vector derivation unit sets the presentation position of the input image in the light receiving region, and converts the projection point into the coordinate system of the input image by using the presentation position (1) to. The information processing apparatus according to any one of (3).
(5) The correction vector derivation unit estimates a range corresponding to the projected image projected by each of the plurality of projection units in the light receiving region, and the range corresponding to each of the plurality of projection units is superimposed. The information processing apparatus according to (4), wherein the presentation position is set in the area.
(6) The information processing apparatus according to (5), wherein the correction vector derivation unit interpolates the three-dimensional point position with respect to the pixel in which the three-dimensional point position is missing in the projection unit.
(7) Further includes a projection control unit that projects a corrected image, which is an image generated by correcting the input image using the correction vector derived by the correction vector derivation unit, onto the projection unit (1). ) To (6).
(8) Further provided with a projection area adjusting unit for adjusting the projection area of the input image.
The information processing apparatus according to any one of (1) to (7), wherein the correction vector derivation unit derives the correction vector using a parameter indicating an adjustment result of the projection area by the projection area adjustment unit.
(9) The information processing apparatus according to (8), wherein the projection area adjusting unit adjusts the projection area based on a user instruction input based on the user interface, and derives the parameter.
(10) The information processing apparatus according to (9), wherein the parameter includes a parameter related to the zoom of the projection area, a parameter related to the shift of the projection area, and a parameter related to the roll of the projection area.
(11) A projection plane viewpoint setting unit for setting the projection plane based on the three-dimensional point position and setting the viewpoint corresponding to the derived projection plane is further provided.
The correction vector derivation unit projects the three-dimensional point position of the pixel of interest onto the light receiving region of the virtual imaging unit corresponding to the viewpoint set by the projection plane viewpoint setting unit (1) to (10). The information processing device described in any of the above.
(12) The information processing apparatus according to (11), wherein the projection surface is a flat surface.
(13) The information processing apparatus according to (11), wherein the projection surface is a cylindrical surface.
(14) The information processing apparatus according to (11), wherein the projection surface is a spherical surface.
(15) The position and the posture are estimated for each of the plurality of captured images obtained by capturing the projected images at different positions and postures, and the three-dimensional point position is derived based on the estimated position and the posture. Further equipped with a position / posture estimation unit,
The information processing according to any one of (11) to (14), wherein the projection plane viewpoint setting unit sets the projection surface and the viewpoint based on the three-dimensional point position derived by the position / orientation estimation unit. Device.
(16) The information processing apparatus according to (15), wherein the position / orientation estimation unit estimates the relative position and the relative posture of the captured image of interest with respect to the captured image as a reference.
(17) The information processing apparatus according to (16), wherein the position / orientation estimation unit adjusts the scale of the three-dimensional point position based on the plurality of relative positions and the relative attitudes.
(18) Further, a corresponding point detecting unit for detecting corresponding points included in the plurality of captured images is provided.
The position / orientation estimation unit estimates the position and the posture based on the corresponding point detected by the corresponding point detecting unit, and derives the three-dimensional point position from any of (15) to (17). The information processing device described in.
(19) The captured image includes a plurality of pattern images of different colors projected by the plurality of projection units.
The information processing apparatus according to (18), wherein the corresponding point detection unit separates a plurality of the pattern images and detects the corresponding points.
(20) The projection of the three-dimensional point position, which is the projection position on the three-dimensional space corresponding to the pixel of interest of the projection unit that projects the input image, at the position and orientation as the viewpoint of the projection image projected by the projection unit. Projecting onto the light receiving area of the virtual image pickup unit, which is a virtual image pickup unit that captures an image, and converting the projection point indicating the position where the three-dimensional point position in the light receiving area is projected into the coordinate system of the input image. An information processing method for deriving a correction vector corresponding to the pixel of interest.

100 projection imaging system, 101 portable terminal device, 102 projector, 151 information processing unit, 152 imaging unit, 181 corresponding point detection unit, 182 camera posture estimation unit, 183 screen reconstruction unit, 184 correction vector derivation unit, 185 projection control Unit, 186 projection area adjustment unit, 201 information processing unit, 202 projection unit, 231 correction vector acquisition unit, 232 image acquisition unit, 233 correction image generation unit

Claims

The projected image is captured at the position and orientation of the three-dimensional point position, which is the projected position in the three-dimensional space corresponding to the pixel of interest of the projection unit that projects the input image, as the viewpoint of the projected image projected by the projection unit. By projecting onto the light receiving area of the virtual imaging unit, which is a virtual imaging unit, and converting the projection point indicating the position where the three-dimensional point position in the light receiving area is projected into the coordinate system of the input image, the said An information processing device equipped with a correction vector derivation unit that derives a correction vector corresponding to the pixel of interest.
The information processing apparatus according to claim 1, wherein the correction vector derivation unit obtains the three-dimensional point position of the pixel of interest based on the known three-dimensional point position corresponding to the peripheral pixels of the pixel of interest.
The information processing device according to claim 1, wherein the correction vector derivation unit derives the correction vector for all the pixels of the projection unit.
The information according to claim 1, wherein the correction vector derivation unit sets a presentation position of the input image in the light receiving region, and converts the projection point into a coordinate system of the input image by using the presentation position. Processing equipment.
The correction vector derivation unit estimates a range corresponding to the projected image projected by each of the plurality of projection units in the light receiving region, and in the region where the range corresponding to each of the plurality of projection units is superimposed. The information processing apparatus according to claim 4, wherein the presentation position is set.
The information processing device according to claim 5, wherein the correction vector derivation unit interpolates the three-dimensional point position with respect to the pixel in which the three-dimensional point position is missing in the projection unit.
The first aspect of the present invention further comprises a projection control unit that projects a corrected image, which is an image generated by correcting the input image using the correction vector derived by the correction vector derivation unit, onto the projection unit. Information processing equipment.
Further, a projection area adjusting unit for adjusting the projection area of the input image is provided.
The information processing apparatus according to claim 1, wherein the correction vector derivation unit derives the correction vector using a parameter indicating an adjustment result of the projection area by the projection area adjustment unit.
The information processing apparatus according to claim 8, wherein the projection area adjusting unit adjusts the projection area based on a user instruction input based on the user interface, and derives the parameter.
The information processing apparatus according to claim 9, wherein the parameter includes a parameter relating to zooming in the projection area, a parameter relating to shift of the projection region, and a parameter relating to roll of the projection region.
A projection plane viewpoint setting unit for setting a projection plane based on the three-dimensional point position and setting the viewpoint corresponding to the derived projection plane is further provided.
The information according to claim 1, wherein the correction vector derivation unit projects the three-dimensional point position of the pixel of interest on the light receiving region of the virtual imaging unit corresponding to the viewpoint set by the projection plane viewpoint setting unit. Processing equipment.
The information processing apparatus according to claim 11, wherein the projection surface is a plane.
The information processing apparatus according to claim 11, wherein the projection surface is a cylindrical surface.
The information processing apparatus according to claim 11, wherein the projection surface is a spherical surface.
Position-orientation estimation that estimates the position and the posture for each of the plurality of captured images obtained by capturing the projected image at different positions and postures, and derives the three-dimensional point position based on the estimated position and the posture. With more parts,
The information processing device according to claim 11, wherein the projection plane viewpoint setting unit sets the projection surface and the viewpoint based on the three-dimensional point position derived by the position / orientation estimation unit.
The information processing device according to claim 15, wherein the position / orientation estimation unit estimates the relative position and relative posture of the captured image of interest with respect to the captured image as a reference.
The information processing apparatus according to claim 16, wherein the position / orientation estimation unit adjusts the scale of the three-dimensional point position based on the plurality of relative positions and the relative attitudes.
Further, a corresponding point detection unit for detecting corresponding points included in the plurality of captured images is provided.
The information processing device according to claim 15, wherein the position / orientation estimation unit estimates the position and the posture based on the corresponding point detected by the corresponding point detecting unit, and derives the three-dimensional point position.
The captured image includes a plurality of pattern images of different colors projected by the plurality of projection units.
The information processing device according to claim 18, wherein the corresponding point detection unit separates a plurality of the pattern images and detects the corresponding points.
The projected image is captured at the position and orientation of the three-dimensional point position, which is the projected position in the three-dimensional space corresponding to the pixel of interest of the projection unit that projects the input image, as the viewpoint of the projected image projected by the projection unit. By projecting onto the light receiving area of the virtual imaging unit, which is a virtual imaging unit, and converting the projection point indicating the position where the three-dimensional point position in the light receiving area is projected into the coordinate system of the input image, the said An information processing method that derives a correction vector corresponding to the pixel of interest.