JP2016014720A - Information processor and method - Google Patents

Abstract

PROBLEM TO BE SOLVED: To provide an information processor capable of easily and precisely calibrating the positional relationship between a projection part and an imaging part.SOLUTION: The information processor estimates a mutual posture as relative position and direction of the projection part and the imaging part based on a relative positional relationship between an image projected by the projection part and the image taken by an imaging part on predetermined plane. The technique is applicable to, for example, a projector or a camera; or an electronic apparatus having both functions of them and a computer controlling the same.

Description

  The present technology relates to an information processing apparatus and method, and more particularly, to an information processing apparatus and method that can calibrate a positional relationship between a projection unit and an imaging unit more easily and with high accuracy.

  Conventionally, there has been a system for projecting an image using a plurality of projectors. In such a system, one image can be projected by appropriately connecting the projection images of the projectors. For this purpose, the projection image is sensed by a camera or the like, and the image projected by each projector is corrected using the sensing result.

  At the time of the correction, not only the internal variables of each camera and projector but also external variables such as the positional relationship of the camera and projector are used. These variables were calibrated at the time of system shipment or installation. . For example, a method of detecting the correspondence between the projector and the camera pixel using the Gray code, or a correspondence image between the projector and the camera pixel using a pattern image lacking any of the four corners of the checker and the checker. A method was conceived (for example, see Non-Patent Document 1).

Ramesh Raskar, Jeroen van Baar, Paul Beardsley, Thomas Willwacher, Srinivas Rao, Clifton Forlines, "iLamps: Geometrically Aware and Self-Configuring Projectors", ACM SIGGRAPH 2003 Conference Proceedings

  However, external variables such as the positional relationship of cameras and projectors may change over time after the system is installed, but in that case, it is necessary to calibrate again using a special device. In order to perform such calibration, complicated work is required. Further, with such a method, it is difficult to perform all calibrations with high accuracy.

  The present technology has been proposed in view of such a situation, and an object thereof is to enable calibration of the positional relationship between a projection unit and an imaging unit more easily and with high accuracy.

  One aspect of the present technology provides a relative position and direction between the projection unit and the imaging unit based on a relative positional relationship between an image projected by the projection unit and the image captured by the imaging unit on a predetermined plane. It is an information processing apparatus provided with the mutual attitude | position estimation part which estimates the mutual attitude | position which is.

  The mutual posture estimation unit calculates the mutual posture based on a distance between a position on the predetermined surface of the image projected by the projection unit and a position on the predetermined surface of the image captured by the imaging unit. The variable to be determined can be estimated.

  The mutual posture estimation unit updates an evaluation value of a distance between a position on the predetermined plane of the image projected by the projection unit and a position on the predetermined plane of the image captured by the imaging unit. And a variable estimation unit that estimates a variable for determining the mutual attitude based on the evaluation value updated by the update unit.

  The update unit includes the position of the processing target pixel included in the image captured by the imaging unit on the predetermined surface, and the predetermined surface of the pixel of the image projected by each projection unit that projects the pixel. The mean square of the distance to the position at can be used as the evaluation value.

  The variable estimation unit may include a rotation matrix of the imaging unit, a translation vector of the imaging unit, a rotation matrix of the projection unit, and a translation vector of the projection unit.

  A provisional mutual attitude estimation unit that estimates a mutual attitude between the projection imaging device having the projection unit and the imaging unit as a provisional mutual attitude; the mutual attitude estimation unit estimated by the temporary mutual attitude estimation unit; The mutual posture can be estimated using the temporary mutual posture as an initial value.

  The temporary mutual posture estimation unit can estimate the temporary mutual posture so that an error in the position of the projection plane of each projection imaging apparatus is minimized.

  Each projection imaging apparatus further includes a projection plane attitude estimation unit that estimates a mutual attitude of the projection unit and the imaging unit with respect to a projection plane on which the image is projected, and the temporary mutual attitude estimation unit includes the projection plane attitude estimation The provisional mutual posture can be estimated using the mutual posture of the projection unit and the imaging unit with respect to the projection plane estimated by the unit.

  The projection plane attitude estimation unit can estimate the projection plane by performing triangulation, and can estimate a mutual attitude of the projection unit and the imaging unit with respect to the estimated projection plane.

  The projection plane orientation estimation unit includes a correspondence relationship between pixel positions of the projection unit and the imaging unit in the projection imaging device, and a correspondence relationship between pixel positions of the projection unit and the imaging unit between the projection imaging devices. Based on the above, it is possible to estimate the mutual attitude of the projection unit and the imaging unit with respect to the projection plane.

  The temporary mutual posture estimation unit can estimate the temporary mutual posture based on a correspondence relationship between pixel positions of the projection unit and the imaging unit between the projection imaging devices.

  The projection plane posture estimation unit detects the projection unit and the imaging unit in the projection imaging apparatus that are detected by causing the projection unit to project a pattern used when determining pixel position correspondence and causing the imaging unit to capture an image. Based on the correspondence relationship of the pixel position with respect to the projection imaging device and the correspondence relationship of the pixel position between the projection unit and the imaging unit between the projection imaging devices, the mutual attitude of the projection unit and the imaging unit with respect to the projection plane can do.

  The image processing apparatus further includes an imaging unit that captures a projected image and obtains a captured image, and the mutual posture estimation unit includes an image projected by a projection unit of another device and the image captured by the imaging unit on the predetermined plane. Based on the relative positional relationship, the mutual posture of the projection unit and the imaging unit can be estimated.

  The image processing apparatus further includes a projection unit that projects an image, and the mutual posture estimation unit can further estimate a mutual posture between the projection unit and the imaging unit.

  The image processing unit may further include an image processing unit that performs image processing for correcting an image to be projected using a variable according to a mutual posture between the projection unit and the imaging unit estimated by the mutual posture estimation unit.

  One aspect of the present technology also provides a relative position between the projection unit and the imaging unit based on a relative positional relationship between an image projected by the projection unit and the image captured by the imaging unit on a predetermined plane. And an information processing method for estimating a mutual posture which is a direction.

  In one aspect of the present technology, based on a relative positional relationship between an image projected by the projection unit and the image captured by the imaging unit on a predetermined surface, the relative position between the projection unit and the imaging unit, and A mutual attitude which is a direction is estimated.

  According to the present technology, information can be processed based on an image. According to the present technology, it is possible to calibrate the positional relationship between the projection unit and the imaging unit more easily and with high accuracy.

It is a figure which shows the main structural examples of a projection imaging system. It is a block diagram which shows the main structural examples of a control apparatus. It is a functional block diagram which shows the structural example of the main functions implement | achieved in CPU. It is a figure explaining the example of the external appearance of a projection imaging device. It is a block diagram which shows the main structural examples of a projection imaging device. It is a block diagram which shows the main structural examples of a projection part. It is a figure which shows the example of the scanning of a laser beam. It is a flowchart explaining the example of the flow of a calibration process. It is a flowchart explaining the example of the flow of a pixel position corresponding | compatible detection process. It is a figure which shows the example of the projection imaging system installed. It is a figure which shows the example of a gray code. It is a figure explaining the example of a mode that the pixel position correspondence of a projection part and an imaging part is calculated | required within an apparatus. It is a figure explaining the example of a mode that the pixel position correspondence of a projection part and an imaging part is calculated | required between apparatuses. It is a flowchart explaining the example of the flow of a projection surface attitude | position estimation process. It is a figure explaining the example of the mode of the estimation of the attitude | position with respect to a projection surface. It is a flowchart explaining the example of the flow of a temporary mutual attitude | position estimation process. It is a figure explaining the example of a mode that the mutual attitude | position of apparatuses is estimated. It is a figure explaining the example of a mode that the mutual attitude | position of apparatuses is estimated. It is a flowchart explaining the example of the flow of a mutual attitude | position re-estimation process. It is a figure which shows the example of a mode that the mutual attitude | position of apparatuses is re-estimated. It is a figure which shows the other structural example of a projection imaging system. It is a figure which shows the further another structural example of a projection imaging system. It is a figure which shows the further another structural example of a projection imaging system. It is a figure which shows the further another structural example of a projection imaging system.

Hereinafter, modes for carrying out the present disclosure (hereinafter referred to as embodiments) will be described. The description will be given in the following order.
1. First embodiment (projection imaging system)

<1. First Embodiment>
<External variable aging and recalibration>
Conventionally, there has been a system for projecting an image using a plurality of projectors. In such a system, one image can be projected by appropriately connecting the projection images of the projectors. For this purpose, the projection image is sensed by a camera or the like, and the image projected by each projector is corrected using the sensing result.

  At the time of the correction, not only the internal variables of each camera and projector but also external variables such as the positional relationship of the camera and projector are used. These variables were calibrated at the time of system shipment or installation. . For example, a method of detecting a correspondence relationship between a projector and a camera pixel using a Gray code, or a projector using a pattern image lacking any of the four corners of the checker and the checker as described in Non-Patent Document 1. And a method for detecting the correspondence between the camera pixels.

  However, external variables such as the positional relationship of cameras and projectors may change over time after the system is installed, but in that case, it is necessary to calibrate again using a special device. In order to perform such calibration, complicated work is required. Further, with such a method, it is difficult to perform all calibrations with high accuracy.

  Therefore, the relative position between the image projected by the projection unit and the image captured by the imaging unit on a predetermined plane so that the positional relationship between the projection unit and the imaging unit can be calibrated more easily and with high accuracy. Based on the relationship, a mutual posture estimation unit that estimates a mutual posture that is a relative position and direction between the projection unit and the imaging unit is provided.

  In other words, the mutual posture estimation unit is configured so that the position of the image projected by each projection unit and the position of the image captured by the imaging unit are as close as possible to each other on the predetermined plane. The posture is obtained, and the mutual posture of the projection unit and the imaging unit is obtained through the predetermined plane.

  By performing calibration in this way, the mutual postures of the plurality of projection units and the imaging unit can be easily obtained. In addition, as an image alignment method, for example, there is a method performed on an image projected by a projection unit. On the other hand, in the case of the method described above, images are aligned on a projection destination surface (projection surface). Perform calibration. Therefore, calibration can be performed with higher accuracy.

  The mutual posture estimation unit estimates a variable for determining the mutual posture based on a distance between a position on a predetermined surface of the image projected by the projection unit and a position on the predetermined surface of the image captured by the imaging unit. You may make it do.

  Further, the mutual posture estimation unit updates an evaluation value of a distance between a position on a predetermined plane of the image projected by the projection unit and a position on the predetermined plane of the image captured by the imaging unit, and the updating unit And a variable estimation unit for estimating a variable for determining the mutual posture based on the evaluation value updated by the above.

  And the update part is the distance between the position on the predetermined plane of the pixel to be processed included in the image captured by the imaging section and the position on the predetermined plane of the pixel of the image projected by each projection section that projects the pixel. May be used as the evaluation value.

  Further, the variable estimation unit may include a rotation matrix of the imaging unit, a translation vector of the imaging unit, a rotation matrix of the projection unit, and a translation vector of the projection unit.

  Moreover, you may make it further provide the temporary mutual attitude | position estimation part which estimates the mutual attitude | position of the projection imaging devices which have a projection part and an imaging part as a temporary mutual attitude | position. The mutual posture estimation unit may estimate the mutual posture using the temporary mutual posture estimated by the temporary mutual posture estimation unit as an initial value.

  The temporary mutual posture estimation unit may estimate the temporary mutual posture so that the error in the position of the projection plane of each projection imaging apparatus is minimized.

  Each projection imaging apparatus may further include a projection plane attitude estimation unit that estimates a mutual attitude of the projection unit and the imaging unit with respect to the projection plane on which the image is projected. Then, the temporary mutual posture estimation unit may estimate the temporary mutual posture using the mutual postures of the projection unit and the imaging unit with respect to the projection plane estimated by the projection plane posture estimation unit.

  The projection plane orientation estimation unit may estimate the projection plane by performing triangulation, and may estimate the mutual orientations of the projection unit and the imaging unit with respect to the estimated projection plane.

  Further, the projection plane orientation estimation unit is based on the correspondence between the pixel positions of the projection unit and the imaging unit in the projection imaging device and the correspondence between the pixel positions of the projection unit and the imaging unit between the projection imaging devices, You may make it estimate the mutual attitude | position of the projection part and imaging part with respect to a projection surface. The temporary mutual posture estimation unit may estimate the temporary mutual posture based on the correspondence relationship between the pixel positions of the projection unit and the imaging unit between the projection imaging devices.

  Note that the pixel position between the projection unit and the imaging unit in the projection imaging apparatus is detected by causing the projection plane orientation estimation unit to project a pattern used when obtaining the pixel position correspondence to the projection unit and causing the imaging unit to capture an image. And the mutual orientation of the projection unit and the imaging unit with respect to the projection plane may be estimated based on the correspondence relationship between the projection unit and the imaging unit.

  Moreover, you may make it further provide the imaging part which images the projected image and acquires a captured image. Then, the mutual posture estimation unit estimates the mutual posture between the projection unit and the imaging unit based on the relative positional relationship between the image projected by the projection unit of another device and the image captured by the imaging unit on a predetermined plane. You may make it do.

  Moreover, you may make it further provide the projection part which projects an image. The mutual posture estimation unit may further estimate the mutual posture between the projection unit and the imaging unit.

  In addition, an image processing unit that performs image processing for correcting an image to be projected using a variable according to the mutual posture between the projection unit and the imaging unit estimated by the mutual posture estimation unit may be further provided.

<Projection imaging system>
FIG. 1 illustrates a main configuration example of a projection imaging system to which a control device that is an embodiment of an information processing device to which the present technology is applied is applied. A projection imaging system 100 shown in FIG. 1 is a system that projects an image. The projection imaging system 100 can project one image using a plurality of projection devices (projection imaging devices) as described above, for example. As illustrated in FIG. 1, the projection imaging system 100 includes a control device 101, projection imaging devices 102-1 to 102-4, and communication cables 103-1 to 103-4.

  The projection imaging apparatus 102-1 is connected to the control apparatus 101 via a communication cable 103-1. The projection imaging device 102-2 is connected to the control device 101 by a communication cable 103-2. The projection imaging apparatus 102-3 is connected to the control apparatus 101 via a communication cable 103-3. The projection imaging apparatus 102-4 is connected to the control apparatus 101 via a communication cable 103-4.

  The control device 101 communicates with each of the projection imaging device 102-1 to the projection imaging device 102-4 and controls their operations. For example, the control apparatus 101 causes the projection imaging apparatus 102-1 to the projection imaging apparatus 102-4 to project an image on the screen 104 or to capture an image (projected image) projected on the screen 104.

  For example, the control device 101 can control the projection imaging device 102-1 to the projection imaging device 102-4 and project them together on the screen 104 by cooperating them. In addition, the control device 101 may have, for example, a mutual posture (for example, a rotation component or a translation component) that is a relative position or direction of the projection unit or the imaging unit included in the projection imaging device 102-1 to the projection imaging device 102-4. Calibration can be performed. Further, for example, the control apparatus 101 can correct the image projected by each of the projection imaging apparatus 102-1 to the projection imaging apparatus 102-4 using the calibration result.

  The projection imaging device 102-1 to the projection imaging device 102-4 are devices that are controlled by the control device 101 and project an image on the screen 104 or capture a projection image projected on the screen 104, respectively. The projection imaging device 102-1 to the projection imaging device 102-4 have the same configuration and the same function. Hereinafter, the projection imaging apparatus 102-1 to the projection imaging apparatus 102-4 will be referred to as the projection imaging apparatus 102 when there is no need to distinguish between them.

  As shown in FIG. 1, the projection imaging apparatus 102-1 includes a projection unit 111-1 that projects an image, and an imaging unit 112-1 that captures a subject and obtains a captured image. Similarly, the projection imaging apparatus 102-2 includes a projection unit 111-2 that projects an image, and an imaging unit 112-2 that captures a subject and obtains a captured image. Similarly, the projection imaging apparatus 102-3 includes a projection unit 111-3 that projects an image, and an imaging unit 112-3 that captures a subject and obtains a captured image. Similarly, the projection imaging apparatus 102-4 includes a projection unit 111-4 that projects an image, and an imaging unit 112-4 that captures a subject and obtains a captured image.

  The projection units 111-1 to 111-4 have the same configuration and the same function. Hereinafter, the projection units 111-1 to 111-4 are referred to as the projection unit 111 when it is not necessary to distinguish between them. In addition, the imaging unit 112-1 to the imaging unit 112-4 have the same configuration and the same function. Hereinafter, the imaging units 112-1 to 112-4 will be referred to as imaging units 112 when there is no need to distinguish them from each other.

  That is, the projection imaging apparatus 102 includes a projection unit 111 and an imaging unit 112.

  The communication cables 103-1 to 103-4 are communication media corresponding to the same standards. In the following, the communication cables 103-1 to 103-4 are referred to as communication cables 103 when it is not necessary to distinguish between them.

  The communication cable 103 is, for example, a cable conforming to HDMI (High-Definition Multimedia Interface). Of course, the communication standard supported by the communication cable 103 is arbitrary, and may be a standard other than HDMI (registered trademark) such as a display port (DisplayPort).

  The screen 104 is an example of a surface (projection surface) on which the projection imaging apparatus 102 projects an image. The screen 104 may be a flat surface or a curved surface. For example, irregularities may be formed on the surface of the screen 104. Further, the color of the screen 104 is arbitrary. Instead of the screen 104, a predetermined three-dimensional structure (for example, a building, a wall, a floor, a ceiling, furniture, an accessory, a living thing, etc.) may be used as the projection surface.

  A projected image 105-1 on the screen 104 is an image projected by the projection unit 111-1. Similarly, the projection image 105-2 is an image projected by the projection unit 111-2, the projection image 105-3 is an image projected by the projection unit 111-3, and the projection image 105-4 is projected by the projection unit. 111-4 is the projected image. In the following, the projected images 105-1 to 105-4 are referred to as projected images 105 when there is no need to distinguish them from each other for explanation.

  As shown in FIG. 1, for example, the projection imaging devices 105-1 to 105-4 are arranged on the screen 104, and the projection imaging devices 102 are arranged so as to form one region. Then, the control device 101 controls each projection imaging device 102 to cause the projection image 107 of the input image 106 to be projected over the projection image 105 in that region. That is, each projection unit 111 is controlled by the control device 101 to project a partial image of the input image 106 (an image assigned to itself) so that a projection image 107 is formed on the screen 104. .

  As described above, the control device 101 improves the resolution of the projection image by cooperating the plurality of projection imaging devices 102 (in other words, increases the image size while suppressing the reduction of the resolution (image quality)). be able to. The input image 106 (projected image 107) may be a moving image or a still image.

  In order to realize such processing, the control device 101 needs to accurately grasp the mutual postures of the projection imaging devices 102 (that is, the mutual postures of the projection units 111 and the imaging units 112). Although this mutual attitude can be expressed as an external variable, this external variable may change over time even if calibration is performed when the projection imaging apparatus 102 or the like is installed. If an error occurs in the display position of each projection image 105 due to the secular change of the external variable, there is a possibility that the projection image 107 that should originally be displayed as one image may be displaced or distorted. That is, the image quality of the projected image may be reduced.

  Therefore, the control device 101 calibrates the external variable using the captured image of each imaging unit 112 or the like. By using the captured image of the imaging unit 112 of each projection imaging apparatus 102, the control apparatus 101 can perform calibration more easily and with high accuracy.

<Control device>
FIG. 2 is a diagram illustrating a main configuration example of the control device 101 which is an embodiment of the information processing device to which the present technology is applied.

  As shown in FIG. 2, in the control device 101, a CPU (Central Processing Unit) 151, a ROM (Read Only Memory) 152, and a RAM (Random Access Memory) 153 are connected to each other via a bus 154.

  An input / output interface 160 is also connected to the bus 154. An input unit 161, an output unit 162, a storage unit 163, a communication unit 164, and a drive 165 are connected to the input / output interface 160.

  The input unit 161 includes an input device that accepts external information such as user input. For example, the input unit 161 includes operation buttons, a touch panel, a camera, a microphone, an input terminal, and the like. Various sensors such as an acceleration sensor, an optical sensor, and a temperature sensor may be included in the input unit 161.

  The output unit 162 includes an output device that outputs information such as images and sounds. For example, the output unit 162 includes a display, a speaker, an output terminal, and the like.

  The storage unit 163 includes, for example, a hard disk, a RAM disk, and a nonvolatile memory. The communication unit 164 includes a network interface, for example. For example, the communication unit 164 is connected to the communication cable 103 and communicates with other devices connected via the communication cable 103. The drive 165 drives a removable medium 171 such as a magnetic disk, an optical disk, a magneto-optical disk, or a semiconductor memory.

  For example, the CPU 151 performs various processes by loading a program stored in the storage unit 163 to the RAM 153 via the input / output interface 160 and the bus 154 and executing the program. The RAM 153 also appropriately stores data necessary for the CPU 151 to execute various processes.

  The program executed by the CPU 151 can be recorded on a removable medium 171 as a package medium or the like and provided to the control device 101, for example. In that case, the program can be installed in the storage unit 163 via the input / output interface 160 by attaching the removable medium 171 to the drive 165.

  This program can also be provided to the control apparatus 101 via a wired or wireless transmission medium such as a LAN, the Internet, or digital satellite broadcasting. In that case, the program can be received by the communication unit 164 via a wired or wireless transmission medium and installed in the storage unit 163.

  In addition, this program can be installed in the ROM 152 or the storage unit 163 in advance.

<Functional block>
The CPU 151 of the control device 101 implements various functions by executing programs. FIG. 3 is a functional block diagram illustrating an example of main functions realized by the CPU 151.

  As illustrated in FIG. 3, the CPU 151 includes functional blocks such as a calibration processing unit 201, an image processing unit 202, a projection control unit 203, and an imaging control unit 204.

  The calibration processing unit 201 performs processing related to calibration of external variables of the projection imaging apparatus 102. Examples of variables that affect the correction of the projection image 105 include internal variables and distortion parameters of the projection unit 111 and the imaging unit 112, and external variables that represent the mutual postures of the projection unit 111 and the imaging unit 112 in the projection imaging apparatus 102. (Also referred to as an external variable in the projection imaging apparatus 102), and an external variable (also referred to as an external variable between the projection imaging apparatus 102) indicating the mutual attitude of the projection unit 111 and the imaging unit 112 between the projection imaging apparatus 102 is there.

  The image processing unit 202 performs processing related to the correction of the projection image using the external variable or the like calibrated by the calibration processing unit 201. The projection control unit 203 performs processing related to the control of the projection unit 111 of the projection imaging apparatus 102. The imaging control unit 204 performs processing related to control of the imaging unit 112 of the projection imaging apparatus 102.

  The calibration processing unit 201 includes a pixel position correspondence detection unit 211, a projection plane posture estimation unit 212, a temporary mutual posture estimation unit 213, and a mutual posture re-estimation unit 214.

  The pixel position correspondence detection unit 211 performs processing related to detection of the correspondence relationship between the pixel positions between the projection units 111 and the imaging units 112. The projection plane posture estimation unit 212 performs processing related to estimation of the mutual postures of the projection units 111 and the imaging units 112 with respect to the projection plane. The provisional mutual attitude estimation unit 213 performs processing related to estimation of the provisional mutual attitude between the projection imaging apparatuses 102. The mutual attitude re-estimation unit 214 performs processing related to the estimation of the mutual attitude between the projection unit 111 and the imaging unit 112.

<Appearance of projection imaging device>
FIG. 4 shows an example of the external appearance of the projection imaging apparatus 102. The projection imaging apparatus 102 includes the projection unit 111 and the imaging unit 112 as described above, and a projection port (lens mechanism) for projecting an image or a camera for imaging a subject in the casing. An optical device such as (lens mechanism) is provided. The projection imaging device 102 may be any size device, but may be a portable (small) device, for example. In that case, as shown in FIG. 4, a battery may be provided in the housing of the projection imaging apparatus 102. By providing the battery, the projection imaging apparatus 102 can be driven without an external power source, so that the degree of freedom of the installation position can be improved.

<Projection imaging device>
FIG. 5 is a block diagram illustrating a main configuration example of the projection imaging apparatus 102.

  As illustrated in FIG. 5, the projection imaging apparatus 102 includes a control unit 301, a projection unit 111, an imaging unit 112, an input unit 311, an output unit 312, a storage unit 313, a communication unit 314, and a drive 315.

  The control unit 301 includes, for example, a CPU, a ROM, a RAM, and the like, and controls each processing unit in the apparatus and executes various processes necessary for the control such as image processing.

  The projection unit 111 is controlled by the control unit 301 to perform processing related to image projection. For example, the projection unit 111 projects the image supplied from the control unit 301 to the outside of the projection imaging apparatus 102 (for example, the screen 104). That is, the projection unit 111 realizes a projection function. The projection unit 111 projects an image by using laser light as a light source and scanning the laser light using a MEMS mirror. Of course, the light source of the projection unit 111 is arbitrary, and is not limited to laser light, and may be an LED, xenon, or the like. Details of the projection unit 111 will be described later.

  The imaging unit 112 is controlled by the control unit 301 to capture a subject outside the apparatus, generate a captured image, and supply the captured image to the control unit 301. That is, the imaging unit 112 implements an imaging function. For example, the imaging unit 112 captures a projection image projected on the screen 104 by the projection unit 111.

  The input unit 311 includes an input device that receives external information such as user input. For example, the input unit 311 includes operation buttons, a touch panel, a camera, a microphone, an input terminal, and the like. Various sensors such as an acceleration sensor, an optical sensor, and a temperature sensor may be included in the input unit 311.

  The output unit 312 includes an output device that outputs information such as images and sounds. For example, the output unit 312 includes a display, a speaker, an output terminal, and the like.

  The storage unit 313 includes, for example, a hard disk, a RAM disk, a nonvolatile memory, and the like. The communication unit 314 includes a network interface, for example. For example, the communication unit 314 is connected to the communication cable 103 and communicates with other devices connected via the communication cable 103. The drive 315 drives a removable medium 321 such as a magnetic disk, an optical disk, a magneto-optical disk, or a semiconductor memory.

  For example, the control unit 301 performs various types of processing by loading a program stored in the storage unit 313 into a RAM built therein and executing the program. The RAM also appropriately stores data necessary for the control unit 301 to execute various processes.

  The program executed by the control unit 301 can be recorded on a removable medium 321 as a package medium or the like and provided to the projection imaging apparatus 102, for example. In that case, the program can be installed in the storage unit 313 by attaching the removable medium 321 to the drive 315.

  This program can also be provided to the projection imaging apparatus 102 via a wired or wireless transmission medium such as a LAN, the Internet, or digital satellite broadcasting. In that case, the program can be received by the communication unit 314 via a wired or wireless transmission medium and installed in the storage unit 313.

  In addition, this program can be installed in advance in the ROM or the storage unit 313 built in the control unit 301.

<Projection part>
FIG. 6 is a block diagram illustrating a main configuration example of the projection unit 111. As shown in FIG. 6, the projection unit 111 includes a video processor 351, a laser driver 352, a laser output unit 353-1, a laser output unit 353-2, a laser output unit 353-3, a mirror 354-1, and a mirror 354-. 2, a mirror 354-3, a MEMS (Micro Electro Mechanical Systems) driver 355, and a MEMS mirror 356.

  The video processor 351 holds an image supplied from the control unit 301 and performs necessary image processing on the image. The video processor 351 supplies the projected image to the laser driver 352 and the MEMS driver 355.

  The laser driver 352 controls the laser output unit 353-1 to the laser output unit 353-3 so that the image supplied from the video processor 351 is projected. The laser output units 353-1 to 353-3 output laser beams having different colors (wavelength ranges) such as red, blue, and green. That is, the laser driver 352 controls the laser output of each color so that the image supplied from the video processor 351 is projected. Note that the laser output unit 353-1 to the laser output unit 353-3 are referred to as a laser output unit 353 when it is not necessary to distinguish between them.

  The mirror 354-1 reflects the laser beam output from the laser output unit 353-1 and guides it to the MEMS mirror 356. The mirror 354-2 reflects the laser beam output from the laser output unit 353-2 and guides it to the MEMS mirror 356. The mirror 354-3 reflects the laser beam output from the laser output unit 353-3 and guides it to the MEMS mirror 356. Note that the mirrors 354-1 to 354-3 are referred to as mirrors 354 when there is no need to distinguish them from each other.

  The MEMS driver 355 controls the driving of the mirror of the MEMS mirror 356 so as to project the image supplied from the video processor 351. The MEMS mirror 356 scans the laser light of each color as shown in the example of FIG. 7, for example, by driving a mirror (mirror) mounted on the MEMS according to the control of the MEMS driver 355. This laser light is output from the projection port to the outside of the apparatus, and is irradiated on the screen 104, for example. As a result, the image supplied from the video processor 351 is projected onto the screen 104.

  In the example of FIG. 6, three laser output units 353 are provided to output laser beams of three colors, but the number of laser beams (or the number of colors) is arbitrary. For example, the number of laser output units 353 may be four or more, or two or less. That is, the number of laser beams output from the projection imaging apparatus 102 (projection unit 111) may be two or less, or four. The number of colors of the laser light output from the projection imaging apparatus 102 (projection unit 111) is also arbitrary, and may be two colors or less, or four colors or more. The configurations of the mirror 354 and the MEMS mirror 356 are also arbitrary, and are not limited to the example of FIG. Of course, the scanning pattern of the laser beam is arbitrary.

<Calibration process flow>
Next, the flow of various processes executed in the control apparatus 101 will be described.

  In order to calibrate an external variable of the projection imaging apparatus 102, the calibration processing unit 201 of the control apparatus 101 executes a calibration process. This process can be executed at an arbitrary timing.

  An example of the flow of calibration processing will be described with reference to the flowchart of FIG. When the calibration process is started, the pixel position correspondence detection unit 211 performs a pixel position correspondence detection process in step S101. When the process ends, the projection plane orientation estimation unit 212 performs a projection plane orientation estimation process in step S102. When the process ends, the temporary mutual posture estimation unit 213 performs a temporary mutual posture estimation process in step S103. When the process ends, the mutual posture re-estimation unit 214 performs a mutual posture re-estimation process in step S104. When the process ends, the calibration process ends.

<Flow of pixel position detection processing>
Next, the flow of the pixel position correspondence detection process executed in step S101 of FIG. 8 will be described with reference to the flowchart of FIG. This will be described with reference to FIGS. 10 to 13 as necessary.

  When the pixel position correspondence detection process is started, the pixel position correspondence detection unit 211 selects the projection unit 111 to be processed from the unprocessed projection units 111 in step S121.

  As shown in FIG. 10, the projection imaging devices 102 and the control device 101 of the projection imaging system 100 are arranged so that an image can be projected onto the screen 104. At that time, at least a part of the projection image 105 projected from each projection imaging apparatus 102 is overlapped (overlapped) as in the example of FIG. In this state, the imaging unit 112 of each projection imaging apparatus 102 can capture at least a part of the projection image 105 projected from the other projection imaging apparatus 102. When the projection surface is a curved surface, correction for that is necessary, and the difficulty of calibration becomes high. Therefore, the screen 104 is preferably a flat surface. In a state where the devices are arranged in this way, the projection unit 111 to be processed is selected.

  In step S122, the pixel position correspondence detection unit 211 projects a gray code image on the selected projection unit. In step S123, the pixel position correspondence detection unit 211 causes all the imaging units 112 to capture the projected gray code image.

  An example of a gray code image is shown in FIG. For example, a predetermined pattern image as shown in FIG. 11A is projected while switching in time series from the projector, and each pattern is captured by the camera. When imaging of all patterns is completed, 1 (white) or 0 (black) of each imaging pattern is detected in each pixel of the camera, and 1,0 patterns are decoded as shown in FIG. 11B. As a result, the position of the projector pixel is acquired. Thereby, the correspondence of pixels can be acquired.

  In step S <b> 124, the pixel position correspondence detection unit 211 determines whether or not the gray code images are projected on all the projection units 111. If it is determined that there is an unprocessed projection unit 111, the process returns to step S121, and the subsequent processes are repeated. The processing from step S121 to step S124 is executed for each projection unit 111, and gray code images are projected onto all projection units 111 in step S124 (that is, there is no unprocessed projection unit 111). If it is determined, the process proceeds to step S125.

  In step S125, the pixel position correspondence detection unit 211 detects the correspondence between the pixel position of the projection unit 111 and the pixel position of the imaging unit 112 in each projection imaging apparatus 102 for each corresponding point. That is, a correspondence relationship between pixel positions is detected between the projection unit 111 and the imaging unit 112 having the same configuration of the projection imaging device 102.

  The corresponding point is a pixel of interest used for detecting a correspondence relationship between pixel positions of a gray code image. That is, the pixel of the projection unit 111 that projects a predetermined pixel of interest in the gray code image and the pixel of the imaging unit 112 that images the pixel of interest are identified and associated (correspondence is detected). ). The corresponding points (pixels of interest) may be all the pixels of the gray code image, or may be some pixels (representative pixels) of the gray code image. In that case, the position of the pixel as the representative pixel is arbitrary.

  In step S125, as shown in FIG. 12, paying attention to a predetermined corresponding point 402 of the Gray code 401, the pixel 403 that projects the corresponding point 402 of the projection unit 111 and the corresponding point of the imaging unit 112 A pixel 404 that captures an image 402 is identified and linked (a correspondence between pixel positions is detected). As described above, the same pixel position correspondence may be detected for all the pixels of the gray code image, or the pixel position correspondence may be detected only for the representative pixel.

  In step S <b> 126, the pixel position correspondence detection unit 211 detects the correspondence between the pixel position of the projection unit 111 and the pixel position of the imaging unit 112 between the projection imaging devices 102 for each corresponding point. That is, a correspondence relationship between pixel positions is detected between the projection unit 111 and the imaging unit 112 which are different configurations of the projection imaging device 102.

  The detection of the pixel position correspondence between the projection imaging devices 102 is also performed in the same manner as in the process of step S125. That is, as shown in FIG. 13, paying attention to a predetermined corresponding point 412 of the Gray code 411, for example, a pixel that projects the corresponding point 412 of the projection unit 111-2 of the projection imaging apparatus 102-2, and projection imaging The pixel that captures the corresponding point 412 of the imaging unit 112-1 of the apparatus 102-1 is identified and linked (the correspondence between the pixel positions is detected). Of course, in this case as well, the same pixel position correspondence may be detected for all the pixels of the gray code image, or the pixel position correspondence may be detected only for the representative pixel.

  When the process of step S126 ends, the pixel position correspondence detection process ends, and the process returns to FIG. By performing the pixel position correspondence detection process as described above, the correspondence relationship between the pixel positions can be detected between each projection unit 111 and each imaging unit 112.

  Note that the correspondence between pixel positions cannot be substantially detected between the projection unit 111 and the imaging unit 112 at a position where the image projected by the projection unit 111 cannot be captured. You may make it abbreviate | omit the detection process of the correspondence of a pixel position.

  In the above description, the gray code is used to detect the correspondence between the pixel positions. However, the present invention is not limited to this gray code, and any image can be used as long as the correspondence between the pixel positions can be specified. A pattern image may be used. For example, an image of a pattern in which one of the checker and the four corners of the checker described in Non-Patent Document 1 is missing may be used, or an image of a pattern other than that may be used.

<Flow of projection plane orientation estimation process>
Next, the flow of the projection plane attitude estimation process executed in step S102 of FIG. 8 will be described with reference to the flowchart of FIG. Description will be made with reference to FIG. 15 as necessary.

  When the projection plane attitude estimation process is started, the projection plane attitude estimation unit 212 selects a projection imaging apparatus 102 to be processed from the unprocessed projection imaging apparatuses 102 in step S141.

  In step S142, the projection plane orientation estimation unit 212 detects the pixel position between the projection unit 111 and the imaging unit 112 in the projection imaging apparatus 102 to be processed, which is detected by the pixel position correspondence detection process (FIG. 9). Triangulation is performed using the correspondence relationship and the calibrated internal variable or external variable to obtain the three-dimensional coordinate position of each corresponding point. At that time, triangulation is performed using previously calibrated internal variables and external variables. This external variable is a value that may contain an error (different from the actual) due to aging.

  For example, as illustrated in FIG. 15, a pixel 421 projected by the projection unit 111 and a pixel 422 captured by the imaging unit 112 at a certain corresponding point are obtained based on the processing result of the pixel position corresponding point detection processing. Then, using these pixel positions, internal variables of the projection unit 111 and the imaging unit 112, external variables 423 indicating their mutual postures, and the like, triangulation is performed, and the three-dimensional point coordinates on which the corresponding points are projected 424 is obtained. The three-dimensional point coordinates 424 are obtained by imaging the position where the projection direction of the projection unit 111 (pixel 421) and the imaging direction of the imaging unit 112 (pixel 422) intersect (that is, the position projected by the pixel 421 and the pixel 422). The coordinates coincide with each other (no positional deviation occurs), and such a process is performed for a plurality of corresponding points to obtain a three-dimensional point coordinate group 425.

  In step S143, the projection plane attitude estimation unit 212 obtains a planar screen model based on the three-dimensional coordinate positions of the corresponding points obtained by the process of step S142.

  In the case of the example of FIG. 15, the projection plane orientation estimation unit 212 obtains a plane that best fits the three-dimensional point coordinate group 425 obtained by the process of step S <b> 142 and sets it as the plane screen model 431. The method of estimating the flat screen model is arbitrary, but the flat screen model may be estimated robustly against outliers using a method such as RANSAC (RANDdom Sample Consensus).

  In step S144, the projection plane attitude estimation unit 212 estimates the attitudes of the projection unit 111 and the imaging unit 112 with respect to the flat screen model obtained by the process of step S143.

  In the case of the example in FIG. 15, the posture of the projection unit 111 and the posture of the imaging unit 112 are each represented by the three-dimensional coordinates (arrow 432 and arrow 433) of the flat screen model 431.

  In step S145, the projection plane attitude estimation unit 212 determines whether all the projection imaging apparatuses 102 have been processed. If it is determined that there is an unprocessed projection imaging apparatus 102, the process returns to step S141, and the subsequent processes are repeated. Each process of step S141 to step S145 is executed for each projection imaging apparatus 102, and it is determined in step S145 that all the projection imaging apparatuses 102 have been processed (there is no unprocessed projection imaging apparatus 102). In this case, the projection plane orientation estimation process ends, and the process returns to FIG.

  By executing the projection plane attitude estimation process as described above, the attitude of the projection unit 111 and the attitude of the imaging unit 112 with respect to the projection plane can be estimated.

<Flow of temporary mutual posture estimation processing>
Next, the flow of the temporary mutual posture estimation process executed in step S103 of FIG. 8 will be described with reference to the flowchart of FIG. This will be described with reference to FIGS. 17 and 18 as necessary.

  When the temporary mutual posture estimation process is started, the temporary mutual posture estimation unit 213 selects a projection imaging device pair to be processed from the unprocessed projection imaging device pairs in step S161. The projection imaging device pair is any two projection imaging devices 102 selected from the group of projection imaging devices 102 included in the projection imaging system 100. That is, the temporary mutual posture estimation unit 213 sets two projection imaging apparatuses 102 as processing targets. Note that combinations in which the projected images do not overlap each other (there is no portion overlapping the projected images) may be omitted.

  In step S162, the provisional mutual posture estimation unit 213 estimates a relative posture that minimizes the sum of squares of errors of the three-dimensional coordinates of the flat screens of the processing target projection imaging device pair.

  For example, in the case of the example shown in FIG. 17, the provisional mutual attitude estimation unit 213 is a flat screen model corresponding to the pixel position 453 of the imaging unit 112-1 of the projection imaging apparatus 102-1 as indicated by a double arrow 451. The three-dimensional point coordinates 424-1 on 431-1 and the three-dimensional on the flat screen model 431-2 corresponding to the pixel position 452 of the projection unit 111-2 of the projection imaging apparatus 102-2 corresponding to the pixel position 453 The condition that the sum of squares of the error of the position of the point coordinates 424-2 on the flat screen is minimized is solved. Such processing is performed for each three-dimensional point coordinate 424 of the three-dimensional coordinate group 425-1 and the three-dimensional coordinate group 425-2 in FIG. At this time, since the flat screen is assumed, the external variables to be calculated are only the roll rotation component and the X and Y translation components (the pitch and yaw are assumed because the flat screen is shared). The rotation component and Z translation component do not change).

  Accordingly, as illustrated in FIG. 18, the three-dimensional coordinates 471 of the projection unit 111-1 and the imaging unit 112 when the positions of the flat screen model 431-1 and the flat screen model 431-2 are approximated as much as possible. -1 three-dimensional coordinates 472, three-dimensional coordinates 473 of the projection unit 111-2, and three-dimensional coordinates 474 of the imaging unit 112-2 (that is, the projection unit 111 and the imaging unit 112 of the two selected projection imaging devices 102). Each three-dimensional coordinate) is obtained. As a result, an external variable between the two selected projection imaging apparatuses 102 (in the case of the examples of FIGS. 17 and 18, the projection imaging apparatus 102-1 and the projection imaging apparatus 102-2) is obtained.

  In step S163, the temporary mutual posture estimation unit 213 determines whether or not all the projection imaging device pairs have been processed. If it is determined that there is an unprocessed projection imaging device pair, the process returns to step S161, and the subsequent processes are repeated. That is, the process of step S162 is performed for all two combinations in the projection imaging apparatus 102 group included in the projection imaging system 100. If it is determined in step S163 that processing has been performed for all projection imaging device pairs (there are no unprocessed projection imaging device pairs), the temporary mutual posture estimation processing ends, and the processing is illustrated in FIG. Return to.

<Flow of mutual posture re-estimation process>
Next, the flow of the mutual attitude re-estimation process executed in step S104 of FIG. 8 will be described with reference to the flowchart of FIG. Description will be made with reference to FIG. 20 as necessary.

  In the mutual posture re-estimation process, for example, as shown in FIG. 20, the external variable is obtained so as to minimize the error (sum of squared errors) of each corresponding point in a predetermined flat screen model 481.

  For example, for the corresponding point 491, the position 491-2 of the corresponding point 491 on the flat screen model 481 viewed from the imaging unit 112-1, and the position 491-1 of the corresponding point 491 on the flat screen model 481 viewed from the projection unit 111-1. , The position 491-3 on the flat screen model 481 of the corresponding point 491 viewed from the projection unit 111-2 is obtained, and the distance between the position 491-2 and the position 491-1, the position 491-2 and the position 491-3, Distances are obtained. And the position on the three-dimensional coordinate of the plane screen model 481 of the projection part 111-1, the imaging part 112-1, and the projection part 111-2 is each estimated so that those distance may become the minimum.

  Similarly, for the corresponding point 492, the position 492-2 of the corresponding point 492 viewed from the imaging unit 112-2 on the flat screen model 481, and the position 492 of the corresponding point 492 viewed from the projection unit 111-1 on the flat screen model 481. 1. A position 492-3 of the corresponding point 492 viewed from the projection unit 111-2 on the flat screen model 481 and a position 492-4 of the corresponding point 492 viewed from the projection unit 111-3 on the flat screen model 481 are respectively obtained. The distance between the position 492-2 and the position 492-1, the distance between the position 492-2 and the position 492-3, and the distance between the position 492-2 and the position 492-4 are obtained. Then, the positions of the projection unit 111-1, the projection unit 111-2, the imaging unit 112-2, and the projection unit 111-3 on the three-dimensional coordinates of the planar screen model 481 are set so that those distances are minimized. Presumed.

  Similarly, for the corresponding point 493, the position 493-2 of the corresponding point 493 on the flat screen model 481 viewed from the imaging unit 112-3, and the position 493 of the corresponding point 493 on the flat screen model 481 viewed from the projection unit 111-2. 1. A position 493-3 on the flat screen model 481 of the corresponding point 493 viewed from the projection unit 111-3 is obtained, and the distance between the position 493-2 and the position 493-1, the position 493-2 and the position 493-3, respectively. And the distance to each other. And the position on the three-dimensional coordinate of the plane screen model 481 of the projection part 111-2, the imaging part 112-3, and the projection part 111-3 is each estimated so that those distance may become the minimum.

  Similarly, for the corresponding point 494, the position 494-2 of the corresponding point 494 viewed from the imaging unit 112-4 on the flat screen model 481 and the position 494 of the corresponding point 494 viewed from the projection unit 111-3 on the flat screen model 481. 1. The position 494-3 on the flat screen model 481 of the corresponding point 494 viewed from the projection unit 111-4 is obtained, and the distance between the position 494-2 and the position 494-1, and the position 494-2 and the position 494-3. And the distance to each other. And the position on the three-dimensional coordinate of the plane screen model 481 of the projection part 111-3, the imaging part 112-4, and the projection part 111-4 is each estimated so that those distance may become the minimum.

  As described above, an error is obtained for each corresponding point, the sum of the least squares thereof is calculated, and the cubic of the predetermined flat screen model 481 of each projection unit 111 and each imaging unit 112 so that the value is minimized. The positions on the original coordinates (in the example of FIG. 20, arrows 501-1 to 501-8) are estimated.

  Then, from the position (posture), an external variable (double arrow 502-1 to double arrow 502-4 in the example of FIG. 20) indicating the mutual posture of the projection unit 111 and the imaging unit 112 with higher accuracy is estimated. Is done.

  Returning to FIG. 19, when the mutual posture re-estimation process is started, the mutual posture re-estimation unit 214 sets the posture of each projection imaging apparatus 102 obtained by the processing of step S <b> 103 of FIG. 8 as an initial value in step S <b> 181. The evaluation value is calculated.

  The evaluation value (error evaluation value E) is calculated using, for example, the following equation (1).

・・・（１）

... (1)

here,
M: Number of imaging units / number of projection units
N: Number of pixels in the imaging unit
X _cn : projected coordinates (X _cn , Y _cn ) of the pixel n of the imaging unit c
A _Cc , D _Cc , R _Cc , T _Cc functions
X _cnp : On-screen projection coordinates (X _cnp , Y _cnp ) of the pixel of the projection unit p corresponding to the pixel n of the imaging unit c
A _Cc , D _Cc , R _Cc , T _Cc , A _Pp , D _Pp , R _Pp , T _Pp functions
I _cnp : _Appearance flag 0/1 (1 when there is a pixel of the projection unit p corresponding to the pixel n of the imaging unit c)
And

The variables A _Cc , D _Cc , R _Cc , T _Cc , A _Pp , D _Pp , R _Pp and T _Pp are as follows.
A _Cc : Internal variable of imaging unit c (fixed value during this calibration)
D _Cc : Distortion parameter of imaging unit c (fixed value at the time of calibration)
R _Cc : rotation matrix of the imaging unit c (external variable)
T _Cc : Translation vector of the imaging unit c (external variable)
A _Pp : Internal variable of projection part p (fixed value at the time of this calibration)
D _Pp : distortion parameter of the projection part p (fixed value at the time of this calibration)
R _Pp : rotation matrix of projection part p (external variable)
T _Pp : Translation vector of projection part p (external variable)

  In this calibration, R and T that minimize the error evaluation value E in Equation (1) are obtained (the idea of bundle adjustment). However, the Z coordinate value and the roll component of the projection unit 111 or the imaging unit 112 serving as a reference are fixed. This can be achieved by forcing the value to 0 when calculating the Jacobian of the target parameter.

  In step S182, the mutual attitude re-estimation unit 214 calculates the update value Δα and calculates the temporary update value α ′.

  The mutual attitude re-estimation unit 214 calculates the temporary update value α ′ using the error evaluation value E, the Jacobian matrix (Jacobian), and the Hessian matrix (Hessian) described above.

The Jacobian matrix (Jacobian) is obtained approximately from the amount of change in the error e _cn for each pixel of the image pickup unit by shifting each of the three rotational component components and the three translation component components by a minute amount. The Jacobian matrix of the rotation component is the rotation matrix when the parameter of infinitesimal rotation ω = (ω ₁ ω ₂ ω ₃ ) ^T is moved by a small amount of Δω, and R (ω), and the current rotation matrix is When defined as R, it is represented by the following formula (2).

・・・（２）

... (2)

The translational component Jacobian matrix (Jacobian) is expressed by the following equation (3) when each of the translation parameters T = (T _x T _y T _z ) ^T is moved by a minute amount of ΔT.

・・・（３）

... (3)

Then, a Hessian matrix (Hessian) is obtained approximately from the Jacobian matrix (Jacobian) and the error e _cn . When the Jacobian matrix (Jacobian) in the above equations (2) and (3) is ∂e _cn / ∂α _i , (i = 0, ..., 5), the Hessian matrix (Hessian) is expressed by the following equation ( 4).

・・・（４）

... (4)

  Furthermore, H ′ obtained by weighting the diagonal elements can be expressed as the following equation (5).

・・・（５）

... (5)

  Here, w is a weight parameter related to the step width of the search position. The larger the weight, the closer to the update step when the search space is first-order approximated, and the smaller the weight, the update when the search space is second-order approximated Get closer to the step.

  Considering this equation (5), the update value α is obtained. In this case, the update value α is expressed as in the following formula (6).

・・・（６）

... (6)

  In step S183, the mutual attitude re-estimation unit 214 updates the evaluation value, the update value, and the weight according to the change in the evaluation value. For example, if ΔE is negative, α and E are updated and w is multiplied by 1/10. If ΔE is not negative, multiply w by 10 (similar to the general Levenberg-Marqurdt method).

  In step S184, the mutual attitude re-estimation unit 214 determines whether each process of step S182 and step S183 has been repeated a predetermined number of times. If it is determined that the predetermined number of times has not been reached, the process returns to step S182 and the subsequent processes are repeated.

  If it is determined in step S184 that the processes in steps S182 and S183 have been repeated a predetermined number of times, the process proceeds to step S185.

  In step S185, the mutual posture re-estimation unit 214 detects the mutual postures of all the projection units 111 and the imaging units 112, and uses external variables (external variables in the projection imaging device 102 and external variables between the projection imaging devices 102). )

  When the process of step S185 ends, the mutual attitude re-estimation process ends, and the process returns to FIG.

  By executing each process as described above, the control apparatus 101 can calibrate the positional relationship between the projection unit 111 and the imaging unit 112 more easily and with high accuracy. By using the projection imaging apparatus 102 that has been calibrated in this way, the projection imaging system 100 can accurately perform geometric correction not only on a flat screen but also on an object with a curved surface or an arbitrary shape surface. Will be able to.

<Configuration example of projection imaging system>
In the above description, in the projection imaging system 100, the control device 101 and each projection imaging device 102 have been described as being connected via the communication cable 103. However, the connection method between the control device 101 and each projection imaging device 102 is as follows. It is optional as long as communication is possible. For example, it may be connected via the network 601 as in the example shown in FIG.

  A network 601 is a communication network serving as a communication medium between the control device 101 and the projection imaging device 102. The network 601 may be any communication network, a wired communication network, a wireless communication network, or both of them. For example, it may be a wired LAN, a wireless LAN, a public telephone line network, a so-called 3G line, a wide area communication network for a wireless mobile body such as a 4G line, or the Internet, or a combination thereof. The network 601 may be a single communication network or a plurality of communication networks. Further, for example, a part of the network 601 may be configured by a communication cable of a predetermined standard such as a USB (Universal Serial Bus) cable or an HDMI (registered trademark) cable.

  Further, the above-described processing related to calibration may be performed by a device other than the control device 101. For example, the projection imaging apparatus 102 may perform this. In the example of FIG. 22, the projection imaging apparatus 102 performs the above-described processing related to calibration. In this case, an input image is input to each projection imaging device 102, and each projection imaging device 102 corrects the image. Each projection imaging apparatus 102 is connected to each other by a communication cable such as an HDMI (registered trademark) cable, and information such as correction information and sensor data such as a captured image is shared among the projection imaging apparatuses 102. It is made to be able to. Accordingly, any projection imaging apparatus 102 can perform the above-described processing relating to calibration. For example, any one of the projection imaging apparatus 102-1 to the projection imaging apparatus 102-4 may perform the above-described processing relating to calibration, and the calibration result may be shared with other projection imaging apparatuses 102, The projection imaging apparatus 102 may cooperate to perform the above-described processing relating to calibration, and the calibration result may be shared with other projection imaging apparatuses 102. In the case of such a configuration, the control device 101 can be omitted.

  In the above description, the projection imaging system 100 has been described as having four projection imaging apparatuses 102. However, the number of projection imaging apparatuses 102 constituting the projection imaging system 100 is arbitrary, and is three or less. Or five or more.

  Further, like the projection imaging apparatus 612 shown in FIG. 23, it may be connected to the network 601 (the control apparatus 101, other projection imaging apparatus 102, etc.) via another information processing apparatus 611. The projection imaging device 612 is the same device as the projection imaging device 102 described above, but this projection imaging device has a communication function such as a mobile phone, a smartphone, a tablet computer, and a notebook computer. It is connected to the network 601 via the information processing device 611. The projection imaging device 612 is controlled and driven by the information processing device 611. By doing so, the information processing apparatus 611 having originally high processing capability can perform the processing related to communication and the processing related to the control of projection and imaging. Therefore, functions (information processing capability and the like) necessary for the projection imaging apparatus 612 can be performed. ) Can be suppressed, and an increase in cost can be suppressed.

  Further, the function of the projection imaging apparatus 102 described above may be realized as a module or the like (that is, a component unit). The information processing apparatus 613 in FIG. 23 is an information processing apparatus with originally high processing capability, such as a mobile phone, a smartphone, a tablet computer, and a notebook computer, and a module having the function of the projection imaging apparatus 102 described above. Is incorporated. That is, the information processing device 613 is a device having both functions of the information processing device 611 and the projection imaging device 612. The projection imaging apparatus 102 can also be realized as such an information processing apparatus.

  Further, as in the example shown in FIG. 23, devices having different functions may be mixed as the projection imaging apparatus 102.

  Further, as in the example illustrated in FIG. 24, the projection imaging system 100 includes the projection device 621, the projection device 623, and the imaging unit 112 that include the projection unit 111 but do not include the imaging unit 112. However, an imaging device 622 or the like that does not have the projection unit 111 may be included. Further, a plurality of projection units 111 and imaging units 112 may be provided in one device. For example, the projection imaging device 624 includes a projection unit 111A, a projection unit 111B, and an imaging unit 112. The external variables of the projection imaging apparatus 624 having such a configuration can perform the above-described calibration as in the case of the projection imaging apparatus 102. Furthermore, in the entire projection imaging system 100, the number of projection units 111 and the number of imaging units 112 do not have to match.

  The series of processes described above can be executed by hardware or can be executed by software. When the above-described series of processing is executed by software, a program constituting the software is installed from a network or a recording medium.

  For example, as shown in FIGS. 2 and 5, the recording medium is distributed by a removable medium 171 or a removable medium 321 on which a program is recorded, which is distributed to distribute a program to a user, separately from the apparatus main body. Composed. The removable medium 171 and the removable medium 321 include a magnetic disk (including a flexible disk) and an optical disk (including a CD-ROM and a DVD). Furthermore, a magneto-optical disk (including MD (Mini Disc)), a semiconductor memory, and the like are also included.

  In this case, for example, in the control device 101, the program can be installed in the storage unit 163 by mounting the removable medium 171 in the drive 165. For example, in the projection imaging apparatus 102, the program can be installed in the storage unit 313 by mounting the removable medium 321 in the drive 315.

  This program can also be provided via a wired or wireless transmission medium such as a local area network, the Internet, or digital satellite broadcasting. In that case, for example, in the control device 101, the program can be received by the communication unit 164 and installed in the storage unit 163. For example, in the projection imaging apparatus 102, the program can be received by the communication unit 314 and installed in the storage unit 313.

  In addition, this program can be installed in advance in a storage unit, a ROM, or the like. For example, in the case of the control device 101, the program can be installed in advance in the storage unit 163, the ROM 153, or the like. For example, in the case of the projection imaging apparatus 102, the program can be installed in advance in the storage unit 313, the ROM in the control unit 301, or the like.

  The program executed by the computer may be a program that is processed in time series in the order described in this specification, or in parallel or at a necessary timing such as when a call is made. It may be a program for processing.

  Further, in the present specification, the step of describing the program recorded on the recording medium is not limited to the processing performed in chronological order according to the described order, but may be performed in parallel or It also includes processes that are executed individually.

  Moreover, the process of each step mentioned above can be performed in each apparatus mentioned above or arbitrary apparatuses other than each apparatus mentioned above. In that case, the device that executes the process may have the functions (functional blocks and the like) necessary for executing the process described above. Information necessary for processing may be transmitted to the apparatus as appropriate.

  In this specification, the system means a set of a plurality of components (devices, modules (parts), etc.), and it does not matter whether all the components are in the same housing. Accordingly, a plurality of devices housed in separate housings and connected via a network and a single device housing a plurality of modules in one housing are all systems. .

  In addition, in the above description, the configuration described as one device (or processing unit) may be divided and configured as a plurality of devices (or processing units). Conversely, the configurations described above as a plurality of devices (or processing units) may be combined into a single device (or processing unit). Of course, a configuration other than that described above may be added to the configuration of each device (or each processing unit). Furthermore, if the configuration and operation of the entire system are substantially the same, a part of the configuration of a certain device (or processing unit) may be included in the configuration of another device (or other processing unit). .

  The preferred embodiments of the present disclosure have been described in detail above with reference to the accompanying drawings, but the technical scope of the present disclosure is not limited to such examples. It is obvious that a person having ordinary knowledge in the technical field of the present disclosure can come up with various changes or modifications within the scope of the technical idea described in the claims. Of course, it is understood that it belongs to the technical scope of the present disclosure.

  For example, the present technology can take a configuration of cloud computing in which one function is shared by a plurality of devices via a network and jointly processed.

  In addition, each step described in the above flowchart can be executed by being shared by a plurality of apparatuses in addition to being executed by one apparatus.

  Further, when a plurality of processes are included in one step, the plurality of processes included in the one step can be executed by being shared by a plurality of apparatuses in addition to being executed by one apparatus.

  In addition, the present technology is not limited to this, and any configuration mounted on such a device or a device constituting the system, for example, a processor as a system LSI (Large Scale Integration), a module using a plurality of processors, a plurality of It is also possible to implement as a unit using other modules, a set obtained by further adding other functions to the unit (that is, a partial configuration of the apparatus), and the like.

In addition, this technique can also take the following structures.
(1) A mutual attitude that is a relative position and direction of the projection unit and the imaging unit on a predetermined plane based on a relative positional relationship between an image projected by the projection unit and the image captured by the imaging unit. An information processing apparatus comprising a mutual posture estimation unit for estimating a position.
(2) The mutual posture estimation unit may be configured based on a distance between a position on the predetermined surface of the image projected by the projection unit and a position on the predetermined surface of the image captured by the imaging unit. The information processing apparatus according to (1), wherein a variable that determines a mutual posture is estimated.
(3) The mutual posture estimation unit
An update unit that updates an evaluation value of a distance between the position on the predetermined plane of the image projected by the projection unit and the position on the predetermined plane of the image captured by the imaging unit;
The information processing apparatus according to (2), further comprising: a variable estimation unit that estimates a variable that determines the mutual attitude based on the evaluation value updated by the update unit.
(4) The update unit includes the position of the processing target pixel included in the image captured by the imaging unit on the predetermined plane, and the pixel of the image projected by each projection unit that projects the pixel. The information processing apparatus according to (3), wherein a mean square of a distance to a position on a predetermined surface is the evaluation value.
(5) The information according to (3) or (4), wherein the variable estimation unit includes a rotation matrix of the imaging unit, a translation vector of the imaging unit, a rotation matrix of the projection unit, and a translation vector of the projection unit. Processing equipment.
(6) It further includes a temporary mutual posture estimation unit that estimates a mutual posture between the projection imaging devices having the projection unit and the imaging unit as a temporary mutual posture,
The information processing apparatus according to any one of (1) to (5), wherein the mutual posture estimation unit estimates the mutual posture using the temporary mutual posture estimated by the temporary mutual posture estimation unit as an initial value.
(7) The information processing apparatus according to (6), wherein the temporary mutual attitude estimation unit estimates the temporary mutual attitude so that an error in a position of a projection plane of each projection imaging apparatus is minimized.
(8) For each projection imaging apparatus, further comprising a projection plane attitude estimation unit that estimates a mutual attitude of the projection unit and the imaging unit with respect to a projection plane on which the image is projected,
The temporary mutual posture estimation unit estimates the temporary mutual posture using the mutual postures of the projection unit and the imaging unit with respect to the projection plane estimated by the projection plane posture estimation unit. (6) or (7 ).
(9) The information according to (8), wherein the projection plane attitude estimation unit estimates the projection plane by performing triangulation, and estimates a mutual attitude of the projection unit and the imaging unit with respect to the estimated projection plane. Processing equipment.
(10) The projection plane orientation estimation unit includes a correspondence relationship between pixel positions of the projection unit and the imaging unit in the projection imaging device, and a pixel position between the projection unit and the imaging unit between the projection imaging devices. The information processing apparatus according to (8) or (9), wherein a mutual attitude of the projection unit and the imaging unit with respect to the projection plane is estimated based on the corresponding relationship.
(11) The information according to (10), wherein the temporary mutual posture estimation unit estimates the temporary mutual posture based on a correspondence relationship between pixel positions of the projection unit and the imaging unit between the projection imaging devices. Processing equipment.
(12) The projection plane orientation estimation unit may detect the projection unit in the projection imaging apparatus detected by causing the projection unit to project a pattern to be used when obtaining the pixel position correspondence and causing the imaging unit to capture an image. Based on the correspondence of the pixel position with the imaging unit and the correspondence of the pixel position between the projection unit and the imaging unit between the projection imaging devices, the projection unit and the imaging unit are mutually connected to the projection plane. The information processing apparatus according to (10) or (11), wherein the posture is estimated.
(13) It further includes an imaging unit that captures the projected image and obtains a captured image,
The mutual posture estimation unit is configured to determine whether the projection unit and the imaging unit are based on a relative positional relationship between an image projected by a projection unit of another device and the image captured by the imaging unit on the predetermined plane. The information processing apparatus according to any one of (1) to (12), wherein a mutual posture is estimated.
(14) It further includes a projection unit that projects an image,
The information processing apparatus according to (13), wherein the mutual posture estimation unit further estimates a mutual posture between the projection unit and the imaging unit.
(15) The image processing unit further includes an image processing unit that performs image processing for correcting an image to be projected using a variable according to the mutual posture between the projection unit and the imaging unit estimated by the mutual posture estimation unit. Thru | or the information processing apparatus in any one of (14).
(16) A mutual attitude that is a relative position and direction of the projection unit and the imaging unit based on a relative positional relationship between an image projected by the projection unit and the image captured by the imaging unit on a predetermined plane. Information processing method to estimate

  DESCRIPTION OF SYMBOLS 100 Projection imaging system, 101 Control apparatus, 102 Projection imaging apparatus, 103 Communication cable, 104 Screen, 111 Projection part, 112 Imaging part, 151 CPU, 201 Calibration processing part, 202 Image processing part, 203 Projection control part, 204 Imaging Control unit, 211 pixel position correspondence detection unit, 212 projection plane posture estimation unit, 213 provisional mutual posture estimation unit, 214 mutual posture re-estimation unit, 301 control unit, 351 video processor, 352 laser driver, 353 laser output unit, 354 mirror , 355 MEMS driver, 356 MEMS mirror, 601 network, 611 information processing device, 612 projection imaging device, 613 information processing device, 621 projection device, 622 imaging device, 623 projection device, 624 throwing Shadow imaging device

Claims (16)

Based on a relative positional relationship between an image projected by the projection unit and the image captured by the imaging unit on a predetermined plane, a mutual attitude that is a relative position and direction between the projection unit and the imaging unit is estimated. An information processing apparatus comprising a mutual posture estimation unit. The mutual posture estimation unit calculates the mutual posture based on a distance between a position on the predetermined surface of the image projected by the projection unit and a position on the predetermined surface of the image captured by the imaging unit. The information processing apparatus according to claim 1, wherein a variable to be determined is estimated. The mutual posture estimation unit
An update unit that updates an evaluation value of a distance between the position on the predetermined plane of the image projected by the projection unit and the position on the predetermined plane of the image captured by the imaging unit;
The information processing apparatus according to claim 2, further comprising: a variable estimation unit that estimates a variable that determines the mutual attitude based on the evaluation value updated by the update unit. The update unit includes the position of the processing target pixel included in the image captured by the imaging unit on the predetermined surface, and the predetermined surface of the pixel of the image projected by each projection unit that projects the pixel. The information processing apparatus according to claim 3, wherein the evaluation value is a mean square of a distance to a position at. The information processing apparatus according to claim 3, wherein the variable estimation unit includes a rotation matrix of the imaging unit, a translation vector of the imaging unit, a rotation matrix of the projection unit, and a translation vector of the projection unit. A provisional mutual attitude estimation unit that estimates a mutual attitude between the projection imaging apparatus having the projection unit and the imaging unit as a provisional mutual attitude;
The information processing apparatus according to claim 1, wherein the mutual posture estimation unit estimates the mutual posture using the temporary mutual posture estimated by the temporary mutual posture estimation unit as an initial value. The information processing apparatus according to claim 6, wherein the temporary mutual attitude estimation unit estimates the temporary mutual attitude so that an error in a position of a projection plane of each projection imaging apparatus is minimized. For each projection imaging apparatus, the projection imaging apparatus further includes a projection plane attitude estimation unit that estimates a mutual attitude of the projection unit and the imaging unit with respect to a projection plane on which the image is projected,
The temporary mutual posture estimation unit estimates the temporary mutual posture using the mutual postures of the projection unit and the imaging unit with respect to the projection plane estimated by the projection plane posture estimation unit. Information processing device. The information processing apparatus according to claim 8, wherein the projection plane attitude estimation unit estimates the projection plane by performing triangulation, and estimates a mutual attitude of the projection unit and the imaging unit with respect to the estimated projection plane. The projection plane orientation estimation unit includes a correspondence relationship between pixel positions of the projection unit and the imaging unit in the projection imaging device, and a correspondence relationship between pixel positions of the projection unit and the imaging unit between the projection imaging devices. The information processing apparatus according to claim 8, wherein the mutual attitude of the projection unit and the imaging unit with respect to the projection plane is estimated based on The information processing apparatus according to claim 10, wherein the temporary mutual posture estimation unit estimates the temporary mutual posture based on a correspondence relationship between pixel positions of the projection unit and the imaging unit between the projection imaging devices. The projection plane posture estimation unit detects the projection unit and the imaging unit in the projection imaging apparatus that are detected by causing the projection unit to project a pattern used when determining pixel position correspondence and causing the imaging unit to capture an image. Based on the correspondence relationship of the pixel position with respect to the projection imaging device and the correspondence relationship of the pixel position between the projection unit and the imaging unit between the projection imaging devices, the mutual orientation of the projection unit and the imaging unit with respect to the projection plane is estimated The information processing apparatus according to claim 10. An imaging unit that captures the projected image and obtains the captured image;
The mutual posture estimation unit is configured to determine whether the projection unit and the imaging unit are based on a relative positional relationship between an image projected by a projection unit of another device and the image captured by the imaging unit on the predetermined plane. The information processing apparatus according to claim 1, wherein the mutual attitude is estimated. A projection unit for projecting an image;
The information processing apparatus according to claim 13, wherein the mutual posture estimation unit further estimates a mutual posture between the projection unit and the imaging unit. The image processing part which performs the image process which correct | amends the image to project using the variable according to the mutual attitude | position of the said projection part estimated by the said mutual attitude | position estimation part and the said imaging part is provided. Information processing device. Based on a relative positional relationship between an image projected by the projection unit and the image captured by the imaging unit on a predetermined plane, a mutual attitude that is a relative position and direction between the projection unit and the imaging unit is estimated. Information processing method.

Images