JPWO2018173183A1 - Optical apparatus control apparatus, optical apparatus control method, and optical apparatus control program - Google Patents

Abstract

The optical device control apparatus (1) is an apparatus for controlling a plurality of optical devices (3) for projecting a plurality of images on an object (51) and a plurality of movement mechanisms (4) for changing the arrangement of these Projection from the plurality of optical devices (3) based on the physical space information acquired by the physical space information acquisition unit (5) when the plurality of optical devices (3) are projecting the image onto the object It is judged whether a shadow (53) is formed, which is an area where the captured image does not reach the object, and information indicating the arrangement of the object (52) forming a shadow when the shadow is detected is obtained An arrangement calculation unit (12) for calculating the arrangement of a plurality of optical devices (3) in which no shadow is formed based on information indicating the arrangement of the object (52) and the object estimation unit (11) 12) based on the arrangement of the plurality of optical devices (3) calculated by A moving mechanism controller which controls a plurality of moving mechanism (4) as a plurality of optical devices to a position where the shadow is not formed (3) is arranged and (13).

Description

  The present invention relates to an optical device control apparatus, an optical device control method, and an optical device control program for controlling an optical device for projecting an image or capturing an image and a moving mechanism for changing the arrangement of the optical device.

  A projector (image projection apparatus) is an optical apparatus that displays an image on an object by applying projection light emitted from a projection optical system to the object, and a camera (imaging apparatus) changes the object to the camera It is an optical device that receives light traveling by an imaging surface and acquires image data based on an object. In other words, the projector displays an image by projecting a two-dimensional image based on the image data into the three-dimensional real space, and the camera projects the three-dimensional real space image onto the two-dimensional imaging surface To generate image data. Thus, the projector and the camera are basically similar optical models except that the traveling direction of the light to be handled is opposite.

  In such an optical model, when an object (light shield) is present in the space between the optical device and the object, the shadow of the object is formed in the image projected on the object, and A part is missing or an object is included in the image projected onto the imaging plane, and a part of the image of the object is missing.

  In order to provide a viewer (user) with a projection image, Patent Document 1 discloses an air flight mechanism unit, a projector unit, a camera, and a number detection unit that detects the number of viewers from a photographed image of the camera. According to the present invention, there is proposed a projection apparatus including: a control unit that determines a place to display a projection image according to the number of viewers and controls a projector unit to display the projection image at the determined place.

  Further, Patent Document 2 proposes an optical projection control device that causes an optical image to be projected at illuminances respectively instructed to a plurality of projectors. In this apparatus, captured image data (in the absence of a light shield) obtained by projecting an optical image on the screen in the absence of an object (light shield) between the screen and the projector (when the light shield is not present) The differential image data corresponding to the difference of the light amount with the image data is acquired, and the projector in which a part of the projection image is shielded is detected based on the difference image data, and the projection of the projector whose projection image is not shielded Correct the illumination of the image.

JP, 2016-208255, A (For example, claim 1) Patent No. 5596427 (For example, claim 3)

  However, although the device described in Patent Document 1 describes the function of changing the projection position, it does not describe the technique of preventing the shadow of the object from being formed in the projection image.

  Further, the device described in Patent Document 2 can remove the shadow of the light shield to some extent by controlling the pixel values of the projected image, but has a problem that the range in which the shadow can be removed is narrow. In particular, when the light shield (eg, the user's hand) moves between the optics and the object, it may be difficult to remove the shadow.

  The present invention has been made to solve the above-mentioned problems, so that even if there is an object between the optical device and the object, the shadow of the light shield is not formed in the projected image, or An optical device control apparatus, an optical device control method, and an optical device control program capable of controlling an optical device and a moving mechanism so that a light shield is not included in a captured image.

  An optical device control device according to an aspect of the present invention is a device that controls a plurality of optical devices that project a plurality of images on an object and a plurality of movement mechanisms that change the arrangement of the plurality of optical devices. The object projected from the plurality of optical devices is based on the physical space information acquired by the physical space information acquiring unit when the plurality of optical devices are projecting an image onto the object. It is determined whether or not a shadow is formed which is an area that does not reach above, and an object estimation unit that acquires information indicating the arrangement of the object that forms the shadow when the shadow is detected; the arrangement of the object And the shadow is formed based on the arrangement of the plurality of optical devices calculated by the arrangement calculation unit, the arrangement calculation unit calculating the arrangement of the plurality of optical devices in which the shadow is not formed based on the information indicating Not in a position Serial and having a moving mechanism controller which a plurality of optical devices for controlling said plurality of moving mechanisms to be disposed.

  An optical device control apparatus according to another aspect of the present invention is a plurality of optical devices for acquiring a plurality of images of the object by capturing an object, and a plurality of moving mechanisms for changing the arrangement of the plurality of optical devices. And, based on the plurality of images acquired by the plurality of optical devices, whether or not there are objects for which the plurality of optical devices form a non-imaging area on the object. , And the plurality of optical devices in which the non-imaging area is not formed based on the information indicating the arrangement of the object and an object estimation unit that acquires information indicating the arrangement of the object when the object is detected. And the plurality of optical devices are arranged at positions where the non-photographing area is not formed based on the arrangement calculation unit which calculates the arrangement of the plurality of optical devices and the arrangement of the plurality of optical devices calculated by the arrangement calculation unit. Multiple transfers And having a moving mechanism controller which controls the mechanism.

  According to the present invention, even when there is a light shield between the optical device and the object, the shadow of the object is formed on the projected image of the optical device by controlling the arrangement of the plurality of optical devices. It is possible to prevent the object from being included in the captured image of the optical device or not.

FIG. 1 is a functional block diagram schematically showing a configuration of an optical device control apparatus according to Embodiment 1 of the present invention. (A) to (c) are perspective views schematically showing a projector and a moving mechanism as an optical device controlled by the optical device control apparatus according to the first embodiment. FIG. 2 is a diagram showing an example of a hardware configuration of an optical device control apparatus according to Embodiment 1. 5 is a flowchart showing the operation of the optical device control apparatus according to Embodiment 1. It is a functional block diagram which shows roughly the structure of the optical instrument control apparatus based on Embodiment 2 of this invention. (A) to (c) are perspective views schematically showing a camera and a moving mechanism as an optical device controlled by the optical device control apparatus according to the second embodiment. FIG. 7 is a diagram showing an example of a hardware configuration of an optical device control apparatus according to Embodiment 2. FIG. 10 is a flowchart showing the operation of the optical device control apparatus according to Embodiment 2. FIG.

  Hereinafter, an optical device control apparatus, an optical device control method, and an optical device control program according to an embodiment of the present invention will be described with reference to the attached drawings. In the first embodiment, an optical apparatus is a projector (image projection apparatus) that projects an image on an object, and an apparatus, a method, and a program that do not cause a shadow to be formed on a projection image will be described. In the second embodiment, the optical device is a camera (imaging device) for imaging an object and acquiring image data in a physical space, and an apparatus that does not cause a dropout (non-imaging area) in a composite image based on the imaged image. Explain how, programs. The following embodiments are merely examples, and various modifications are possible within the scope of the present invention.

<< 1 >> First Embodiment
<< 1-1 >> Configuration FIG. 1 is a functional block diagram schematically showing a configuration of the optical device control apparatus 1 according to the first embodiment. The optical device control device 1 is a device capable of implementing the optical device control method according to the first embodiment. The optical device control apparatus 1 is an apparatus capable of executing the optical device control program according to the first embodiment.

  2 (a) to 2 (c) show a plurality of projectors 3 (3a, 3b) as optical devices controlled by the optical device control apparatus 1 according to the first embodiment and a plurality of moving mechanisms 4 (4a, 4b) Is a schematic perspective view.

  The optical device control apparatus 1 includes a plurality of moving mechanisms (for example, robot arms) 4 for changing the arrangement of the plurality of projectors 3 (3a and 3b) and the plurality of projectors 3 (3a and 3b), and acquisition of real space information It is connected to a unit (for example, a camera or a sensor) 5 and controls the operation of the plurality of projectors 3 (3a, 3b) and the operation of the plurality of moving mechanisms 4 (4a, 4b). The physical space information acquisition unit 5 may be an apparatus separate from the projector 3 (3a, 3b), or may be an apparatus integrated with any of the plurality of projectors 3 (3a, 3b). The optical device control apparatus 1, the plurality of projectors 3 (3a, 3b), the plurality of moving mechanisms 4 (4a, 4b), and the real space information acquisition unit 5 are objects (lights on the object 51 in the real space An image projection system capable of projecting an image (synthetic projection image) 54 without the shadow 53 on the object 51 is configured in the case where the shielding object 52 is present. The object 52 is, for example, the moving hand of the user 50 working on the object 51, and in this case, the shadow 53 moves on the object 51 by the movement of the hand of the user 50.

  The optical device control device 1 includes an object estimation unit 11, an arrangement calculation unit 12, a movement mechanism control unit 13, a parameter calculation unit 14, an image data generation unit 15, and a storage unit 16.

  The object estimation unit 11 receives the physical space information from the physical space information acquisition unit 5, detects the shadow 53 generated on the target object 51, and estimates the object 52 causing the shadow 53. For example, the object estimation unit 11 selects the plurality of projectors 3 based on the physical space information acquired by the physical space information acquisition unit 5 when the plurality of optical devices 3 are projecting the projection image 54 onto the object 51. It is determined whether or not the shadow 53 which is an area where the plurality of projection images 54 projected from (3a, 3b) do not reach the object 51 is formed, and the shadow 53 is formed when the shadow 53 is detected. The information which shows the arrangement | positioning (position and shape) of the object 52 to carry out is acquired.

  The arrangement calculation unit 12 acquires information necessary for the arrangement (position and posture) of the plurality of projectors 3 (3a, 3b) from the storage unit 16 and does not form the shadow 53 on the object 51 Calculate the arrangement (position and posture) of 3a, 3b). For example, based on the information indicating the arrangement of the object 52, the arrangement calculation unit 12 calculates the arrangement (for example, the position and orientation) of the plurality of projectors 3 (3a, 3b) in which the shadow 53 is not formed. Here, the position is the coordinates of the plurality of projectors 3 (3a, 3b) shown in the xyz orthogonal coordinate system shown in FIGS. 2A to 2C, and the posture is the plurality of projectors 3 (3a, 3b) ) In the direction of the optical axis (one-dot chain line) AX.

  The movement mechanism control unit 13 controls the operation of the plurality of movement mechanisms 4 (4a, 4b) for moving the plurality of projectors 3 (3a, 3b) based on the result calculated by the arrangement calculation unit 12. The moving mechanism control unit 13 arranges the plurality of projectors 3 (3a, 3b) at positions where the shadow 53 is not formed based on the arrangement of the plurality of projectors 3 (3a, 3b) calculated by the arrangement calculation unit 12 Thus, the plurality of moving mechanisms 4 (4a, 4b) are controlled. The plurality of moving mechanisms 4 (4a, 4b) are mechanisms capable of changing the position and orientation of the plurality of projectors 3 (3a, 3b). The plurality of moving mechanisms 4 (4a, 4b) are robot arms. However, the plurality of moving mechanisms 4 (4 a, 4 b) may be a self-propelled robot, an air moving device such as a drone, or other mechanisms.

  The storage unit 16 is a characteristic of the projector such as the focal length 161 of each of the plurality of projectors 3 (3a, 3b) to be controlled, the brightness 162, and various image generation parameters 163 used for image generation. Record information about

  The parameter calculation unit 14 uses the positions of the plurality of projectors 3 (3a, 3b) moved by the moving mechanism control unit 13 and the information regarding the plurality of projectors 3 (3a, 3b) acquired from the storage unit 16 Each of (3a, 3b) calculates parameters of an image necessary for generating a projection image, and stores them in the storage unit 16 as an image generation parameter 163. As parameters in the plurality of projectors 3 (3a, 3b), for example, there are internal parameters unique to the device such as focal length in the projector, image center, lens distortion, etc., and external parameters representing position and attitude of the projector.

  The image data generation unit 15 generates an image based on the information acquired from the image generation parameter 163. The plurality of projectors 3 (3a, 3b) project the image generated by the image data generation unit 15, the projection image in the initial state, and the like on the physical space.

  FIG. 3 is a diagram showing an example of the hardware configuration of the optical device control apparatus 1 according to the first embodiment. The optical device control device 1 may be a computer. The optical device control apparatus 1 includes a central processing unit (CPU) 31 as a processor as an information processing unit, a graphics processing unit (GPU) 32, a main memory 33, a storage 34, and a data transfer path for data exchange. A certain bus 35 is provided. The optical device control device 1 includes a plurality of moving mechanisms that respectively movably support the camera 5a as the physical space information acquisition unit, the plurality of projectors 3 (3a, 3b), and the plurality of projectors 3 (3a, 3b) 4 (4a, 4b) are connected. A sensor 5b may be connected to the optical device control device 1 as a physical space information acquisition unit, in addition to or instead of the camera 5a.

  The CPU 31 is an arithmetic device for executing the entire operation of the optical device control device 1.

  The main memory 33 is a storage device in which the hardware of the optical device control device 1 can erase and rewrite data. The main memory 33 is volatile, but operates faster than the storage 34, and is used to store data in use or data that needs to be used immediately. For example, an optical device control program for performing a shadow removal process shown in FIG. 4 described later is stored in the main memory 33. The optical device control program is executed by the CPU 31. When the CPU 31 executes the optical device control program, all or part of the object estimation unit 11, the arrangement calculation unit 12, the movement mechanism control unit 13, the parameter calculation unit 14, and the image data generation unit 15 shown in FIG. It can be realized.

  The storage 34 is a storage device capable of erasing and rewriting data by the hardware of the optical device control device 1 and corresponds to the storage unit 16 in FIG. 1. The information stored in the storage develops necessary information in the main memory 33 when the program is executed.

  The camera 5a is a device that captures an image necessary to obtain real space information.

  The sensor 5 b is a device that acquires a value necessary to acquire real space information. The sensor 5 is, for example, a GPS (Global Positioning System) which measures a position, an acceleration sensor which detects an acceleration, a geomagnetic sensor which measures an azimuth, and a depth sensor which measures a distance to an object.

  The plurality of projectors 3 (3a, 3b) are image projection devices required to project an image on the physical space. The plurality of moving mechanisms 4 (4a, 4b) are devices for moving the plurality of projectors 3 (3a, 3b).

<< 1-2 >> Operation FIG. 4 is a flowchart showing an operation (shadow removal processing) of the optical device control apparatus 1 according to the first embodiment. The process shown in FIG. 4 is executed each time the physical space information acquisition unit 5 (5a, 5b) acquires physical space information (for example, in a fixed cycle).

  In step S11, the image data generation unit 15 of the optical device control device 1 provides the image data to the plurality of projectors 3 (3a, 3b), and a desired image is generated toward the object 51 (if the object 52 does not exist , And project the projection image that can be viewed by the user 50 (see FIG. 2A).

  In the next step S12, the optical device control apparatus 1 causes the physical space information acquisition unit 5 to acquire physical space information including the object 51. When the physical space information acquisition unit 5 is a camera, the physical space information acquisition unit 5 acquires image data as physical space information, and when the physical space information acquisition unit 5 is a sensor, the physical space information acquisition unit 5 A detection value (for example, distance information in the case of a depth sensor) is acquired as real space information.

  In the next step S13, the object estimation unit 11 detects whether the shadow 53 which is a dark area is formed by the object 52 on the object 51 based on the information acquired in step S12, and does not detect the shadow 53. In the case (NO in step S13), it can be determined that the desired image is projected on the object 51, so the shadow removal process (image interpolation process) is ended. The process of detecting which of the plurality of projectors 3 (3a, 3b) the projected image from the object 52 is blocked by the object 52 is known image processing. For example, the process described in Patent Document 2 A method may be adopted. When shadow 53 is detected (YES in step S13), the process proceeds to step S14, object estimating unit 11 estimates (calculates) the position and shape of object 52, and the process proceeds to step S15. In step S15, the placement calculation unit 12 calculates the placement (position and orientation) of the plurality of projectors 3 (3a, 3b).

  In the next step S16, the moving mechanism control unit 13 controls the plurality of moving mechanisms 4 (4a, 4b) to obtain a plurality of positions (position and attitude) calculated in step S15 up to the positions calculated in step S16. The projector 3 (3a, 3b) is moved (see FIG. 2 (b)).

  In the next step S17, the parameter calculation unit 14 calculates a parameter indicating how to make the pixel value of the projection image of each of the plurality of projectors 3 (3a, 3b) become an image from which the shadow is removed. , And stored in the image generation parameter 163.

  In the next step S18, the image data generation unit 15 generates an image to be projected based on the image generation parameter 163 by each of the plurality of projectors 3 (3a, 3b).

  In the next step S19, the projection images generated in step S17 are projected by the plurality of projectors 3 (3a, 3b) (see FIG. 2C).

<< 1-3 >> Effects As described above, according to the optical device control device 1, the optical device control method, and the optical device control program of the first embodiment, as shown in FIG. When the shadow 53 of the projection image is detected, the position and shape of the object 52 are estimated, and as shown in FIG. 2B, the plurality of projectors 3 (3a, 3b) are moved based on the estimation result. , Setting of projectors (position and attitude) and parameters. For this reason, even when the object 52 is present on the object 51, as shown in FIG. 2C, it is possible to project a desired image (composite projection image) without shadow on the object 51. it can.

<< 2 >> Second Embodiment
<< 2-1 >> Configuration In the first embodiment, the arrangement of the plurality of projectors 3 (3a, 3b) is calculated, the plurality of projectors 3 (3a, 3b) are moved, and the plurality of projectors 3 (3a) are moved. , 3b) by generating a projection image according to the arrangement (position and posture), a shadowless composite projection image (54 in FIG. 2C) can be displayed on the object (51 in FIG. 2C). The optical device control device 1 that performs control for projection has been described.
On the other hand, in the second embodiment, the arrangement of a plurality of optical devices (cameras) is calculated, the plurality of optical devices (cameras) are moved, and the arrangement (position and posture of a plurality of optical devices (cameras) The optical device control apparatus 2 will be described which performs control for removing an arbitrary object arranged to hide an object from a display image by acquiring an image according to. However, the optical device may be spatial information acquisition means other than a camera, such as a depth sensor.
The projector and the camera are similar optical models, and when it is desired to obtain a photographed image from which an arbitrary object has been removed, it is possible to generate a display image from which the object is removed by processing similar to the processing in the first embodiment. It is.

  FIG. 5 is a functional block diagram schematically showing the configuration of the optical device control apparatus 2 according to the second embodiment. The optical device control device 2 is a device capable of implementing the optical device control method according to the second embodiment. The optical device control apparatus 2 is an apparatus capable of executing the optical device control program according to the second embodiment.

  6 (a) to 6 (c) show a plurality of cameras 6 (6a, 6b) and a plurality of moving mechanisms 7 (7a, 7b) as optical devices controlled by the optical device control apparatus 2 according to the second embodiment. Is a schematic perspective view.

  The optical device control device 2 includes a plurality of cameras 6 (6a, 6b) and a plurality of moving mechanisms (for example, robot arms) 7 (7a, 7b) for changing the arrangement of the plurality of cameras 6 (6a, 6b), respectively. And the display device 8 to control the operation of the plurality of cameras 6 (6a, 6b) and the operation of the plurality of moving mechanisms 7 (7a, 7b). The optical device control device 2, the plurality of cameras 6 (6a, 6b), the plurality of moving mechanisms 7 (7a, 7b), and the display device 8 are objects (light shields) on the object 61 in real space An imaging system capable of displaying on the display device 8 the composite image 61a without the missing part (the image 62a of the object 62 in FIG. 6A) in the display image of the display device 8 even when 62 exists. Configure The object 62 may be, for example, an object that exists on the object 61, that is, between the object 61 and the plurality of cameras 6 (6a, 6b) and is moving.

  The optical device control device 2 includes an object estimation unit 21, an arrangement calculation unit 22, a movement mechanism control unit 23, a parameter calculation unit 24, an image combining unit 26, and a storage unit 27.

  The object estimation unit 21 receives physical space information from a plurality of cameras 6 (6a, 6b), and estimates an unnecessary area of the object 62 in order to obtain an arbitrary image. For example, in the object estimation unit 21, a plurality of cameras 6 (6a, 6b) shoot an object 61, and a plurality of cameras 6 (6a, 6b) are captured based on a plurality of images acquired by the plurality of cameras 6 (6a, 6b). 6a, 6b) determine whether or not an object (light shield) 62 forming a non-imaging area exists on the object 61, and when such an object 62 is detected, the arrangement (position of the object 62) And information indicating the shape) is obtained (see FIG. 6A). As shown in FIG. 6A, the non-shooting area of the object 61 becomes an image 62a of the object 62 in the display image of the display device 8. When the physical space information acquisition unit is a sensor, the physical space information acquisition unit 5 acquires a detection value (for example, distance information in the case of a depth sensor) as physical space information.

  The arrangement calculation unit 22 acquires information necessary for the arrangement (position and posture) of the plurality of cameras 6 (6a, 6b) from the storage unit 27 and can obtain an arrangement (position and posture) from which the image from which the object 62 is removed can be acquired. calculate. Information on the plurality of cameras 6 (6a, 6b) is stored in advance in the storage unit 16. For example, the arrangement calculation unit 22 acquires information necessary for the arrangement (position and orientation) of the plurality of cameras 6 (6a, 6b) from the storage unit 16 and, based on the information indicating the arrangement of the object 62, the target 61 The arrangement (e.g., position and orientation) of the plurality of cameras 6 (6a, 6b) for preventing formation of a non-photographing area on the top is calculated. Here, the position is the coordinates of a plurality of cameras 6 (6a, 6b) shown in the xyz orthogonal coordinate system shown in FIGS. 6A to 6C, and the posture is a plurality of cameras 6 (6a, 6b) In the direction of the optical axis (one-dot chain line) AX.

  The movement mechanism control unit 23 sets the plurality of cameras 6 at positions where the non-photographing area is not formed on the target object 61 based on the arrangement of the plurality of cameras 6 (6a, 6b) calculated by the arrangement calculation unit 22. The plurality of moving mechanisms 7 (7a, 7b) are controlled such that (6a, 6b) are arranged (see FIG. 6 (b)). The plurality of moving mechanisms 7 (7a, 7b) are mechanisms capable of changing the position and attitude of the plurality of cameras 6 (6a, 6b). The plurality of moving mechanisms 7 (7a, 7b) are robot arms, but may be self-propelled robots, airborne devices, or other mechanisms.

  The storage unit 27 records information on camera characteristics, such as focal lengths 271 of each of the plurality of cameras 6 (6a, 6b) to be controlled and various image generation parameters 272 used for image generation. Do.

  The parameter calculation unit 24 uses the positions of the plurality of cameras 6 (6a, 6b) moved by the plurality of moving mechanisms 7 (7a, 7b) and information about each of the cameras 6 (6a, 6b) acquired from the storage unit 27. The parameters of the image necessary to interpolate the images acquired by the cameras 6 (6a, 6b) are calculated and stored as the image generation parameters 272 of the storage unit 27. Parameters in the plurality of cameras 6 (6a, 6b) include, for example, internal parameters specific to the device such as focal length in the camera, image center, lens distortion, etc., and external parameters representing the position and attitude of the camera.

  The image data generation unit 25 processes the image data acquired by the plurality of cameras 6 (6a, 6b) based on the information acquired from the image generation parameter 272, and is missing in one of the plurality of image data. Interpolated image generation processing is performed to interpolate an existing image portion with other image data.

  The image synthesizing unit 26 synthesizes the image generated by the image data generating unit 25 and generates a synthesized image from which the object 62 is removed (see FIG. 6C).

  FIG. 7 is a diagram showing an example of a hardware configuration of the optical device control apparatus 2 according to the second embodiment. The optical device control device 2 may be a computer. The optical device control device 2 includes a CPU 41 which is a processor as an information processing unit, a GPU 42, a main memory 43, a storage 44, and a bus 45 which is a data transfer path for data exchange. The optical device control device 2 is connected with a plurality of cameras 6 (6a, 6b) and a plurality of moving mechanisms 7 (7a, 7b) for supporting the plurality of cameras 6 (6a, 6b) movably. . The CPU 41, the GPU 42, the main memory 43, and the storage 44 are the same as the CPU 31, the GPU 32, the main memory 33, and the storage 34 in FIG. The CPU 41 stores, in the main memory 43, an optical device control program for performing an object removal process shown in FIG. The optical device control program is executed by the CPU 41. When the CPU 41 executes the optical device control program, the object estimation unit 21, the arrangement calculation unit 22, the movement mechanism control unit 23, the parameter calculation unit 24, the image data generation unit 25, and the image combination unit 26 shown in FIG. All or part can be realized.

<< 2-2 >> Operation FIG. 8 is a flowchart showing an operation (object removal processing) of the optical device control apparatus 2 according to the second embodiment. The process shown in FIG. 8 is executed each time the camera 6 (6a, 6b) acquires physical space information (for example, at a constant cycle).

  In step S21, the optical device control apparatus 2 causes the camera 6 (6a, 6b) as a physical space information acquisition unit to acquire physical space information including the object 61 (see FIG. 6A).

  In the next step S22, based on the information acquired in step S21, the object estimation unit 21 detects whether or not the object 62 forming a non-imaging area exists in the object 61, and does not detect the object 62 ( Since it can be determined in step S22 that the object 61 can be photographed, the object removal process (image interpolation process) is ended. The process of detecting which camera of the plurality of cameras 6 (6a, 6b) the object 62 appears (that is, whether the object 61 is hidden by the object 62) is known image processing However, for example, the method described in Patent Document 2 may be adopted. When object 62 is detected (YES in step S22), the process proceeds to step S23, object estimating unit 21 estimates (calculates) the position and shape of object 62, and the process proceeds to step S24. In step S24, the placement calculation unit 22 calculates the placement (position and posture) of the plurality of cameras 6 (6a, 6b).

  In the next step S25, the moving mechanism control unit 23 controls the plurality of moving mechanisms 7 (7a, 7b) to obtain a plurality of positions (position and attitude) calculated in step S24 up to the positions calculated in step S25. The camera 6 (6a, 6b) is moved (see FIG. 6 (b)).

  In the next step S26, the parameter calculation unit 24 calculates a parameter indicating how to make the pixel value of the projection image of each of the plurality of cameras 6 (6a, 6b) become an image from which the object 62 is removed. And stored in the image generation parameter 272.

  In the next step S27, the image data generation unit 25 changes the image data acquired by each of the plurality of cameras 6 (6a, 6b) based on the image generation parameter 272. That is, the image data generation unit 25 generates an interpolation image from the captured image using each of the cameras 6 (6a, 6b) using the image generation parameter 272.

  In the next step S28, the image combining unit 26 combines the plurality of images (interpolated images) generated in step S27 and causes the display device 8 to display the combined image (see FIG. 6C).

<< 2-3 >> Effects As described above, according to the optical device control apparatus 2, the optical device control method, and the optical device control program of the second embodiment, as shown in FIG. When an object 62 forming a non-imaging area is detected, the position and shape of the object 62 are estimated, and as shown in FIG. 6B, a plurality of cameras 6 (6a, 6b) are based on the estimation result. To set the arrangement (position and attitude) and parameters of the plurality of cameras 6 (6a, 6b). Therefore, even when the object 62 is present on the target object 61, as shown in FIG. 6C, a desired composite image without the object 62 can be displayed on the display device 8.

  1, 2 optical device control device, 3, 3a, 3b projector (optical device), 4, 4a, 4b moving mechanism, 5 real space information acquisition unit, 5a real space information acquisition unit (camera), 5b real space information acquisition unit (Sensors) 6, 6a, 6b Cameras (Optical Instruments) 7, 7a, 7b Moving Mechanisms, 8 Displays, 11 Object Estimator, 12 Placement Calculator, 13 Moving Mechanism Controller, 14 Parameter Calculator, 15 Images Data generation unit, 16 storage units, 21 object estimation unit, 22 arrangement calculation unit, 23 movement mechanism control unit, 24 parameter calculation unit, 25 image data generation unit, 26 image combination unit, 27 storage unit, 51 object, 52 object (Light shield), 53 shadows, 54 images (composite projection image), 61 objects, 61a displayed objects, 62 objects (Light shield), 62a displayed object 64 image (composite image), AX optical axis.

Claims (10)

An optical device control apparatus for controlling a plurality of optical devices for projecting a plurality of images on a target object and a plurality of moving mechanisms for changing the arrangement of the plurality of optical devices.
Images projected from the plurality of optical devices are placed on the object based on the physical space information acquired by the physical space information acquisition unit when the plurality of optical devices are projecting an image onto the object. An object estimation unit that determines whether or not a shadow is formed that is an area that does not reach f, and acquires information indicating an arrangement of an object that forms the shadow when the shadow is detected;
An arrangement calculation unit configured to calculate an arrangement of the plurality of optical devices in which the shadow is not formed based on the information indicating the arrangement of the object;
A moving mechanism control unit that controls the plurality of moving mechanisms such that the plurality of optical devices are arranged at a position where the shadow is not formed based on the arrangement of the plurality of optical devices calculated by the arrangement calculation unit; ,
An optical device control apparatus characterized by having.   The optical device control apparatus according to claim 1, wherein the movement mechanism control unit changes the position and the attitude of each of the plurality of optical devices by controlling the plurality of movement mechanisms.   Projection of the plurality of optical devices is configured to generate one combined image in which the plurality of images projected by the plurality of optical devices are connected according to the position and orientation of each of the plurality of optical devices. The optical instrument controller according to claim 2, wherein a parameter indicating a condition is calculated. An optical device control apparatus for controlling a plurality of optical devices for acquiring a plurality of images of the object by photographing an object and a plurality of moving mechanisms for changing the arrangement of the plurality of optical devices,
Based on the plurality of images acquired by the plurality of optical devices, the plurality of optical devices determine whether or not an object forming a non-imaging area exists on the object, and the object is detected. An object estimation unit that acquires information indicating the arrangement of the object when the
An arrangement calculation unit configured to calculate an arrangement of the plurality of optical devices in which the non-shooting area is not formed based on the information indicating the arrangement of the object;
Movement mechanism control for controlling the plurality of movement mechanisms such that the plurality of optical devices are arranged at a position where the non-shooting area is not formed based on the arrangement of the plurality of optical devices calculated by the arrangement calculation unit Department,
An optical device control apparatus characterized by having.   The optical device control apparatus according to claim 4, wherein the movement mechanism control unit changes the position and the attitude of each of the plurality of optical devices by controlling the plurality of movement mechanisms.   Photographing of the plurality of optical devices, wherein one composite image in which the plurality of images photographed by the plurality of optical devices are connected is generated according to the position and posture of each of the plurality of optical devices The optical instrument control apparatus according to claim 5, wherein a parameter indicating a condition is calculated. An optical device control method for controlling a plurality of optical devices for projecting a plurality of images on an object and a plurality of moving mechanisms for changing the arrangement of the plurality of optical devices.
Images projected from the plurality of optical devices are placed on the object based on the physical space information acquired by the physical space information acquisition unit when the plurality of optical devices are projecting an image onto the object. Determining whether or not a shadow is formed which is an area which does not arrive at the area, and obtaining information indicating the arrangement of the object forming the shadow when the shadow is detected;
Calculating the arrangement of the plurality of optical devices in which the shadow is not formed based on the information indicating the arrangement of the object;
Controlling the plurality of moving mechanisms such that the plurality of optical devices are arranged at a position where the shadow is not formed based on the calculated arrangement of the plurality of optical devices;
An optical instrument control method comprising: An optical device control method for controlling a plurality of optical devices for acquiring a plurality of images of the object by photographing an object and a plurality of moving mechanisms for changing the arrangement of the plurality of optical devices,
Based on the plurality of images acquired by the plurality of optical devices, the plurality of optical devices determine whether or not an object forming a non-imaging area exists on the object, and the object is detected. Obtaining information indicating the placement of the object when the
Calculating the arrangement of the plurality of optical devices in which the non-shooting area is not formed based on the information indicating the arrangement of the object;
Controlling the plurality of moving mechanisms such that the plurality of optical devices are arranged at a position where the non-shooting area is not formed based on the calculated arrangement of the plurality of optical devices.
An optical instrument control method comprising: An optical device control program for controlling a plurality of optical devices for projecting a plurality of images on a target object and a plurality of moving mechanisms for changing the arrangement of the plurality of optical devices.
On the computer
Images projected from the plurality of optical devices are placed on the object based on the physical space information acquired by the physical space information acquisition unit when the plurality of optical devices are projecting an image onto the object. Determining whether or not a shadow is formed that is an area that does not reach f, and acquiring information indicating an arrangement of an object that forms the shadow when the shadow is detected;
A process of calculating the arrangement of the plurality of optical devices in which the shadow is not formed based on the information indicating the arrangement of the object;
A process of controlling the plurality of movement mechanisms such that the plurality of optical devices are arranged at a position where the shadow is not formed based on the calculated arrangement of the plurality of optical devices;
An optical device control program characterized by performing: An optical device control program for controlling a plurality of optical devices for acquiring a plurality of images of the object by photographing an object and a plurality of movement mechanisms for changing the arrangement of the plurality of optical devices,
On the computer
Based on the plurality of images acquired by the plurality of optical devices, the plurality of optical devices determine whether or not an object forming a non-imaging area exists on the object, and the object is detected. Acquiring information indicating the arrangement of the object when the
A process of calculating the arrangement of the plurality of optical devices in which the non-shooting area is not formed based on the information indicating the arrangement of the object;
A process of controlling the plurality of moving mechanisms such that the plurality of optical devices are arranged at a position where the non-shooting area is not formed based on the calculated arrangement of the plurality of optical devices;
An optical device control program characterized by performing:

Images