JP2021158446A - Image projection device, image projection method and program - Google Patents

Abstract

To more easily maintain image quality and to prevent deterioration in image quality such as a change in brightness of a part of a projected image and a lack of an image.SOLUTION: An image projection device includes: a plurality of projection units that performs projection from different image projection angles so as to form an overlapping image in which a plurality of projection images are overlapping in an identical area; an imaging unit that generates a captured image by imaging the identical area; an obstacle detection unit that detects an obstacle between the projection units and the identical area on the basis of the captured image; and a control unit that executes control for causing the projection units to project the projection images; wherein the control unit performs projection by switching the plurality of projection units for each frame and executes control so as to specify a position of a shadow of the projection images, based on captured images obtained by capturing the projection images projected by being switched for each frame by the imaging unit and based on the projection images.SELECTED DRAWING: Figure 16

Description

The present invention relates to an image projection device, an image projection method and a program.

A so-called projection mapping technique is known in which an image is projected from a plurality of image projection devices onto one object to be projected to form an image (for example, Patent Document 1). In Patent Document 1, when a region where the light for projecting an image is blocked occurs, the image projection device is controlled so as to prohibit the projection of the light onto the region.

Special Table 2007-520945

The process of partially prohibiting the light projected from the image projection device as in Patent Document 1 is complicated, and a method capable of maintaining the image quality of the projected image more easily has been required. In addition, the brightness of the projected image is reduced because the projection corresponding to the blocked area is prohibited.

The present invention provides an image projection device, an image projection method, and a program that can more easily maintain the image quality and prevent changes in the brightness of a part of the projected image and deterioration of the image quality such as chipping. The purpose.

The image projection device of the present invention generates an captured image by imaging a plurality of projection units projecting from different image projection angles and the same region so that a plurality of projected images form overlapping overlapping images in the same region. An obstacle detection unit that detects an obstacle between the projection unit and the same region based on the captured image, and a control unit that controls the projection image to be projected onto the projection unit. The control unit is based on an captured image obtained by switching and projecting the plurality of projection units for each frame and capturing the projected image switched for each frame by the imaging unit, and a projected image. Therefore, it is characterized in that it is controlled so as to specify the position of the shadow of the projected image.

In the image projection method of the present invention, the same projected images are projected by a plurality of projection devices projecting from different image projection angles so that the plurality of projected images form overlapping overlapping images in the same region, and the respective projected images are duplicated. A step of detecting an obstacle between the projection device and the same region based on the captured image of the region, a step of projecting the projection image on the projection device, and a plurality of projection images for each frame. A step of switching the projection device to project, and controlling to specify the position of the shadow of the projected image based on the captured image obtained by switching the projected image frame by frame by the imaging device and the projected image. including.

The program of the present invention covers the same area in which each projected image is projected by a plurality of projection devices projecting from different image projection angles so that a plurality of projected images form overlapping overlapping images in the same area. A step of detecting an obstacle between the projection device and the same region based on the captured image, a step of projecting the projection image on the projection device, and the plurality of projection devices for each frame of the projection image. The step of controlling to specify the position of the shadow of the projected image based on the captured image obtained by switching and projecting the projected image and the projected image switched frame by frame by the imaging device and the projected image.
Let the computer run.

According to the present invention, it is possible to more easily maintain the image quality and prevent the change in the brightness of a part of the projected image and the occurrence of deterioration in the image quality such as chipping.

FIG. 1 is a functional block diagram showing a configuration of an image projection system according to a first embodiment of the present invention. FIG. 2 is a block diagram showing an example of a configuration for realizing the functional block shown in FIG. FIG. 3 is a diagram showing an example of the positional relationship between the projected object, the projection device, and the image pickup device for projecting an image by the plurality of projection devices and imaging by the plurality of image pickup devices. FIG. 4 is a diagram showing an example of the relationship between the first projected image, the second projected image, the overlapping image, the first captured image, and the second captured image. FIG. 5 is a diagram showing an example of a case where there is an obstacle within the angle of view of projection by the projection device and within the angle of view of imaging by the imaging device. FIG. 6 is a diagram showing an example of the relationship between the first projected image, the second projected image, the overlapping image, the first captured image, and the second captured image. FIG. 7 is a diagram showing an example of the relationship between the first projected image, the second projected image, the duplicated image, the first captured image, and the second captured image when the corrected image is projected. FIG. 8 is a flowchart showing the flow of processing related to the detection of obstacles and shadow areas. FIG. 9 is a schematic diagram showing an example of the relationship between the optical axes of the projection device and the image pickup device, which are treated as one set. FIG. 10 is a schematic view showing a mechanism of a configuration in which an optical axis on which a projection device projects an image via a half mirror and an optical axis on which an imaging device images an overlapping region are provided so as to overlap each other. FIG. 11 is a functional block diagram showing a configuration of an image projection system according to a second embodiment of the present invention. FIG. 12 is a block diagram showing an example of a configuration for realizing the functional block shown in FIG. FIG. 13 is a diagram showing an example of projecting an image on an overlapping region and imaging the overlapping region when there is no obstacle. FIG. 14 is a diagram showing an example of projecting an image on an overlapping region and imaging the overlapping region when there is an obstacle. FIG. 15 is a diagram showing an example of projection of an image on an overlapping region and imaging of the overlapping region after control corresponding to the detection of an obstacle is performed. FIG. 16 is a flowchart showing the flow of processing related to the detection of obstacles and shadow areas. FIG. 17 is a diagram showing a configuration example when there is only one imaging unit.

Hereinafter, embodiments according to the present invention will be described in detail with reference to the accompanying drawings. The present invention is not limited to this embodiment, and when there are a plurality of embodiments, the present invention also includes a combination of the respective embodiments. Further, in the following embodiments, the same parts are designated by the same reference numerals, so that duplicate description will be omitted.

[First Embodiment]
FIG. 1 is a functional block diagram showing a configuration of an image projection system 1 according to a first embodiment of the present invention. The image projection system 1 includes a plurality of projection units, a plurality of imaging units, a storage unit 11, an image acquisition unit 12, a shadow area identification unit 13, an image correction unit 14, and a control unit 15. In FIG. 1 and FIG. 11 described later, the projection unit 21 and the projection unit 22 are illustrated as a plurality of projection units. Further, in FIG. 1 and FIG. 11 described later, an imaging unit 31 and an imaging unit 32 are illustrated as a plurality of imaging units.

FIG. 2 is a block diagram showing an example of a configuration for realizing the functional block shown in FIG. As shown in FIG. 2, the image projection system 1 includes, for example, a plurality of projection devices, a plurality of image pickup devices, and an information processing device 50. In FIGS. 1 and 2 and FIGS. 11 and 12 described later, the flow of the data flow in the information processing apparatus 50 is indicated by a solid arrow, and the optical projection and imaging are indicated by a broken arrow.

The information processing device 50 is a so-called computer. The connection form between the information processing device 50 and the plurality of projection devices 61 and 62 and the connection form between the information processing device 50 and the plurality of imaging devices 71 and 72 may be wired or wireless. It may be a mixture of wired and wireless. Further, the specific connection form may be a bus interface such as USB (Universal Serial Bus), a network communication line, or a dedicated connection form. ..

Each projection device shown in FIG. 2 includes one projection unit. In FIG. 2 and FIG. 12 described later, a projection device 61 and a projection device 62 are illustrated as a plurality of projection devices. The projection devices 61 and 62 are so-called projectors, and project an image onto the projected object 80. The projected object 80 is a so-called screen. The projected object 80 is, for example, a diffusion type, a reflective type, or a regression type screen, and is provided on the assumption that the projected surface can be visually recognized. The projected object 80 may be provided on the assumption that the back surface of the projected surface can be visually recognized, such as a transmissive screen.

The projection device 61 includes a projection unit 21. The projection device 62 includes a projection unit 22. The projection units 21 and 22, respectively, are a display element, a light source that irradiates the display element with light, an optical member such as a lens that causes the light reflected or transmitted by the display element to converge as an image on the projectile 80, and an image input from the outside. It is equipped with a control circuit or the like that operates the display element according to the data. Examples of display elements include, but are not limited to, liquid crystal on silicon (LCOS) devices, digital mirror devices (DMDs), and liquid crystal devices, and can be changed as appropriate. ..

Each imaging device shown in FIG. 2 includes one imaging unit. In FIG. 2 and FIG. 12 described later, an image pickup device 71 and an image pickup device 72 are illustrated as a plurality of image pickup devices. The imaging device 71 includes an imaging unit 31. The imaging device 72 includes an imaging unit 32. The image pickup units 31 and 32 each include an image pickup element that functions as a so-called digital camera, a circuit that generates an image based on the output of the image pickup element, and outputs the image as an image pickup image. Examples of the image pickup device include a CMOS (Complementary Metal Oxide Semiconductor) image sensor and a CCD (Charge Coupled Device) image sensor, but the present invention is not limited to these, and can be changed as appropriate.

FIG. 3 is a diagram showing an example of the positional relationship between the projected object 80, the projection devices 61, 62, and the image pickup devices 71, 72 for projecting an image by the plurality of projection devices and imaging by the plurality of imaging devices. .. As shown in FIGS. 2 and 3, the projection device 61 and the image pickup device 71 are treated as one set P1. That is, the projection unit 21 and the imaging unit 31 are provided so as to form a set P1. Further, the projection device 62 and the image pickup device 72 are treated as another set P2. That is, the projection unit 22 and the imaging unit 32 are provided so as to form a set P2.

In FIG. 3, the angle of view of the projection by the projection device 61 is shown within the acute angle formed by the two solid lines L1. Further, the angle of view of the projection by the projection device 62 is shown within the acute angle formed by the two solid lines L2. As shown in FIG. 3, the projection device 61 and the projection device 62 project images from different image projection angles so that the projected images overlap each other in the overlapping region 81 of the projected object 80 to form an overlapping image. Will be installed. That is, the plurality of projection units 21 form a duplicate image in which the projection image projected by the projection unit 21 shown in FIGS. 1 and 2 and the plurality of projection images produced by the projection image projected by the projection unit 22 overlap in the same region. , 22 project from different image projection angles. Here, the overlapping area 81 functions as the same area.

In FIG. 3, the angle of view of the image captured by the imaging device 71 is shown within the acute angle formed by the two broken lines L3. Further, the angle of view of the image captured by the imaging device 72 is shown within the acute angle formed by the two broken lines L4. The imaging device 71 shown in FIG. 3 is provided so as to be paired with the projection device 61, and images the overlapping region 81 from an angle corresponding to the image projection angle of the projection device 61 to be paired with the projection device 61 to generate an captured image. Further, the imaging device 72 images the overlapping region 81 from an angle corresponding to the image projection angle of the projection device 62 which is provided to be paired with the projection unit 22 and generates an captured image. That is, the imaging unit 31 shown in FIGS. 1 and 2 images the overlapping region 81 from an angle corresponding to the image projection angle of the projection unit 21 which is provided to be paired with the projection unit 21 and captures the captured image. Generate. Further, the imaging unit 32 is provided so as to be paired with the projection unit 22, and images the overlapping region 81 from an angle corresponding to the image projection angle of the projection unit 22 to be paired with the projection unit 22 to generate an captured image.

In the following description, unless otherwise specified, the image projected by the projection unit 21 is referred to as the first projection image. Further, the image projected by the projection unit 22 is referred to as a second projection image. Further, the image formed in the overlapping region 81 of the projected object 80 by the images projected by the projection unit 21 and the projection unit 22 is regarded as the overlapping image. Further, the image captured by the imaging unit 31 is set as the first captured image. Further, the image captured by the imaging unit 32 is used as the second captured image.

FIG. 4 is a diagram showing an example of the relationship between the first projected image, the second projected image, the overlapping image, the first captured image, and the second captured image. The relationship shown in FIG. 4 corresponds to the arrangement shown in FIG. The trapezoidal distortion of the image generated by the projection devices 61 and 62 projecting the image obliquely to the projected body 80 and the imaging devices 71 and 72 imaging the image from an oblique angle is appropriately corrected in advance. It shall be. For example, the projection unit 21 of the projection device 61 of FIG. 2 may be configured to perform a predetermined trapezoidal distortion correction process according to the installation position on the first projected image. Further, the imaging unit 31 of the imaging device 71 of FIG. 2 may be configured to perform a predetermined trapezoidal distortion correction process according to the installation position on the first captured image. Further, the projection device 62 of FIG. 2 may be configured in the same manner as the projection device 61, and the image pickup device 72 may be configured in the same manner as the image pickup device 71. Here, the predetermined trapezoidal distortion correction process may be any generally used trapezoidal distortion correction process.

The projection unit 21 and the projection unit 22 project an image based on the same image data unless the image correction unit 14 described later generates a corrected image. FIG. 4 and the like exemplify a case where an image corresponding to the projected image data 11b (see FIG. 2) stored in the storage unit 11 is projected. In the description of the embodiment, the case where the image corresponding to the projected image data 11b stored in the storage unit 11 is projected is taken as an example, but the projected image is not limited to this. Images corresponding to the projected image data read from the external storage device connected to the image projection system 1, the projected image data input from the external information processing device connected to the image projection system 1, and the like are projected. May be good.

As shown in FIG. 3, when there is no obstacle in either the angle of view of the projection by the projection device 61 or the angle of view of the projection by the projection device 62, the projection unit 21 is as shown in the overlapping image V1 shown in FIG. An image similar to the image projected by the projection unit 22 is formed in the overlapping region 81. Further, as in the first captured image 31a and the second captured image 32a shown in FIG. 4, the captured images by the imaging unit 31 and the imaging unit 32 are the same images as the overlapping image V1. This means that there are no obstacles in either the angle of view of the projection by the projection device 61 or the angle of view of the projection by the projection device 62, and the image captured by the image pickup device 71 is captured by the image pickup device 72. This is because there are no obstacles in any of the angles of view of the image.

FIG. 5 is a diagram showing an example of a case where there is an obstacle OB within the angle of view of the projection by the projection device 61 and within the angle of view of the image captured by the image pickup device 71. FIG. 6 is a diagram showing an example of the relationship between the first projected image, the second projected image, the overlapping image, the first captured image, and the second captured image. The relationship shown in FIG. 6 corresponds to the arrangement shown in FIG.

As shown in FIG. 5, when there is an obstacle OB within the angle of view of the projection by the projection device 61 and the angle of view of the image captured by the image pickup device 71, the obstacle OB is the projected light for projecting the image by the projection device 61. Block a part of. Therefore, as in the overlapping image V2 shown in FIG. 6, an image including a shadow region DA corresponding to the light blocked by the obstacle OB is formed in the overlapping region 81. Although not shown, even when there is no obstacle OB within the angle of view of the projection by the projection device 61 and there is an obstacle OB within the angle of view of the projection by the projection device 62, the shadow region DA is included as in the overlapping image V2. The image is formed in the overlapping region 81. As described above, when there is an obstacle OB between one of the plurality of projection portions and the overlapping region 81, the obstacle OB causes a shadow region DA in the overlapping image V2.

Further, since the obstacle OB is within the angle of view of the image captured by the imaging device 71, the image capturing unit 31 displays an image in which the obstacle OB is located inside the shadow region DA as in the first captured image 31b shown in FIG. It is imaged. On the other hand, there is no obstacle OB within the angle of view of the image captured by the imaging device 72. Therefore, as in the second captured image 32b shown in FIG. 6, the captured image by the imaging unit 32 is the same as the duplicate image V2. When there is no obstacle OB within the angle of view of the image captured by the imaging device 71 and there is an obstacle OB within the angle of view of the image captured by the imaging device 72, the contents of the first captured image 31b and the contents of the second captured image 32b. Is reversed.

As described above, when there is an obstacle OB within the angle of view of the image captured by one of the image pickup device 71 or the image pickup device 72 and there is no obstacle OB within the angle of view of the image imaged by the other, the contents and the first image of the first image captured image 31b. 2 There is a difference from the content of the captured image 32b. Based on the difference, it is possible to identify whether the obstacle OB is in the angle of view of the projection device 61 or the projection device 62. The information processing device 50 performs processing according to the specific and specific results.

As shown in FIG. 2, the information processing device 50 includes a storage unit 11 and a calculation unit 51. The storage unit 11 includes a software program (hereinafter, simply referred to as a program) and a storage device capable of storing data. Examples of the storage device included in the storage unit 11 include, but are not limited to, a hard disk drive, a solid state drive, a flash memory, and the like, and can be changed as appropriate. The storage unit 11 may be a combination of a reading device for a recording medium such as an optical disc and a recording medium set in the reading device. As shown in FIG. 2, the storage unit 11 stores the image projection program 11a and the projected image data 11b. The image projection program 11a is a program read and executed by the calculation unit 51.

The arithmetic unit 51 includes an arithmetic circuit that realizes various functions by reading a program and executing processing like a CPU (Central Processing Unit). The calculation unit 51 of the first embodiment functions as an image acquisition unit 12, a shadow area identification unit 13, an image correction unit 14, and a control unit 15 shown in FIGS. 1 and 2 by reading and executing the image projection program 11a. do.

The image acquisition unit 12 acquires an image projected by the projection units 21 and 22. Specifically, the image acquisition unit 12 reads and acquires the projected image data 11b from, for example, the storage unit 11. The projected image data 11b acquired by the image acquisition unit 12 can be referred to by the shadow area identification unit 13 and the image correction unit 14. Further, unless the corrected image is generated by the image correction unit 14 described later, the projection units 21 and 22 project an image corresponding to the content of the projected image data 11b acquired by the image acquisition unit 12 (see FIG. 4).

The shadow region specifying unit 13 identifies a shadow region generated in the overlapping region 81 due to an obstacle between one projection unit and the overlapping region 81 based on the captured image. Specifically, the shadow region specifying unit 13 performs image analysis for comparing the content of the image captured by the imaging units 31 and 32 with the content of the image acquired by the image acquisition unit 12. More specifically, the shadow region specifying unit 13 has a decrease in brightness in the tendency of the brightness distribution of the contents of the captured images by the imaging units 31 and 32, based on the tendency of the brightness distribution corresponding to the contents of the projected image data 11b. If a subregion is generated, the subregion is extracted as a shadow region. As an example, in the image analysis, which is a process of specifying a shadow region, when the shadow region of the first captured image is specified, the image acquisition unit 12 acquires the first projected image, and the first captured image and the first projection. The difference from the image is performed and a difference signal is generated. Based on this difference signal, it may be configured to specify a partial region in which the brightness has decreased by a certain amount. Similarly, when specifying the shadow region of the second captured image, it is preferable to specify a partial region in which the brightness is reduced by a certain amount from the difference from the second projected image. Further, when there is a partial region inside the shadow region where there is a characteristic change such as a change in brightness or a change in color different from the decrease in brightness, the partial region is extracted as an image of an obstacle. For example, in the difference signal between the first captured image and the first projected image, a partial area of the difference signal that has a difference amount of the difference signal and is different from the feature of the part where the brightness is reduced by a certain amount is specified, and the area of the obstacle. It may be configured to be specified as. By extracting the image of the shadow region and the obstacle, it becomes possible to determine which of the image captured by the imaging unit 31 and the image captured by 32 contains the obstacle. Such a determination may be made by the shadow area specifying unit 13 or by the control unit 15. Further, when the shadow region specifying unit 13 specifies the shadow region, it is preferable to further specify the projection unit that is paired with the imaging unit that captures the captured image including the obstacle that causes the shadow region. Since the projection units associated with the imaging unit that captures the captured image are configured as one set, if the imaging unit can be specified, the corresponding projection unit can be easily specified. Such identification may be performed by the shadow area specifying unit 13 or may be performed by the control unit 15. Further, the information regarding the specified projection unit may be stored in the storage unit 11.

The image correction unit 14 generates a corrected image in which a region corresponding to a shadow region in an image projected by one or more projection units other than one projection unit having an obstacle within the angle of view of the projection is corrected. Specifically, the image correction unit 14 determines the brightness of a partial region of the projected image data 11b at a position and size corresponding to the shadow region so that the overlapping image in the overlapping region 81 is drawn with the original brightness. Generates a corrected image that has been corrected to raise. The degree of correction corresponds to the degree of decrease in brightness caused by the shadow region extracted by the shadow region specifying unit 13 in the contents of the projected image data 11b. For example, when the second captured image has a shadow region, it is preferable to perform correction for increasing the brightness based on the difference signal obtained by the image analysis in the shadow region specifying unit 13. When the first captured image has a shadow region, the same correction may be performed.

The control unit 15 comprehensively controls each configuration and each functional configuration of the image projection system 1. Specifically, the control unit 15 of the first embodiment projects the corrected image on the projection unit. Here, the control unit 15 projects the corrected image by a projection unit different from the projection unit that is paired with the imaging unit that captures the captured image including an obstacle within the angle of view of the projection that causes the specified shadow region. It is good to let it.

FIG. 7 is a diagram showing an example of the relationship between the first projected image, the second projected image, the duplicated image, the first captured image, and the second captured image when the corrected image is projected. The relationship shown in FIG. 7 corresponds to the arrangement shown in FIG. Further, FIG. 7 assumes a case where the corrected image 14a is generated in response to the result of extracting the images of the shadow region DA and the obstacle OB shown in FIG. In FIG. 7, a partial region having a position and a size corresponding to a shadow region and having an increased brightness in the corrected image 14a is shown as a correction region BA.

In the example shown in FIG. 6, the detection that "there is an obstacle OB within the angle of view projected by the projection unit 21" is detected in the storage unit 11 in the "second captured image" which is the image captured by the imaging unit 22. Based on the fact that a shadow that should not be present in the projected image is projected, it is configured so that it is determined that "the obstacle OB is within the angle of view of the projection by the projection unit 21". Here, in the detection that "there is an obstacle OB within the angle of view of the projection by the projection unit 21", since the image of the obstacle OB is included in the first captured image, the image of the projection by the projection unit 21 It may be configured so that it is determined that there is an obstacle OB in the corner. The image correction unit 14 generates a corrected image 14a in which correction for increasing the brightness of a partial region having a position and a size corresponding to the shadow region DA in the second captured image is performed on the content of the projected image data 11b. As shown in FIG. 7, the control unit 15 projects the corrected image 14a on the projection unit 22 as a second projection image. The control unit 15 causes the projection unit 21 to project the contents of the projected image data 11b. As a result, the decrease in the brightness of the shadow region caused by the obstacle OB in the projection corresponding to the projected image data 11b by the projection unit 21 is offset by the increase in the brightness of the BA due to the projection corresponding to the corrected image 14a by the projection unit 22. Therefore, like the overlapping image V3 shown in FIG. 7, an image similar to the content of the projected image data 11b is formed in the overlapping region 81.

Further, as shown by the first captured image 31c and the second captured image 32c shown in FIG. 7, the shadow region DA captured by the first captured image 31b and the second captured image 32b shown in FIG. 6 is not captured. The control unit 15 can confirm that the correction of the shadow region is normally performed based on the first captured image 31c and the second captured image 32c.

Note that, in FIG. 1 and FIG. 11 described later, the data flow of the output image is shown so as to reach the projection units 21 and 22 from the image acquisition unit 12 via the image correction unit 14, but the data flow is limited to this. Not. There may be a data flow from the image acquisition unit 12 to the direct projection units 21 and 22. Regardless of which data flow is adopted, when the control unit 15 generates a corrected image such as the corrected image 14a, the corrected image is projected onto a projection unit where the obstacle is not within the angle of view, and the obstacle is generated. The image acquired by the image acquisition unit 12 is projected onto the projection unit within the angle of view. In the configuration illustrated in FIG. 1 and FIG. 11 described later, the image correction unit 14 has acquired the image correction unit 12 with respect to the projection unit that does not output the corrected image and projects the content corresponding to the image acquisition unit 12. Output corresponding to the content of the image is performed as it is.

FIG. 8 is a flowchart showing the flow of processing related to the detection of obstacles and shadow areas. As a premise that such processing is performed, an image is projected by the projection units 21 and 22 corresponding to the image acquired by the image acquisition unit 12.

First, the imaging units 31 and 32 acquire a plurality of captured images by imaging the overlapping region 81 (step S1). The shadow area identification unit 13 detects images of the shadow area and obstacles based on the captured image acquired in the process of step S1 (step S2). When the image of the shadow area and the obstacle is not detected in the process of step S2 (step S2; No), the process ends.

When an image of a shadow region and an obstacle is detected in the process of step S2 (step S2; Yes), the image correction unit 14 compensates for the decrease in the brightness of the shadow region, that is, the portion corresponding to the shadow region. A corrected image in which the brightness of the region is increased is generated (step S3). Then, the control unit 15 projects the corrected image onto the projection unit having no obstacle within the angle of view of the image projection specified based on the process of step S2 (step S4). In the case of the example shown in FIG. 7, it is the projection unit 22 that is treated as a projection unit having no obstacle within the angle of view of the image projection in the process of step S4.

After that, as in step S1, a plurality of captured images are acquired. The control unit 15 determines whether or not the decrease in brightness of the shadow region has been eliminated based on the plurality of captured images (step S5). When it is determined in the process of step S5 that the decrease in brightness of the shadow region has been eliminated (step S5; Yes), the process ends. On the other hand, if it is determined in the process of step S5 that the decrease in brightness of the shadow region has not been resolved (step S5; No), the process returns to the process of step S3.

It is desirable that the projection unit and the imaging unit treated as one set have the optical axis of the projection of the image by the projection unit and the optical axis of the image acquisition by the imaging unit as coincident as possible.

FIG. 9 is a schematic view showing an example of the relationship between the optical axes of the projection device 61 and the image pickup device 71, which are treated as one set P1. In FIG. 9, the projection device 61 and the image pickup device 71 are arranged side by side so that the optical axis of the image projected onto the overlap area 81 by the projection device 61 and the optical axis of the image pickup of the overlap area 81 by the image pickup device 71 correspond to each other. ing. As a result, the angle of view of the projection device 61 and the angle of view of the image pickup device 71 can be matched.

FIG. 10 is a schematic diagram showing a mechanism of a configuration in which an optical axis on which a projection device 61 projects an image via a half mirror 90 and an optical axis on which an image pickup device 71 images an overlapping region 81 are provided so as to overlap each other. As shown in FIG. 10, by providing the half mirror 90 on the optical axis on which the projection device 61 projects an image, the light emitted by the projection device 61 for projecting the image is transmitted, and the reflected light from the overlapping region 81 is transmitted. It is possible to reflect a part of the light and point it at an angle different from the optical axis. Specifically, as shown in FIG. 10, the image pickup device 71 is arranged so that the half mirror 90 corresponds to a reflection angle that reflects a part of the reflected light from the overlapping region 81. As a result, the optical axis on which the projection device 61 projects an image between the overlapping region 81 and the half mirror 90 can be aligned with the optical axis on which the imaging device 71 images the overlapping region 81. As a result, the angle of view of the projection device 61 and the angle of view of the image pickup device 71 can be matched with higher accuracy.

In the description with reference to FIGS. 9 and 10, the set P1 is taken as an example, but the same configuration can be obtained when there is a set P2 or a third or more set (not shown).

As described above, in the first embodiment, the decrease in brightness caused by the obstacle can be compensated by projecting the corrected image. Therefore, it is possible to prevent a change in the brightness of a part of the projected image and deterioration of the image quality such as chipping, and it is possible to more reliably maintain the image quality of the image projected on the overlapping region 81.

Further, in the first embodiment, by adopting the half mirror 90, the angle of view of the projection device 61 and the angle of view of the image pickup device 71 can be matched with higher accuracy.

[Second Embodiment]
Next, a second embodiment different from the first embodiment will be described. Regarding the description of the second embodiment, the same reference numerals are given to the same configurations as those of the first embodiment, and the description thereof will be omitted.

FIG. 11 is a functional block diagram showing the configuration of the image projection system 100 according to the second embodiment of the present invention. FIG. 12 is a block diagram showing an example of a configuration for realizing the functional block shown in FIG. As shown in FIG. 11, the image projection system 100 includes an obstacle detection unit 16 and a projection control unit 17 instead of the shadow region identification unit 13 and the image correction unit 14 included in the image projection system 1. Specifically, as shown in FIG. 12, the calculation unit 51 reads out and executes the image projection program 11c stored in the storage unit 11 included in the information processing device 50 of the image projection system 100. As a result, the obstacle detection unit 16 and the projection control unit 17 are realized as functions that replace the shadow area identification unit 13 and the image correction unit 14. In other words, the image projection program 11a and the image projection program 11c are the same except for the difference in the realized functions. The calculation unit 51 of the second embodiment functions as an image acquisition unit 12, a control unit 15, an obstacle detection unit 16 and a projection control unit 17 shown in FIGS. 11 and 12 by reading and executing the image projection program 11c. do.

The obstacle detection unit 16 has at least a function related to extracting an image of an obstacle among the functions provided by the shadow area identification unit 13. That is, the obstacle detection unit 16 detects an obstacle between one projection unit and the overlapping region based on the images captured by the imaging units 31 and 32. Here, when the obstacle detection unit 16 detects an obstacle, it may be configured to further specify a projection unit that is paired with an imaging unit that captures an captured image including an obstacle that causes a shadow region. Since the projection units associated with the imaging unit that captures the captured image are configured as one set, if the imaging unit can be specified, the corresponding projection unit can be easily specified. Such identification may be performed by the obstacle detection unit 16 or the projection control unit 17. Further, the information regarding the specified projection unit may be stored in the storage unit 11.

As a function of the obstacle detection unit 16, the function related to the extraction of the shadow area included in the shadow area identification unit 13 is not essential, but may be possessed. In the description of the second embodiment, it is assumed that the function provided by the obstacle detection unit 16 and the function provided by the shadow area identification unit 13 are the same.

When an obstacle between one projection unit and the overlapping region is detected, the projection control unit 17 projects an image on a projection unit different from the one projection unit. Further, when an obstacle is not detected, the projection control unit 17 operates a plurality of projection units so as to switch which of the plurality of projection units projects an image on the overlapping region 81 at a predetermined cycle. In this way, the projection control unit 17 switches and projects the projected image projected by the plurality of projection units 21 and 22 for each frame. Here, the projection control unit 17 uses a projection unit different from the projection unit, which is paired with the imaging unit that captures the captured image including an obstacle within the angle of view of the projection that causes the specified shadow region, to produce the projected image. It is good to project it.

FIG. 13 is a diagram showing an example of projection of an image on the overlapping region 81 and imaging of the overlapping region 81 when there is no obstacle. In the description with reference to FIGS. 13 to 15, the image formed in the overlapping region 81 is described as a projected image.

In the example shown in FIG. 13, the projection unit 21 projects an image corresponding to the projected image data 11b as the first projected image. On the other hand, during the period when the first projected image is projected, the second projected image is not projected. In FIG. 13, the fact that the second projected image is not projected is illustrated by a black filled rectangle NL. Although not shown, the configuration for projecting an image is switched at a predetermined cycle. Therefore, there is a period during which the second projected image is projected and the first projected image is not projected. Specifically, for example, the configuration for projecting an image according to the update cycle of the frame image is alternately switched between the projection unit 21 and the projection unit 22. It may be switched every frame, or it may be switched every predetermined number of frames.

As shown in FIG. 3, when there is no obstacle within the angle of view of the projection by the projection devices 61 and 62, the same as the image projected by the projection device 61 or the projection device 62 as in the projection image V4 shown in FIG. The image of is formed in the overlapping region 81. Further, as in the first captured image 31d and the second captured image 32d shown in FIG. 13, the captured images by the imaging unit 31 and the imaging unit 32 are the same images as the projected image V4.

FIG. 14 is a diagram showing an example of projecting an image on the overlapping region 81 and imaging the overlapping region 81 when there is an obstacle. FIG. 14 shows a state in which the operation control of the plurality of projection units by the projection control unit 17 in response to the detection of an obstacle has not yet been performed.

As shown in FIG. 5, when there is an obstacle OB within the angle of view of the projection by the projection device 61 and the angle of view of the image captured by the image pickup device 71, the obstacle OB is the projected light for projecting the image by the projection device 61. Block a part of. Therefore, during the period in which the projection unit 21 is projecting the image corresponding to the projected image data 11b as the first projected image, it corresponds to the light blocked by the obstacle OB as in the projected image V5 shown in FIG. An image including the shadow region DA is formed in the overlapping region 81. Further, since the obstacle OB is within the angle of view of the image captured by the imaging device 71, the image capturing unit 31 displays an image in which the obstacle OB is located inside the shadow region DA as in the first captured image 31e shown in FIG. It is imaged. On the other hand, there is no obstacle OB within the angle of view of the image captured by the imaging device 72. Therefore, as in the second captured image 32e shown in FIG. 14, the captured image by the imaging unit 32 is the same as the projected image V4.

FIG. 15 is a diagram showing an example of projection of an image on the overlapping region 81 and imaging of the overlapping region 81 after the control corresponding to the detection of the obstacle is performed. When the state shown in FIG. 14 occurs, the obstacle detection unit 16 first captures that the obstacle OB is within the angle of view of the projection by the projection device 61 and the angle of view of the image captured by the image pickup device 71. Detect based on the image 31e. After such detection, the projection control unit 17 controls the operation so that the projection device 62 projects the image on the overlapping region 81 and the projection device 61 does not project the image. As a result, as shown in FIG. 15, the first projected image is not projected, and the image corresponding to the projected image data 11b is projected as the second projected image. Since there are no obstacles within the angle of view of the projection by the projection device 62, an image similar to the image projected by the projection device 62 is formed in the overlapping region 81 as in the projection image V6 shown in FIG.

Further, since there is an obstacle within the angle of view of the image captured by the imaging device 71, the image captured by the imaging unit 31 includes an image of the obstacle OB as in the first captured image 31f shown in FIG. On the other hand, since there is no obstacle in the angle of view of the image captured by the imaging device 72, the image captured by the imaging unit 32 becomes the same image as the projected image V6 as in the second captured image 32f shown in FIG. ..

FIG. 16 is a flowchart showing the flow of processing related to the detection of obstacles and shadow areas. As a premise that such processing is performed, the projection of the image by the projection unit 21 or the projection unit 22 corresponding to the image acquired by the image acquisition unit 12 is performed while switching at a predetermined cycle.

First, the imaging units 31 and 32 acquire a plurality of captured images by imaging the overlapping region 81 (step S11). The obstacle detection unit 16 detects an image in the shadow region based on the captured image acquired in the process of step S11 (step S12). If the image in the shadow region is not detected in the process of step S12 (step S12; No), the process ends.

When the shadow region is detected in the process of step S12 (step S12; Yes), the projection control unit 17 identifies the projection unit that projected the image on the overlapping region 81 during the period when the shadow region was detected (step S12; Yes). Step S13). Then, the projection control unit 17 projects the image on a projection unit different from the projection unit specified based on the process of step S13 (step S14).

After that, as in step S11, a plurality of captured images are acquired. The obstacle detection unit 16 determines whether or not the shadow region has been eliminated based on the plurality of captured images (step S15). When it is determined in the process of step S15 that the decrease in brightness of the shadow region has been eliminated (step S15; Yes), the process ends. On the other hand, if it is determined in the process of step S15 that the decrease in brightness of the shadow region has not been resolved (step S15; No), the process returns to the process of step S13.

In the description with reference to FIGS. 14 and 15, the obstacle detection unit 16 determines in which angle of view the projection units 21 and 22 the obstacle OB is located based on the first captured image and the second captured image. It is being detected. By performing such control, there is a remarkable effect that the obstacle OB that caused the shadow region DA can be more reliably identified within which angle of view of the projection devices 61 and 62.

On the other hand, the control for eliminating the occurrence of the shadow region in the image projected on the overlapping region 81 is not limited to this. For example, the obstacle detection unit 16 identifies whether the projection unit 21 or the projection unit 22 that was projecting the image at the timing when the shadow region DA is generated in the overlapping region 81. Based on the specific result, the projection control unit 17 causes the image to be projected by a projection unit different from the specified projection unit. This also eliminates the occurrence of a shadow region in the image projected on the overlapping region 81. In this case, the number of imaging units may be one. Therefore, in the second embodiment, it is not essential that the projection unit and the imaging unit are a set. According to this, the configuration can be further simplified.

Further, as described with reference to the flowcharts of FIGS. 14, 15 and 16, the obstacle detection unit 16 has an obstacle within the angle of view of the projection by the projection device 61 and within the angle of view of the image captured by the image pickup device 71. The presence of OB is detected based on the first captured image 31e. After such detection, the projection control unit 17 controls the operation so that the projection device 62 projects the image on the overlapping region 81 and the projection device 61 does not project the image, but the operation is not limited to this.

For example, the projection control unit 17 controls the projection of a plurality of projection devices to perform projection while switching the projection device for each frame, and within the projection angle of the projection device 61 and within the image capture angle of the image pickup device 71. The presence of the obstacle OB is detected based on the first captured image 31e, and the shadow region is specified based on the second captured image 32e and the first projected image 11b. The area corresponding to the shadow area of the second projection image projected in the frame of is set high to compensate for the decrease in the brightness of the image area corresponding to the shadow area by the correction of the brightness, and the corrected second projection. The image may be switched frame by frame and projected. When switching and projecting for each frame, projection may be performed while switching, including projection by a projection device such as a projection device 61 including an obstacle OB within the angle of view of the projection. Further, even if the projection control unit 17 controls the projection of each projection device to be switched and projected at high speed so that the projection is performed at a very high frame rate when the projection is switched for each frame. good. Here, a very high frame rate is preferably a frame rate of 30 Hz or 60 Hz or higher, which is common in video display.

By projecting the image projected on the overlapping region 81 at a very high frame rate in this way, the influence of the shadow region and the influence of the corrected projected image are visually offset and the shadow is shadowed. It is possible to reduce the perception of the area.

Further, when the number of projection devices is set to three or more, the projection control unit 17 prevents the projection device having an obstacle OB within the projection angle of view from projecting, and projects the image onto the overlapping region 81. May be controlled so that it is performed on a projection device having no obstacle OB within the projection angle of view. Here, the projection device having no obstacle OB within the projection angle of view controlled by the projection control unit 17 may be a predetermined projection device, or the obstacle OB may be within a plurality of projection angles of view. It is also possible to control the projection device without the above to project while switching for each frame. In addition, when projecting while switching frame by frame with a projection device that does not have obstacles OB within a plurality of projection angles of view, the frame rate is different from the frame rate when projecting using all the projection devices. It may be controlled so as to project. For example, if it is composed of three projection devices, one of which contains an obstacle OB within the projection angle of view and projects while switching the projection device at a frame rate of 30 Hz, the obstacle is within the projection angle of view. When projecting by the remaining two projection devices that do not include the object OB, it may be controlled to project at 20 Hz, which is 2/3 of 30 Hz, for example.

FIG. 17 is a diagram showing a configuration example when there is only one imaging unit. In the example shown in FIG. 17, the image pickup device 71 and the image pickup device 72 are omitted, and the image pickup device 75 is provided in place of them. The image pickup apparatus 75 may be provided at a position independent of the projection apparatus 61 and the projection apparatus 62. In the example shown in FIG. 17, the image pickup apparatus 75 is provided at a position facing the overlapping region 81. The imaging unit included in the imaging device 75 is the same as the imaging unit 31 included in the imaging device 71 and the imaging unit 32 included in the imaging device 72, and is set so as to include the overlapping region 81 in the imaging range. In FIG. 17, the angle of view of the image captured by the imaging device 75 is shown within the acute angle formed by the two broken lines L5.

In the second embodiment, various processes and controls related to the projection of the corrected image described in the first embodiment are omitted. As described above, the second embodiment is the same as that of the first embodiment except for the matters noted.

As described above, in the second embodiment, the image quality of the projected image is more easily maintained by projecting the image by a projection unit different from the projection unit in which the obstacle is detected within the angle of view, and one of the projected images. It is possible to prevent changes in the brightness of the part and deterioration of image quality such as chipping.

Further, in the second embodiment, since the projection unit and the imaging unit are paired, it is more certain that the obstacle OB that caused the shadow region DA is within the angle of view of the projection devices 61 and 62. Can be identified.

Although two projection units 21 and 22 are illustrated in FIGS. 1 and 11, the number of projection units may be 3 or more. That is, in the configuration shown in FIG. 2 and the like, the number of projection devices may be three or more. In the first embodiment, the brightness of the projected image by one or more projection units having no obstacle on the optical axis may be corrected. The degree of correction may be such that the projection images of the plurality of projection units having no obstacles on the optical axis are overlapped so that the decrease in brightness of the shadow region due to the projection units of the obstacles on the optical axis is offset. In the second embodiment, the image may be projected by one or more projection units having no obstacle on the optical axis. That is, it is sufficient that the image is not projected by the projection unit having an obstacle on the optical axis.

Further, although two imaging units 31 and 32 are illustrated in FIGS. 1 and 11, the number of imaging units may be 3 or more. In the first embodiment, the number of imaging units corresponds to the number of projection units. In the second embodiment, as described with reference to FIGS. 9, 10 and 11, an imaging unit may be provided so as to be paired with the projection unit, or a plurality of imaging units may be provided at positions independent of the projection unit. A unit may be provided.

Further, the functional configuration shown in the image projection system 1 shown in FIG. 1 and the image projection system 100 shown in FIG. 11 may be a functional configuration included in one device. That is, the present invention is not limited to the system configuration by combining a plurality of devices as shown in FIGS. 2 and 12, and may be realized by one device having the functions shown in FIGS. 1 and 11. That is, the configuration with reference numeral 1 and the configuration with reference numeral 100 may be an image projection device. To give a specific example, the configuration with reference numeral 1 and the configuration with reference numeral 100 may be an integrated device provided like a so-called projection television.

Further, the contents of the projected image data 11b illustrated in FIG. 4 and the like, the positions and shapes of the obstacle OB and the shadow region DA illustrated in FIG. 6 and the like, and the contents of the corrected image 14a illustrated in FIG. 7 are merely examples. Not limited to. The present invention can project an image having arbitrary contents. Further, the present invention can generate a corrected image according to a shadow region generated corresponding to the shape of an obstacle within the angle of view of projection.

Further, the projection control unit 17 of the second embodiment is based on the captured image obtained by capturing the projected image projected by switching each frame by the plurality of imaging units 21 and 22, and the contents of the projected image data 11b which is the projected image. , The position of the shadow of the projected image may be specified. That is, the projection control unit 17 may have a function related to extraction of the shadow area DA included in the shadow area identification unit in the first embodiment and the obstacle detection unit 16 in the second embodiment.

Although the embodiments of the present invention have been described above, the present invention is not limited to the contents of these embodiments. The components described above include those that can be easily assumed by those skilled in the art, those that are substantially the same, that is, those in a so-called equal range. Furthermore, the components described above can be combined as appropriate. Further, various omissions, replacements or changes of the components can be made without departing from the gist of the above-described embodiment.

1,100 Image projection system 11a, 11c Image projection program 12 Image acquisition unit 13 Shadow area identification unit 14 Image correction unit 15 Control unit 16 Obstacle detection unit 17 Projection control unit 21,22 Projection unit 31,32 Imaging unit 50 Information processing Equipment 61,62 Projection equipment 71,72,75 Imaging equipment

Claims (5)

Multiple projection units that project from different image projection angles so that multiple projected images form overlapping overlapping images in the same area.
An imaging unit that captures the same region and generates an captured image,
An obstacle detection unit that detects an obstacle between the projection unit and the same region based on the captured image, and an obstacle detection unit.
A control unit that controls the projection of the projected image on the projection unit is provided.
The control unit projects the projected image by switching the plurality of projection units for each frame, and projects the projected image based on the captured image obtained by switching the projected image for each frame by the imaging unit and the projected image. An image projection device characterized in that it controls to specify the position of a shadow in an image. When an obstacle is detected by the obstacle detection unit, a projection unit that is paired with an imaging unit that captures the captured image including an obstacle that causes a shadow region is further specified.
The control unit is characterized in that the projected image is projected by a projection unit different from the projection unit that is paired with the imaging unit that captures the captured image including an obstacle that causes the specified shadow region. The image projection device according to claim 1. A plurality of the imaging units are provided so as to form a pair with each of the plurality of projection units, and the same region is imaged from an angle corresponding to the image projection angle of each of the paired projection units.
The pair of projection unit and imaging unit is characterized in that the optical axis on which the projection unit projects an image and the optical axis on which the imaging unit images the same region are provided so as to overlap with each other via a half mirror. The image projection device according to claim 1 or 2. Based on an image taken of the same area in which each projected image is projected by a plurality of projection devices projecting from different image projection angles so that a plurality of projected images form an overlapping overlapping image in the same area. The step of detecting an obstacle between the projection device and the same area,
The step of projecting the projected image on the projection device,
The projected image is projected by switching the plurality of projection devices for each frame, and the position of the shadow of the projected image is determined based on the captured image obtained by switching the projected image for each frame by the imaging device and the projected image. Steps to control to identify and
An image projection method comprising. Based on an image taken of the same area in which each projected image is projected by a plurality of projection devices projecting from different image projection angles so that a plurality of projected images form an overlapping overlapping image in the same area. The step of detecting an obstacle between the projection device and the same area,
The step of projecting the projected image on the projection device,
The projected image is projected by switching the plurality of projection devices for each frame, and the position of the shadow of the projected image is determined based on the captured image obtained by switching the projected image for each frame by the imaging device and the projected image. Steps to control to identify and
A program that causes a computer to run.

Images