WO2020170662A1 - Projection device, and method and program for controlling same - Google Patents

Abstract

The present invention provides a projection device, and a method and program for controlling the projection device, with which it is possible to correct a projection image conformed to a projection surface without being affected by a ghost of a captured image.　A system control unit (8): estimates a first captured image outputted from an image capture element (6) using a control for capturing the image of a screen SC using the image capture element (6) in a state in which the image is not projected on the screen SC, and a state in which the screen SC is illuminated with an illumination unit (7), and also estimates a second captured image obtained, on the basis of a correction coefficient stored in advance in a ROM, by capturing an image of the screen SC using the image capture element (6) in a state in which the screen SC is not illuminated with the illumination unit (7), and a state in which a test image is projected on the screen SC; and generates correction data for correcting the color and/or brightness of an image to be projected on the screen SC, the correction being made on the basis of the characteristic differences between the estimated second captured image and the test image.

Description

Projection apparatus, control method thereof, and control program

The present invention relates to a projection device, a control method thereof, and a control program.

A system that combines an imaging device and a projection device has been proposed. For example, in Patent Document 1, a half mirror that transmits the light from the projection unit to the screen and guides the light from the screen to the imaging unit, and a lens arranged between the half mirror and the projection unit, A projector having this half mirror and a lens arranged between the image pickup unit is described.

Patent Document 2 describes a projector that includes a projection unit and a color imaging element that images a projection surface on which light is projected from the projection unit, in the same housing. In this projector, a captured image obtained by capturing an image of the projection surface with a color image sensor is divided into a plurality of areas, and the color of the projection surface corresponding to the area should be detected based on the image of each area and projected. Image color correction is performed.

Japanese Patent Laid-Open No. 2016-149618 Japanese Patent Laid-Open No. 2006-349792

As described in Patent Document 2, when a projection image is corrected according to the state of the projection surface, a reference image such as a white image is projected onto the projection surface in order to detect the state of the projection surface. In that state, it is necessary to image the projection surface. However, in the system described in Patent Document 1 in which imaging and projection are performed using a common optical system, strong light from a light source causes the optical system to project when an image is projected on the projection surface. It will pass. If the projection surface is imaged in this state, strong light for projection may be mixed into the imaging element, and a ghost may occur in the captured image. When a ghost is generated in the captured image, it becomes impossible to accurately recognize the state of the projection surface on which the reference image is projected, and it becomes impossible to correct the projected image with high accuracy.

The present invention has been made in view of the above circumstances, and provides a projection device and a control method and a control program therefor capable of correcting a projection image in accordance with a projection surface without being affected by a ghost of a captured image. The purpose is to

A projection device of the present invention illuminates the optical system for projecting an image from a display unit onto a projection target object, an imaging device for imaging the projection range of the image by the optical system through the optical system, and the projection target object. An illumination unit, and an imaging control unit that performs at least control to image the projection target by the imaging device in a state where the image is not projected on the projection target and the projection target is illuminated by the illumination unit. Based on the first captured image output from the image sensor by the control and the information stored in the storage unit in advance, the projection target object is not illuminated by the illumination unit and a test image is displayed. Image estimation unit that estimates a second captured image obtained by capturing an image of the projection target by the image sensor in the state of being projected on the projection target, and characteristics of the test image and the second captured image A correction data generation unit that generates correction data for correcting at least one of the color and the brightness of the image to be projected on the projection target based on the difference.

A control method of a projection device according to the present invention includes an optical system for projecting an image from a display unit onto a projection target object, an imaging device for imaging a projection range of the image by the optical system through the optical system, and the projection target object. A method of controlling a projection device, comprising: an illuminating unit that illuminates the image capturing device, the image capturing device in a state where the image is not projected onto the projection target and the projection target is illuminated by the lighting unit. The projection target based on an imaging control step for performing at least control of imaging the projection target, a first captured image output from the image sensor by the control, and information stored in advance in a storage unit. An image estimation step of estimating a second captured image obtained by capturing an image of the projection target by the image sensor in a state where an object is not illuminated by the illumination unit and a test image is projected on the projection target. , Correction data generation for generating correction data for correcting at least one of color and brightness of the image to be projected on the projection target based on the characteristic difference between the test image and the second captured image And a step.

A control program for a projection device according to the present invention is an optical system for projecting an image from a display unit onto a projection target object, an image sensor for capturing a projection range of the image by the optical system through the optical system, and the projection target object. Is a control program of a projection device having an illuminating section for illuminating the object, wherein the image pickup element is used in a state where the image is not projected on the projection object and the projection object is illuminated by the illuminating section. The projection target based on an imaging control step for performing at least control of imaging the projection target, a first captured image output from the image sensor by the control, and information stored in advance in a storage unit. An image estimation step of estimating a second captured image obtained by capturing an image of the projection target by the image sensor in a state where an object is not illuminated by the illumination unit and a test image is projected on the projection target. , Correction data generation for generating correction data for correcting at least one of color and brightness of the image to be projected on the projection target based on the characteristic difference between the test image and the second captured image The steps are for causing a computer to execute the steps.

According to the present invention, it is possible to provide a projection apparatus, a control method and a control program therefor capable of correcting a projection image that matches a projection surface without being affected by a ghost of a captured image.

FIG. 1 is a schematic diagram showing an external configuration of a projector 100 that is an embodiment of a projection device of the present invention. FIG. 2 is a schematic diagram showing an internal configuration of projector 100 shown in FIG. 1. It is a schematic diagram which shows the structural example of the display part 1 shown in FIG. FIG. 4 is a functional block diagram of a system control unit 8 shown in FIG. 3. 7 is a flowchart for explaining a correction coefficient generation process. 9 is a flowchart for explaining a process of estimating a second captured image by the image estimation unit 82. 9 is a flowchart for explaining a modified example of the estimation process of the second captured image by the image estimation unit 82.

FIG. 1 is a schematic diagram showing an external configuration of a projector 100 that is an embodiment of a projection device of the present invention. The projector 100 includes a main body 100A and an optical unit 100B supported by the main body 100A. The optical unit 100B may be configured to be detachable from the main body 100A.

FIG. 2 is a schematic diagram showing an internal configuration of the projector 100 shown in FIG. FIG. 3 is a schematic diagram showing a configuration example of the display unit 1 shown in FIG. The projector 100 is configured to project an image on a screen SC which is one of the projection objects and at least to capture an image of a projection range on the screen SC where the image is projected.

As shown in FIG. 2, the projector 100 includes a display unit 1, a projection optical system 2, a common optical system 3 forming an optical system, an optical member 4, an image pickup optical system 5, an image pickup element 6, and an illumination. The unit 7 and the system control unit 8 are provided. The display unit 1 and the system control unit 8 are housed in the main body 100A, for example. The projection optical system 2, the common optical system 3, the optical member 4, the imaging optical system 5, and the imaging device 6 are housed in the optical unit 100B, for example.

The display unit 1 displays an image for projection based on the input image data. As shown in FIG. 3, the display unit 1 includes a light source unit 40 and a light modulation element 44.

The light source unit 40 includes a light source 41 that emits white light, a color wheel 42, and an illumination optical system 43. The light source 41 is configured to include a light emitting element such as a laser or an LED (Light Emitting Diode). The color wheel 42 is arranged between the light source 41 and the illumination optical system 43. The color wheel 42 is a disk-shaped member, and is provided with an R filter that transmits red light, a G filter that transmits green light, and a B filter that transmits blue light along the circumferential direction thereof. The color wheel 42 is rotated around an axis, and white light emitted from the light source 41 is time-divided into red light, green light, and blue light, which are guided to the illumination optical system 43. The light emitted from the illumination optical system 43 enters the light modulation element 44.

The light modulation element 44 spatially modulates the light emitted from the illumination optical system 43 based on the image data, and emits the spatially modulated light to the projection optical system 2 (see FIG. 2).

The display unit 1 shown in FIG. 3 is an example in which a DMD (Digital Micromirror Device) is used as the light modulation element 44. It is also possible to use a (Systems) element, a liquid crystal display element, or the like.

The display unit 1 may display an image using a self-luminous organic EL (electro-luminescence) display element and cause the displayed image to enter the projection optical system 2. Alternatively, a device that displays an image by scanning laser light may be used.

The projection optical system 2 is a system to which light from the display unit 1 is incident, and is composed of, for example, a relay optical system including at least one lens. The light that has passed through the projection optical system 2 enters the optical member 4, is reflected by the optical member 4, and enters the common optical system 3.

The common optical system 3 projects the light that has passed through the projection optical system 2 onto the screen SC and forms an image of a subject on the screen SC side, and is configured by, for example, a relay optical system.

In the example of FIG. 1, the common optical system 3 includes a lens group 31 including at least one lens, an optical member 32, a lens group 33 including at least one lens, and a lens group 34 including at least one lens. Equipped with. The lens group 31, the optical member 32, the lens group 33, and the lens group 34 are arranged on the optical path in this order from the screen SC side. As shown in FIG. 1, the lens 31a located closest to the screen SC of the lens group 31 is exposed from the housing of the optical unit 100B.

The optical member 32 is a member for bending the optical path of the common optical system 3, and for example, a half mirror, a beam splitter, or a polarizing member is used.

The image from the display unit 1 that has passed through the projection optical system 2 passes through the projection optical system 2, the optical member 4, and the common optical system 3 in this order and is projected on the screen SC. On the screen SC, the range where the image is projected from the common optical system 3 is called the projection range of the screen SC.

The subject light that has entered the lens group 31 from the screen SC side passes through the lens group 31, is reflected by the optical member 32, and enters the lens group 33. The lens group 33 forms an intermediate image formed by the subject light at a position IG between the lens group 33 and the lens group 34. This intermediate image passes through the lens group 34, enters the optical member 4, passes through the optical member 4, and enters the imaging optical system 5.

The image pickup optical system 5 is for forming an intermediate image formed at the position IG on the image pickup element 6. The image pickup optical system 5 is arranged in front of the image pickup device 6 and focuses the subject light transmitted through the optical member 4 to form an image on the image pickup device 6. The imaging optical system 5 is configured to include at least one lens.

As the image sensor 6, for example, a CCD (Charge Coupled Device) image sensor or a CMOS (Complementary Metal Oxide Semiconductor) image sensor is used.

The optical member 4 is composed of, for example, a half mirror, a beam splitter, or a polarizing member. The optical member 4 reflects the image from the display unit 1 that has passed through the projection optical system 2 and guides it to the common optical system 3, and the subject light imaged at the position IG by the lens group 33 of the common optical system 3 is reflected. The intermediate image based on the image is guided to the image pickup device 6 through the image pickup optical system 5. In this way, the image pickup device 6 picks up an image projection range of the common optical system 3 through the common optical system 3 for projecting the image from the display unit 1 on the screen SC.

The illumination unit 7 is for illuminating a projection target such as the screen SC, and as shown in FIG. 1 as an example, a plurality of light emitting elements 7a arranged in a circle around the optical axis of the lens 31a. And a driver for driving them.

The system control unit 8 controls an image to be incident on the projection optical system 2 from the display unit 1, image pickup control of a subject by the image pickup device 6, light emission control of the illumination unit 7, and the like, and includes a processor and a storage unit. It is provided with a ROM (Read Only Memory) and a RAM (Random Access Memory) which are configured.

As a processor, a programmable logic device (CPU), which is a general-purpose processor that executes programs to perform various processes, that can change the circuit configuration after manufacturing such as CPU (Central Processing Unit) and FPGA (Field Programmable Gate Array) ( Included is a dedicated electrical circuit that is a processor having a circuit configuration specifically designed to execute a specific process such as Programmable Logic Device (PLD) or ASIC (Application Specific Integrated Circuit). More specifically, the structures of these processors are electric circuits in which circuit elements such as semiconductor elements are combined.

The processor of the system control unit 8 may be configured by one of the various processors described above, or may be a combination of two or more processors of the same type or different types (for example, a combination of a plurality of FPGAs or a combination of a CPU and an FPGA). Combination).

The projector 100 having the above configuration uses a projection mode when projecting a desired image on a desired projection target. In this projection mode, the image is displayed in a state in which the image from the display unit 1 is projected onto the projection target object and the projection target object is not illuminated by the illumination unit 7 (hereinafter referred to as “with projection/without illumination”). Done.

　Here, I will explain the overall flow when the projection mode is set. When the projector 100 is installed in front of the screen SC and the projection mode is set in that state, first, the system control unit 8 causes the screen SC to be in a non-illuminated state by the illumination unit 7 and to provide a test image (for example, a white image). It is possible to obtain an image of the screen SC by the image pickup device 6 in that state without forming a state (hereinafter, also referred to as “no illumination/test image projection state”) projected on the screen SC. The second captured image is estimated by calculation. After that, the system control unit 8 generates correction data for correcting the projection image based on the estimated second captured image and the test image. When the generation of the correction data is completed, the system control unit 8 corrects the image to be projected based on the correction data, and projects the corrected image from the display unit 1 on the screen SC. As a result, a desired image is displayed.

FIG. 4 is a functional block diagram of the system control unit 8 shown in FIG. The processor of the system control unit 8 functions as the imaging control unit 81, the image estimation unit 82, the correction data generation unit 83, and the image projection control unit 84 by executing the program including the control program of the projection device.

The imaging control unit 81, in a state where the image from the display unit 1 is not projected on the projection target object and the projection target object is illuminated by the illumination unit 7 (hereinafter, referred to as “no projection/illumination state”), At least control for imaging the projection target by the imaging element 6 (hereinafter referred to as first imaging control) is performed. The “non-projection/illumination state” is realized, for example, by controlling the display unit 1 to the off state and causing the illumination unit 7 to emit light.

Further, the image pickup control unit 81 projects the image from the display unit 1 onto the projection target object and causes the imaging unit 6 to image the projection target object while the illumination unit 7 does not illuminate the projection target object. It also controls.

The image estimation unit 82 includes a first captured image output from the image sensor 6 by the first image capturing control of the image capturing control unit 81, and information stored in advance in the ROM in the system control unit 8 (hereinafter, correction coefficient). The above second captured image obtained by capturing the screen SC by the image sensor 6 in the "no illumination/test image projection state" is estimated based on the above.

In order to actually acquire this second captured image, it is necessary to capture the screen SC with the image sensor 6 while projecting the test image onto the screen SC. However, in this state, a ghost may occur in the captured image output from the image sensor 6. Therefore, in the present embodiment, the image estimation unit 82 can eliminate the influence of the ghost by estimating the second captured image by calculation without projecting the test image.

The details of the method for estimating the second captured image by the image estimation unit 82 will be described below, but before that, the details of the correction coefficient used for estimating the second captured image will be described.

The projector 100 is equipped with a calibration mode for generating a correction coefficient and storing it in the ROM. FIG. 5 is a flowchart for explaining the correction coefficient generation process. Before the projector 100 is set to the calibration mode, first, the reference projection target object is prepared. As the reference projection object, for example, a monochromatic screen such as white without a pattern (hereinafter referred to as a reference screen) is used.

When the reference screen is installed in front of the projector 100 and the calibration mode is set, the imaging control unit 81 controls the display unit 1 and the illumination unit 7 to form the “no projection/illumination state”. In this state, the image pickup device 6 causes the reference screen to be picked up (step S1). Step S1 corresponds to the above-mentioned first imaging control.

The system control unit 8 acquires the captured image Ga of the projection range in the captured image output from the image sensor 6 by the imaging in step S1 and stores it in the RAM (step S2). The captured image Ga constitutes the third captured image.

Next, the imaging control unit 81 controls the display unit 1 to project the test image on the reference screen, and further controls the illumination unit 7 to control the reference screen to the non-illuminated state. The image pickup device 6 causes the image pickup device 6 to pick up an image of the reference screen (referred to as "with test image projection/without illumination") (step S3). Step S3 corresponds to the above-mentioned second image pickup control. Note that in this step S3, another projector of the same type as the projector 100 may be prepared, a test image may be projected from this projector, and in that state, the reference screen may be captured by the projector 100. This makes it possible to obtain a captured image without being affected by the ghost.

The system control unit 8 acquires the captured image Gb of the projection range in the captured image output from the image sensor 6 by the imaging in step S3 and stores it in the RAM (step S4). The captured image Gb constitutes the fourth captured image.

Next, the system control unit 8 sets the pixel value ga(x, y) at the coordinates (x, y) of the captured image Ga and the pixel value gb(x, y) at the coordinates (x, y) of the captured image Gb. The process of calculating the ratio β(x, y) is performed for all coordinates, and the ratio β(x, y) is stored in the ROM as a correction coefficient (step S5).

In step S5, specifically, the system control unit 8 generates a correction coefficient for each coordinate (x, y) by the calculation of the following expression (A).

Β(x,y)={pixel value gb(x,y)}/{pixel value ga(x,y)}(A)

The calibration mode is now complete. Next, details of the second captured image estimation process will be described.

FIG. 6 is a flowchart for explaining the estimation processing of the second captured image by the image estimation unit 82. The process shown in FIG. 6 is started when the projector 100 is installed in front of the screen SC and the projection mode is set in that state.

First, the imaging control unit 81 controls the display unit 1 and the illumination unit 7 to form a “no projection/illumination state”. In addition, here, the imaging control unit 81 controls the light emission amount of the illumination unit 7 to the first light amount L1. Then, in this “without projection/with illumination state”, the imaging control unit 81 causes the imaging element 6 to perform imaging of the screen SC (step S11).

The image estimation unit 82 acquires the captured image C1 of the projection range in the captured image output from the image sensor 6 by the imaging in step S11 and stores it in the RAM (step S12). The captured image C1 constitutes a first captured image.

Next, the imaging control unit 81 changes the light emission amount of the illumination unit 7 to the second light amount L2 different from the first light amount L1 while maintaining the “no projection/illumination state”, and the state is changed to that state. Then, the image pickup device 6 causes the image pickup of the screen SC to be executed (step S13).

The image estimation unit 82 acquires the captured image C2 of the projection range in the captured image output from the image sensor 6 by the imaging in step S13 and stores it in the RAM (step S14). The captured image C2 constitutes a first captured image.

Next, the image estimation unit 82 obtains the ratio α of the first light amount L1 and the second light amount L2, and based on this ratio α, the captured image C1, and the captured image C2, the captured image C1 and the captured image C1. The reflected light amount component RL(x, y) from the screen SC and the ambient light amount component EL(x, y) included in the pixel value of each coordinate (x, y) of C2 and the place where the projector 100 is placed. And are calculated (step S15).

Note that the pixel value of the coordinate (x, y) of the captured image C1 is C1(x, y), the pixel value of the coordinate (x, y) of the captured image C2 is C2(x, y), and α=L2/ If L1 is set, the relationship between the following expression (B) and expression (C) is established.

C1(x,y)=RL(x,y)+EL(x,y) (B)
C2(x,y)=α×RL(x,y)+EL(x,y) (C)

Therefore, the image estimation unit 82 calculates the formula (B), the formula (C), C1(x, y) acquired in step S12, and C2(x, y) acquired in step S14. From the ratio α, the reflected light amount component RL(x, y) and the ambient light amount component EL(x, y) can be derived.

The reflected light amount component RL(x, y) and the ambient light amount component EL(x, y) calculated in this way are information in the “no projection/illumination state”. On the other hand, the correction coefficient stored in the ROM is obtained in the captured image (captured image Ga in FIG. 5) obtained in the “without projection/with illumination state” and in the “test image projection/without illumination state”. This is the ratio with the captured image (captured image Gb in FIG. 5). Further, the ambient light amount component EL(x, y) does not change between “without projection/with illumination” and “with test image projection/without illumination”. Therefore, by correcting only the reflected light amount component RL(x, y) to the value in the “test image projection/non-illumination state” using this correction coefficient, the “test image projection/non-illumination state” is obtained. It is possible to estimate each pixel value of the second captured image obtained in ".

Specifically, after step S15, the image estimation unit 82 calculates the pixel value G(x,y) of the coordinate (x,y) of the second captured image to be estimated by the calculation of the following formula (D). , Estimate the second captured image.

G(x,y)=β(x,y)×RL(x,y)+EL(x,y) (D)

The above is the details of the estimation process of the second captured image by the image estimation unit 82.

The correction data generation unit 83 shown in FIG. 4 operates in the projection mode based on the characteristic difference between the second captured image estimated by the image estimation unit 82 and the test image used in the calibration mode. The correction data for correcting the color or brightness of the image to be projected on the screen SC is generated. The generated correction data is stored in the ROM of the system control unit 8.

The characteristic difference between the test image and the second captured image is specifically the difference in color or brightness between the images.

The correction data generation unit 83, for example, compares the test image with the second captured image, recognizes an area where the color difference is equal to or more than a predetermined value as a pattern within the projection range, and displays the image on the pattern portion. When projected, color correction data for making it difficult to visually recognize this pattern is generated. Alternatively, the correction data generation unit 83 compares the test image with the second captured image, recognizes an area in which the difference in brightness is equal to or more than a predetermined value as a pattern within the projection range, and displays the image in the pattern portion. Brightness correction data for making it difficult to visually recognize this pattern when projected is generated. The correction data generation unit 83 may generate both the brightness correction data and the color correction data and store them in the ROM.

In the projection mode, the image projection controller 84 shown in FIG. 4 corrects the color and/or the brightness of the image data to be input to the light modulation element 44 based on the correction data stored in the ROM, and corrects the correction. The subsequent image data is input to the light modulation element 44, and an image based on the corrected image data is projected on the screen SC.

By the correction processing by the image projection control unit 84, even if there is a pattern on the screen SC, the image can be visually recognized in the state where the pattern is erased.

As described above, according to the projector 100, in the projection mode, it is possible to estimate the captured image of the projection range in the state where the test image is projected on the screen SC without projecting the test image on the screen SC. Therefore, the influence of the ghost can be eliminated, and the accuracy of the correction data generated by the correction data generation unit 83 can be increased. Therefore, even if there is a pattern on the screen SC, the pattern can be erased with high accuracy to display the image, and the quality of the projected image can be improved.

Further, the projector 100 is equipped with a calibration mode in which a correction coefficient necessary for estimating the second captured image is generated and stored in the ROM. The characteristics of the display unit 1, the image pickup device 6, and other components such as the optical system may change due to deterioration over time or the ambient temperature of use. Therefore, for example, the user can optimize the correction coefficient according to the state or the usage state of the projector 100 by setting the calibration mode and generating and storing the correction coefficient immediately before the user sets the projection mode. become. As a result, the projection image can be corrected with high accuracy using the correction data, regardless of the state or usage of the projector 100.

FIG. 7 is a flow chart for explaining a modified example of the estimation process of the second captured image by the image estimation unit 82. Since the flowchart shown in FIG. 7 is the same as that of FIG. 6 from step S11 to step S14, the description is omitted. Of the steps S11 to S14 shown in FIG. 7, the control performed in step S11 constitutes the first control, and the control performed in step S13 constitutes the second control.

After step S14 of FIG. 7, the imaging control unit 81 sets the light emission amount of the illumination unit 7 to be different from the first light amount L1 and the second light amount L2 while keeping the “no projection/illumination state”. The light amount is changed to the third light amount L3, and in this state, the image pickup device 6 causes the screen SC to be imaged (step S21). The control performed in step S21 constitutes the third control.

The image estimation unit 82 acquires the captured image C3 of the projection range in the captured image output from the image sensor 6 by the imaging in step S21 and stores it in the RAM (step S22). The captured image C3 constitutes a first captured image.

Next, the image estimation unit 82 obtains a ratio α1 of the first light amount L1 and the second light amount L2, and based on the ratio α1, the captured image C1, and the captured image C2, the captured image C1 and the captured image C1. The reflected light amount component RL1(x, y) from the screen SC and the ambient light amount component EL1(x, y) included in the pixel value of each coordinate (x, y) of C2 and the place where the projector 100 is placed. And are calculated (step S23).

The pixel value of the coordinate (x, y) of the captured image C1 is C1(x, y), the pixel value of the coordinate (x, y) of the captured image C2 is C2(x, y), and α1=L2/L1 Then, the relationship between the following expression (B1) and expression (C1) is established.

C1(x,y)=RL1(x,y)+EL1(x,y) (B1)
C2(x,y)=α1×RL1(x,y)+EL1(x,y) (C1)

Therefore, the image estimation unit 82 calculates the formula (B1), the formula (C1), C1(x, y) acquired in step S12, and C2(x, y) acquired in step S14. From the ratio α1, the reflected light amount component RL1(x, y) and the ambient light amount component EL1(x, y) can be derived.

Next, the image estimation unit 82 obtains a ratio α2 of the second light amount L2 and the third light amount L3, and based on this ratio α2, the captured image C2, and the captured image C3, the captured image C2 and the captured image C2. The reflected light amount component RL2(x, y) from the screen SC and the ambient light amount component EL2(x, y) contained in the pixel value of each coordinate (x, y) of C3 and the place where the projector 100 is placed. And are calculated (step S24).

The pixel value of the coordinate (x, y) of the captured image C2 is C2(x, y), the pixel value of the coordinate (x, y) of the captured image C3 is C3(x, y), and α2=L3/L2 Then, the relationship between the following expression (B2) and expression (C2) is established.

C2(x,y)=RL2(x,y)+EL2(x,y) (B2)
C3(x,y)=α2×RL2(x,y)+EL2(x,y) (C2)

Therefore, the image estimation unit 82 calculates the formula (B2), the formula (C2), the C2(x, y) acquired in step S14, and the C3(x, y) acquired in step S22. From the ratio α2, the reflected light amount component RL2(x, y) and the ambient light amount component EL2(x, y) can be derived.

Next, the image estimation unit 82 obtains the absolute value AB1 of the first difference between the reflected light amount component RL1(x, y) and the reflected light amount component RL2(x, y), and obtains it as the environmental light amount component EL1(x, y) The absolute value AB2 of the second difference between the ambient light amount components EL2(x, y) is obtained. Then, the image estimation unit 82 determines whether or not the absolute value AB1 exceeding the predetermined threshold TH exists and the absolute value AB2 exceeding the threshold TH exists (step S25).

When the absolute value AB1 and the absolute value AB2 that exceed the threshold value TH do not exist (step S25: NO), the image estimation unit 82 uses, for example, one of the following formulas (D1), (D2), and (D3). The second captured image is estimated (step S28).

G(x,y)=β(x,y)×RL1(x,y)+EL1(x,y) (D1)
G(x,y)=β(x,y)×RL2(x,y)+EL2(x,y) (D2)
G(x,y)=β(x,y)×{RL1(x,y)+RL2(x,y)}/2+{EL1(x,y)+EL2(x,y)}/2 ( D3)

If either the absolute value AB1 or the absolute value AB2 that exceeds the threshold value TH exists (step S25: YES), the image estimation unit 82 determines that the estimation is impossible (step S26). Then, after step S26, the image estimation unit 82 instructs the imaging control unit 81 to change at least one of the light emission amounts (L1, L2, L3) from the illumination unit 7. In response to this instruction, in step S27, at least one of the light emission amounts (L1, L2, L3) is changed, and then the processing from step S11 onward is performed again.

According to the above modification example, the estimation accuracy of the second captured image can be improved, and the correction of the projection image using the correction data can be performed with higher accuracy.

Note that, in the above description, the projector 100 has the calibration mode, but the calibration mode is not essential. The correction coefficient to be stored in the ROM may be generated and stored in the factory when the projector 100 is manufactured.

As described above, the following items are disclosed in this specification.

(1)
An optical system that projects an image from the display unit onto a projection target,
An image pickup device for picking up the projection range of the image by the optical system through the optical system,
An illumination unit that illuminates the projection target,
An imaging control unit that performs at least control to image the projection target by the image sensor in a state where the image is not projected on the projection target and the projection target is illuminated by the illumination unit;
Based on the first captured image output from the image sensor by the control and the information stored in advance in the storage unit, the projection target is not illuminated by the illumination unit and the test image is projected. An image estimation unit that estimates a second captured image obtained by imaging the projection target by the image sensor in the state of being projected on the target,
A correction data generation unit that generates correction data for correcting at least one of the color and the brightness of the image to be projected on the projection target based on the characteristic difference between the test image and the second captured image. And a projection device.

(2)
The projection device according to (1),
The information stored in the storage unit is the reference projection by the image pickup device in a state in which a predetermined reference projection object is illuminated by the illumination unit and the image is not projected on the reference projection object. The third imaged image obtained by imaging the object, and the reference projection object in a state where the illumination unit is not illuminated and the test image is projected on the reference projection object by the image sensor. A projection device that is generated based on a fourth captured image obtained by capturing the reference projection target.

(3)
(2) The projection device as described above,
The said information is a projection apparatus which is the ratio of the said 3rd captured image and the said 4th captured image.

(4)
The projection device according to any one of (1) to (3),
The imaging control unit performs the control a plurality of times by changing the brightness of the illumination light from the illumination unit,
The image estimating unit is based on a plurality of the first captured images output from the image sensor by the control of the plurality of times, the brightness of the illumination light in the plurality of times of the control, and the information. And a projection device that estimates the second captured image.

(5)
(4) The projection device as described above,
The image estimation unit, based on the plurality of first captured images, and the brightness of the illumination light in the plurality of times of the control, the amount of reflected light from the projection target, the placement of the projection device. A projection device that calculates the ambient light amount of a place, adds the ambient light amount to a value obtained by correcting the reflected light amount with the information, and estimates the second captured image.

(6)
(5) The projection device as described above,
The above-mentioned imaging control part performs the above-mentioned control 3 times of the 1st control, the 2nd control, and the 3rd control as the above-mentioned control of the above-mentioned plurality of times,
The image estimation unit calculates the two first captured images output from the image sensor by the first control and the second control, and the brightness of the illumination light in the first control and the second control. Based on the first reflected light amount and the first environmental light amount, the two first captured images output from the image sensor by the second control and the third control, and the second Based on the brightness of the illumination light in the control and the third control, to calculate the second reflected light amount and the second environmental light amount, the first reflected light amount and the second reflected light amount A projection device that disables estimation of the second captured image when one of the differences and a second difference between the first ambient light amount and the second ambient light amount exceeds a threshold value.

(7)
The projection device according to (6),
The projection apparatus which makes the said control of the said 3 times again which changed the brightness of the said illumination light, when the said image estimation part makes the estimation of the said 2nd captured image impossible.

(8)
The projection device according to any one of (1) to (7),
A projection apparatus having a mode of generating the information and storing the information in the storage unit.

(9)
Projection having an optical system for projecting an image from a display unit onto a projection target object, an image sensor for capturing a projection range of the image by the optical system through the optical system, and an illumination unit for illuminating the projection target object A method for controlling a device,
An imaging control step for performing at least control to image the projection target by the image sensor in a state where the image is not projected on the projection target and the projection target is illuminated by the illumination unit;
Based on the first captured image output from the image sensor by the control and the information stored in advance in the storage unit, the projection target is not illuminated by the illumination unit and the test image is projected. An image estimation step of estimating a second captured image obtained by imaging the projection target by the image sensor in the state of being projected on the target;
A correction data generation step of generating correction data for correcting at least one of the color and the brightness of the image to be projected on the projection target based on the characteristic difference between the test image and the second captured image. And a method for controlling a projection device, comprising:

(10)
(9) A method of controlling a projection device according to the above item,
The information stored in the storage unit is the reference projection by the image pickup device in a state in which a predetermined reference projection object is illuminated by the illumination unit and the image is not projected on the reference projection object. The third imaged image obtained by imaging the object, and the reference projection object in a state where the illumination unit is not illuminated and the test image is projected on the reference projection object by the image sensor. A method for controlling a projection device, which is generated based on a fourth captured image obtained by capturing the reference projection target.

(11)
(10) A method for controlling a projection device according to the above item,
The said information is a ratio of the said 3rd picked-up image and the said 4th picked-up image, The control method of the projection apparatus.

(12)
The control method for a projection device according to any one of (9) to (11),
The imaging control step, the control is performed a plurality of times by changing the brightness of the illumination light from the illumination unit,
The image estimation step is based on the plurality of first captured images output from the image sensor by the control of the plurality of times, the brightness of the illumination light in the plurality of times of the control, and the information. And a method of controlling the projection device for estimating the second captured image.

(13)
A method for controlling a projection device according to (12),
In the image estimating step, based on the plurality of first captured images and the brightness of the illumination light in the plurality of times of control, the amount of reflected light from the projection target and the placement of the projection device are set. A method for controlling a projection device, which calculates an ambient light amount of a place, adds the ambient light amount to a value obtained by correcting the reflected light amount based on the information, and estimates the second captured image.

(14)
A method of controlling a projection device according to (13), comprising:
The above-mentioned imaging control step performs the above-mentioned control 3 times of the 1st control, the 2nd control, and the 3rd control as the above-mentioned control of a plurality of times,
In the image estimation step, the two first captured images output from the image sensor by the first control and the second control and the brightness of the illumination light in the first control and the second control are included. Based on the first reflected light amount and the first environmental light amount, the two first captured images output from the image sensor by the second control and the third control, and the second Based on the brightness of the illumination light in the control and the third control, to calculate the second reflected light amount and the second environmental light amount, the first reflected light amount and the second reflected light amount If one of the differences and the second difference between the first ambient light amount and the second ambient light amount exceeds a threshold value, the projection device of the projection device that disables the estimation of the second captured image. Control method.

(15)
A method of controlling a projection device according to (14), comprising:
In the image estimating step, when the estimation of the second captured image is disabled, the control method of the projection device that causes the control to be repeated three times with the brightness of the illumination light changed.

(16)
Projection having an optical system for projecting an image from a display unit onto a projection target object, an image sensor for capturing a projection range of the image by the optical system through the optical system, and an illumination unit for illuminating the projection target object A control program for the device,
An imaging control step for performing at least control to image the projection target by the image sensor in a state where the image is not projected on the projection target and the projection target is illuminated by the illumination unit;
Based on the first captured image output from the image sensor by the control and the information stored in advance in the storage unit, the projection target is not illuminated by the illumination unit and the test image is projected. An image estimation step of estimating a second captured image obtained by imaging the projection target by the image sensor in the state of being projected on the target;
A correction data generation step of generating correction data for correcting at least one of the color and the brightness of the image to be projected on the projection target based on the characteristic difference between the test image and the second captured image. And a control program of a projection device for causing a computer to execute the following.

Although various embodiments have been described with reference to the drawings, it goes without saying that the present invention is not limited to such examples. It is obvious to those skilled in the art that various changes or modifications can be conceived within the scope described in the claims, and naturally, these also belong to the technical scope of the present invention. Understood. Further, the constituent elements in the above-described embodiments may be arbitrarily combined without departing from the spirit of the invention.

This application is based on the Japanese patent application filed on February 19, 2019 (Japanese Patent Application No. 2019-027524), the contents of which are incorporated herein by reference.

According to the present invention, it is possible to provide a projection device capable of correcting a projection image that matches the color of the projection surface, a control method thereof, and a control program.

100 projector 100A body part 100B optical unit 7a light emitting element 1 display section 40 light source unit 41 light source 42 color wheel 43 illumination optical system 44 light modulation element 2 projection optical system 3 common optical systems 31, 33, 34 lens group 32 optical member IG position SC screen 4 Optical member 5 Imaging optical system 6 Imaging device 7 Illumination unit 8 System control unit 81 Imaging control unit 82 Image estimation unit 83 Correction data generation unit 84 Image projection control unit

Claims (16)

1. An optical system that projects an image from the display unit onto a projection target,
   An image pickup device for picking up an image of a projection range of the image by the optical system through the optical system,
   An illumination unit that illuminates the projection target,
   An imaging control unit that performs at least control of imaging the projection target by the image sensor in a state where the image is not projected on the projection target and the projection target is illuminated by the illumination unit,
   Based on a first captured image output from the image sensor by the control and information stored in advance in a storage unit, the projection target is in a non-illuminated state by the illumination unit and the test image is projected. An image estimation unit that estimates a second captured image obtained by capturing an image of the projection target by the image sensor in a state of being projected on the target,
   A correction data generation unit that generates correction data for correcting at least one of the color and the brightness of the image to be projected on the projection target based on the characteristic difference between the test image and the second captured image. And a projection device.
2. The projection device according to claim 1, wherein
   The information stored in the storage unit is the reference projection by the imaging device in a state where a predetermined reference projection target is illuminated by the illumination unit and the image is not projected on the reference projection target. A third captured image obtained by capturing an image of the object, and the reference projection object by the image sensor in a state where the illumination unit does not illuminate and the test image is projected on the reference projection object. A projection device that is generated based on a fourth captured image obtained by capturing the reference projection target.
3. The projection device according to claim 2, wherein
   The projection device, wherein the information is a ratio of the third captured image to the fourth captured image.
4. The projection device according to any one of claims 1 to 3,
   The imaging control unit performs the control a plurality of times by changing the brightness of the illumination light from the illumination unit,
   The image estimating unit is based on a plurality of the first captured images output from the image sensor by the control of the plurality of times, the brightness of the illumination light in the control of the plurality of times, and the information. And a projection device that estimates the second captured image.
5. The projection device according to claim 4, wherein
   The image estimation unit sets the amount of light reflected from the projection target and the projection device based on the plurality of first captured images and the brightness of the illumination light in the plurality of times of the control. A projection device that calculates the ambient light amount of a place, adds the ambient light amount to a value obtained by correcting the reflected light amount with the information, and estimates the second captured image.
6. The projection device according to claim 5, wherein
   The imaging control unit performs the control three times as the first control, the second control, and the third control, as the plurality of times of the control,
   The image estimation unit determines the two first captured images output from the image sensor by the first control and the second control and the brightness of the illumination light in the first control and the second control. Based on the first reflected light amount and the first environmental light amount, the two first captured images output from the image sensor by the second control and the third control, and the second Based on the brightness of the illumination light in the control and the third control, to calculate the second reflected light amount and the second environmental light amount, the first reflected light amount and the second reflected light amount A projection device that disables estimation of the second captured image when one of the differences and a second difference between the first ambient light amount and the second ambient light amount exceed a threshold value.
7. The projection device according to claim 6, wherein
   The projection apparatus which makes the said control of the said 3 times again which changed the brightness of the said illumination light, when the said image estimation part makes the estimation of the said 2nd captured image impossible.
8. The projection device according to any one of claims 1 to 7, wherein
   A projection apparatus having a mode of generating the information and storing the information in the storage unit.
9. Projection including an optical system for projecting an image from a display unit onto a projection target object, an imaging element for capturing a projection range of the image by the optical system through the optical system, and an illumination unit for illuminating the projection target object A method for controlling a device,
   An imaging control step of at least performing control for imaging the projection target by the image sensor in a state where the image is not projected onto the projection target and the projection target is illuminated by the illumination unit;
   Based on a first captured image output from the image sensor by the control and information stored in advance in a storage unit, the projection target is in a non-illuminated state by the illumination unit and the test image is projected. An image estimation step of estimating a second captured image obtained by imaging the projection target by the image sensor in the state of being projected on the target;
   A correction data generation step of generating correction data for correcting at least one of color and brightness of the image to be projected on the projection target based on the characteristic difference between the test image and the second captured image. And a method for controlling a projection device, comprising:
10. A method of controlling a projection device according to claim 9, wherein
    The information stored in the storage unit is the reference projection by the imaging device in a state where a predetermined reference projection target is illuminated by the illumination unit and the image is not projected on the reference projection target. A third captured image obtained by capturing an image of the object, and the reference projection object by the image sensor in a state where the illumination unit does not illuminate and the test image is projected on the reference projection object. A method for controlling a projection device, which is generated based on a fourth captured image obtained by capturing the reference projection target.
11. A method for controlling a projection device according to claim 10, wherein
    The said information is a ratio of the said 3rd picked-up image and the said 4th picked-up image, The control method of the projection apparatus.
12. A method for controlling a projection device according to any one of claims 9 to 11, comprising:
    In the imaging control step, the control is performed a plurality of times by changing the brightness of the illumination light from the illumination unit,
    The image estimation step is based on the plurality of first captured images output from the image sensor by the plurality of times of control, the brightness of the illumination light in the plurality of times of control, and the information. And a method for controlling the projection device for estimating the second captured image.
13. A method for controlling a projection device according to claim 12, wherein
    In the image estimating step, based on the plurality of first captured images and the brightness of the illumination light in the plurality of times of control, the amount of reflected light from the projection target and the placement of the projection device are set. A method of controlling a projection device, which calculates an ambient light amount of a place, adds the ambient light amount to a value obtained by correcting the reflected light amount by the information, and estimates the second captured image.
14. A method for controlling a projection device according to claim 13, wherein
    The imaging control step, as the control of the plurality of times, performs the control three times of the first control, the second control, and the third control,
    In the image estimating step, the two first captured images output from the image sensor by the first control and the second control and the brightness of the illumination light in the first control and the second control are included. Based on the first reflected light amount and the first environmental light amount, the two first captured images output from the image sensor by the second control and the third control, and the second Based on the brightness of the illumination light in the control and the third control, to calculate the second reflected light amount and the second environmental light amount, the first reflected light amount and the second reflected light amount If the difference between the first and the second difference between the first amount of ambient light and the second amount of ambient light exceeds a threshold value, the projection device that disables estimation of the second captured image Control method.
15. A method for controlling a projection device according to claim 14, wherein
    In the image estimating step, when the estimation of the second captured image is disabled, the control method of the projection device which causes the control to be repeated three times with the brightness of the illumination light changed.
16. Projection including an optical system for projecting an image from a display unit onto a projection target object, an imaging element for capturing a projection range of the image by the optical system through the optical system, and an illumination unit for illuminating the projection target object A control program for the device,
    An imaging control step of at least performing control for imaging the projection target by the image sensor in a state where the image is not projected onto the projection target and the projection target is illuminated by the illumination unit;
    Based on a first captured image output from the image sensor by the control and information stored in advance in a storage unit, the projection target is in a non-illuminated state by the illumination unit and the test image is projected. An image estimation step of estimating a second captured image obtained by imaging the projection target by the image sensor in the state of being projected on the target;
    A correction data generation step of generating correction data for correcting at least one of color and brightness of the image to be projected on the projection target based on the characteristic difference between the test image and the second captured image. And a control program of a projection device for causing a computer to execute the following.
    

Images