JP2022120607A - Projection apparatus and projection method - Google Patents

Abstract

To provide a projection apparatus and a projection method capable of more accurately expressing the color of a projection image.SOLUTION: A projection apparatus 1 comprises a projection image generation unit 33 and a projection unit. The projection image generation unit, on the basis of image data of a projection area including an object and its peripheral area photographed while first light is applied to the projection area and the peripheral area, determines the pixel color of the center of the projection area when assuming a state in which the object does not exist in the projection area in the image data, and on the basis of the pixel color of the center and the pixel color of the peripheral area, determines a pixel color other than the pixel of the center of the projection area when assuming the state in which the object does not exist in the projection area. The projection image generation unit thereby generates virtual image data being image data of the projection area when assuming the state in which the object does not exist in the projection area, and generates image data of the projection image projected on the projection area on the basis of the virtual image data. The projection unit projects the projection image on the projection area.SELECTED DRAWING: Figure 3

Description

The present invention relates to a projection apparatus and projection method.

Projection mapping is used to project images onto the walls of buildings, three-dimensional structures, and the like. For example, in projection mapping described in Patent Document 1, an image of a certain object is created and projected onto another object (target object).

JP 2017-192040 A

When a projection image is created and projected onto a target object using the above-described method, the colors of the target object may not accurately represent the colors of the projection image that should be displayed. In particular, when an image of an object having a different shape is projected onto the target object, the color of the projected image that should be displayed cannot be represented accurately, and the user may feel uncomfortable.

SUMMARY OF THE INVENTION An object of the present invention is to provide a projection apparatus and a projection method capable of expressing the colors of a projected image more accurately.

A projection apparatus according to a first aspect of the present invention provides image data of a projection area in which an object exists and the surrounding area photographed in a state in which the projection area and the surrounding area are irradiated with a first light. determining a pixel color at the center of the projection area when it is assumed that the object does not exist in the projection area in the image data, and based on the pixel color at the center and the pixel colors of the peripheral area and determining a pixel color of pixels other than the central pixel in the projection area assuming that the object does not exist in the projection area, thereby performing the above a projection image generation unit that generates virtual image data that is image data of a projection area, generates image data of a projection image that is projected onto the projection area based on the virtual image data, and projects the projection image onto the projection area. and a projection unit for projecting on.

A projection device according to a second aspect of the present invention provides image data of a projection region and a peripheral region photographed in a state in which a projection region in which an object exists and the peripheral region are irradiated with a first light. Based on this, first image data is generated as image data of the projection area assuming that the object does not exist in the projection area, and a state in which the second light is irradiated from the first image data. a projection image generation unit that generates image data obtained by subtracting second image data, which is image data of the object in the projection area photographed in the projection area, as image data of the projection image; and a projection unit for projecting on.

A projection method executed by a projection apparatus according to a third aspect of the present invention is a projection area in which an object exists and the surrounding area photographed in a state where the first light is applied to the projection area and the surrounding area. determining a pixel color at the center of the projection area when it is assumed that the object does not exist in the projection area in the image data, based on the image data of the area; The state in which the object does not exist in the projection area is determined by determining the color of pixels other than the central pixel in the projection area, assuming the state in which the object does not exist in the projection area, based on the color and the color. generating virtual image data that is image data of the projection area in the hypothetical case, generating image data of a projection image to be projected onto the projection area based on the virtual image data, and projecting the projection image onto the projection area It is projected by

A projection method executed by a projection apparatus according to a fourth aspect of the present invention is a projection area in which an object exists and the surrounding area photographed in a state where the first light is applied to the projection area and the surrounding area. generating first image data, which is image data of the projection region assuming that the object does not exist in the projection region, based on the image data of the region; image data obtained by subtracting the second image data, which is the image data of the object in the projection area photographed in the state where the is irradiated, is generated as the image data of the projection image, and It is projected by

According to the present invention, it is possible to provide a projection apparatus and a projection method capable of expressing the colors of a projected image more accurately.

1 is a schematic diagram showing the configuration of a projection system according to a first embodiment; FIG. 1 is a front view of a projection system according to Embodiment 1; FIG. 1 is a block diagram of an apparatus main body according to a first embodiment; FIG. 2 is a diagram showing a room in which projection images are projected according to the first embodiment; FIG. 1 is a side view of furniture in a room according to Embodiment 1. FIG. 4 is a flow chart showing an example of processing of the projection system according to the first embodiment; 4 is a flow chart showing an example of processing of the projection system according to the first embodiment; 4 is a flow chart showing an example of processing of the projection system according to the first embodiment; FIG. 4 is a diagram showing an image captured with 100% white light in Embodiment 1; FIG. 4 is a diagram showing an image captured with 50% gray light in Embodiment 1; 4 is a diagram showing a destination area of an object in Embodiment 1. FIG. 4 is a diagram showing a destination area of an object in Embodiment 1. FIG. 4 is a diagram showing color setting of an image projected onto an object region in Embodiment 1. FIG.

[Embodiment 1]
BEST MODE FOR CARRYING OUT THE INVENTION Hereinafter, embodiments of the present invention will be described with reference to the drawings. In Embodiment 1, an operation example of a projection system (projection device) will be described. A projection system according to the present invention is an apparatus in which a projector mechanism and a camera mechanism are integrated, and is an example of applying a projection device. Although the image projection method of the present invention is described as a method executed by a projection system, it can also be said that it is executed by a projection apparatus. Also, when the projection method of the present invention is executed as a program, the computer in the projection system executes the program, but it can also be said that the computer in the projection apparatus executes the program.

The projection system described below can be realized as a device such as a projector, a digital camera, a video camera, etc., but the applicable device is not limited to this.

FIG. 1 is a schematic diagram showing a conceptual configuration of a projection system 1. As shown in FIG. The projection system 1 includes a camera 10 as an imaging unit, a TOF (Time of Flight) sensor 11 , a projection unit 12 and an apparatus body 13 . The projection system 1 is also connected to another computer P. The computer P is, for example, a PC (Personal Computer), and includes a display unit such as a display, and input units such as buttons, a mouse, a keyboard, and a touch panel. Further, the computer P is equipped with a device driver (image processing software) for the user to input instructions to the projection system 1 . The user operates the projection system 1 via the computer P by operating the input section while viewing the display section of the computer P. FIG.

Each component of the projection system 1 will be described below. The camera 10 is attached to the projection system 1 and captures images of the peripheral area of the projection system 1 (in particular, the area projected by the projection unit 12). The camera 10 is a visible light camera and can photograph an object irradiated with light from the projection unit 12 .

The TOF sensor 11 is a 3D-TOF sensor that irradiates an object existing within a detection area with infrared light and receives reflected infrared light from the object. The detection area of the TOF sensor 11 is the same as the photographing area photographed by the camera 10 . The TOF sensor 11 measures the distance between the projection system 1 and an object existing within the detection area (imaging area) by measuring the time from the timing when the irradiation light is output until the reflected light returns to the TOF sensor 11. do. Hereinafter, the distance between the projection system 1 and an object existing within the detection area (imaging area) is simply referred to as "object distance".

The projection unit 12 functions as a projector having a light emitting light source such as an LED (Light Emitting Diode), a lens, and the like, and projects (irradiates) a projection image generated by a projection image generation unit 33, which will be described later, onto a projection area. This projection area is the same as the imaging area of the camera 10 . At this time, the color temperature and brightness of the light emitted from the projection unit 12 are controlled so as to correspond to the projected image. Specifically, the color temperature of the emitted light can be changed by adjusting the RGB parameters of the light source.

Further, the projection unit 12 irradiates the projection area with light when the camera 10 captures an image, so that the projection system 1 can obtain image data captured with desired light. Also at this time, the color temperature and brightness of the light emitted from the projection unit 12 are controlled so as to correspond to the properties of the image data to be obtained.

FIG. 2 is a front view of the projection system 1. FIG. A camera 10 in front of the projection system 1 captures image data and a TOF sensor 11 measures the distance of the object captured by the camera 10 . Then, the projection unit 12 can project the projection image onto an arbitrary area captured by the camera 10 .

FIG. 3 is a block diagram of the device body 13. As shown in FIG. The apparatus main body 13 is a part that constitutes the computer of the projection system 1 and has a memory 20 , an I/O section 21 and an information processing section 30 .

Memory 20 is comprised of volatile memory, non-volatile memory, or a combination thereof. The number of memories 20 is not limited to one, and a plurality of memories may be provided. The volatile memory may be, for example, RAM (Random Access Memory) such as DRAM (Dynamic Random Access Memory) and SRAM (Static Random Access Memory). The non-volatile memory may be, for example, PROM (Programmable ROM), EPROM (Erasable Programmable Read Only Memory), Flash Memory.

Memory 20 is used to store one or more instructions. Here, one or more instructions are stored in memory 20 as software modules. The information processing unit 30 can perform the following processes by reading one or more instructions from the memory 20 and executing them. Further, the memory 20 stores image data obtained by photographing and image data of a projection image obtained by processing described later.

An I/O (Input/Output) unit 21 is a hardware interface that inputs and outputs information to and from an external device of the projection system 1 . In this embodiment, the I/O section 21 is connected to the computer P and used for information transmission/reception between the information processing section 30 and the computer P. FIG.

The information processing unit 30 is composed of arbitrary processors such as a CPU (Central Processing Unit), MPU (Micro Processing Unit), FPGA (Field-Programmable Gate Array), DSP (Digital Signal Processor), and ASIC (Application Specific Integrated Circuit). be done.

Note that the memory 20 may be provided outside the information processing section 30 and may include a memory built into the information processing section 30 . Moreover, the memory 20 may include a storage that is located away from the processors that make up the information processing section 30 . In this case, the processor can access the memory 20 via an I/O (Input/Output) 21 .

The information processing unit 30 reads out software (computer program) from the memory 20 and executes the software (computer program) to realize the functions of the shooting control unit 31, the distance specifying unit 32, the projection image generation unit 33, the projection control unit 34, and the like. Each component will be described below.

The shooting control unit 31 controls shooting of the camera 10 . Specifically, the photographing control unit 31 has a function of controlling execution of photographing operations and adjusting white balance, exposure, contrast, etc. during photographing. Further, the imaging control unit 31 controls the projection unit 12 at the time of imaging so that light having a desired color temperature and luminance is applied to the imaging target. For example, the imaging control unit 31 adjusts the RGB parameters of the light source of the projection unit 12 .

Further, the imaging control unit 31 acquires image data shot by the camera 10 and stores it in the memory 20 . Further, the photographing control unit 31 controls the TOF sensor 11 at the time of photographing to measure the distance of an object existing within the detection area (imaging area).

The distance specifying unit 32 compares the distance of the object within the detection area detected by the TOF sensor 11 with the image data captured by the camera 10 to specify the distance of the object in the image data. The distance specifying unit 32 specifies an object having a distance difference of a predetermined threshold or more from surrounding objects in the image data, and causes the display unit of the computer P to display information on the specified object together with the image data.

The projection image generator 33 generates a projection image to be projected onto the projection area of the projection unit 12 . For example, the projection image generation unit 33 can process a plurality of image data captured by the camera 10 to generate image data of a projection image. The details of this processing will be described later.

The projection control unit 34 controls projection by the projection unit 12 . Specifically, the projection control unit 34 adjusts the color temperature and brightness of the light source of the projection unit 12 and the aperture of the projection unit 12 so that the projection image generated by the projection image generation unit 33 is displayed on the projection unit 12. . Further, the projection control unit 34 may control the projection unit 12 to project an image pre-stored in the memory 20 instead of the projection image generated by the projection image generation unit 33 .

FIG. 4A shows an object on which the projection system 1 captures image data and uses the image data to project a projection image. FIG. 4A shows that in a room R of a house, a television T, a television stand S, and a bookshelf B, which are pieces of furniture (three-dimensional objects), are arranged with a wall W at the back. The room R is included in all of the imaging area of the camera 10, the detection area of the TOF sensor 11, and the projection area of the projection unit 12. FIG.

4B is a side view of room R and projection system 1. FIG. The front of the projection system 1 faces the wall W and each piece of furniture so that the wall W and each piece of furniture can be photographed and an image can be projected. The distance between the projection system 1 and the wall W is d1, the distance between the projection system 1 and the bookshelf B is d2, the distance between the projection system 1 and the television T is d3, and the distance between the projection system 1 and the television stand S is d4. , d1>d2>d3>d4.

In this example, the user of the projection system 1, who is a resident of the room, plans to change the layout by moving the television T and the television stand S to the position of the bookshelf B. FIG. Here, the user uses the projection system 1 to project the images of the television T and the television stand S onto the bookshelf B without moving the television T and the television stand S. By simulating the redecoration, the user can confirm in advance whether the redecoration intended by the user is satisfactory.

5A-5C are flow charts showing an example of the processing performed by the projection system 1 in the situation described above. The processing of the projection system 1 will now be described with reference to FIGS. 5A-5C.

First, the user operates the computer P to irradiate the room R with 100% white light (first light) and project an instruction through the I/O unit 21 to capture an image for white balance. Output to system 1. In response to the instruction, the photographing control unit 31 controls the camera 10 to photograph the room R while controlling the projection unit 12 to irradiate 100% white light (step S11). At this time, a white sheet of paper prepared in advance by the user for adjusting the white balance is arranged in the room R, and the camera 10 photographs the white sheet together with the interior of the room R. Note that 100% white light means that each of the RGB components is 100%.

The photography control unit 31 uses the image data obtained by photography to adjust the white balance at the time of photography so that the white paper in the image data appears white. A mode in which the white balance of the camera is adjusted with 100% white light emitted from the projection system 1 in this manner is referred to as a first mode.

Next, the user puts away the white paper outside the imaging area, and then instructs the projection system 1 to generate a projection image by operating the computer P. FIG. In response to the instruction from the I/O unit 21, the photographing control unit 31 controls the camera 10 to photograph the room R while controlling the projection unit 12 to irradiate 100% white light. (Step S12). By this photographing, the wall W, the television T, the television stand S, and the bookshelf B are photographed as still images. Note that the white balance set in step S11 is used as the white balance at the time of photographing in step S12. This photographing mode is referred to as a second mode.

FIG. 6A shows an image taken in the second mode. This photographed image is hereinafter referred to as PIC1. The imaging control unit 31 stores PIC1 in the memory 20 as color image data having RGB signals.

Further, in step S12, the photographing control unit 31 controls the TOF sensor 11 during photographing by the camera 10 to measure the distance of the object within the detection area. Further, the distance specifying unit 32 compares the distance of the object in the detection area detected by the TOF sensor 11 with the PIC1 photographed by the camera 10, and determines the wall W, the bookshelf B, the television T, and the television stand S in the PIC1. specify the distances d1 to d4, respectively.

After step S12, the photographing control unit 31 controls the camera 10 to photograph the room R while controlling the projection unit 12 to emit 50% gray light (second light) (step S13). Note that 50% gray light means that the RGB components are each 50%. By this photographing, the wall W, the television T, the television stand S, and the bookshelf B are photographed as still images. Note that the white balance set in step S11 is used as the white balance at the time of photographing in step S13. This photographing mode is referred to as a third mode.

FIG. 6B shows an image taken in the third mode. This photographed image serves as reference data when generating a projection image, and is hereinafter referred to as PIC2. The imaging control unit 31 stores the PIC2 in the memory 20 as color image data having RGB signals.

It should be noted that at the time of photographing in steps S11 to S13 shown above, the photographing control unit 31 sets the exposure of the camera 10 to an appropriate state according to the properties of the image data. Further, the irradiation light in each step is uniform light with almost no unevenness. Furthermore, in the photographing in steps S11 to S13, it is preferable to keep the room R as a dark room so that light other than the irradiation light does not enter the room R.

Next, the projection image generation section 33 outputs the image of the PIC1 stored in the memory 20 to the computer P via the I/O section 21 . The computer P operates the device driver to display the image of PIC1 on the display unit. The user operates the input unit of the computer P to select the television T and the television stand S as objects (subjects) to be moved from the image of the PIC1. The selected image data of the television T and the television stand S is a two-dimensional array data string whose position is represented by the x and y axes, and is a data string having two-dimensional data for each of the RGB color components. This data string is expressed as RealDataObj1[x, y]. The computer P outputs the information of RealDataObj1[x, y] to the projection image generation section 33 via the I/O section 21 according to the operation of the input section of the computer P. FIG.

Note that the projection image generation unit 33, based on information on the distances of the wall W, the television T, the television stand S, and the bookshelf B in the PIC1 specified by the distance specifying unit 32, , may be identified as object candidate regions. In this case, since there is a difference in the distance of each object (distances are different), the projected image generation unit 33 can recognize the wall W, the television T, the television stand S, and the bookshelf B as candidate regions for different objects. .

The projection image generation unit 33 outputs information indicating the identified object candidate regions to the computer P together with the image of the PIC1. The projection image generation unit 33 may set the outer frame of the recognized object as information indicating the candidate area of the object by image processing, for example, and superimpose the information of the outer frame on the image of the PIC1. The user operates the input unit of the computer P to select the television T and the television stand S as objects (subjects) to be moved from the object candidate areas. The computer P outputs the information of the selected RealDataObj1[x, y] to the projection image generator 33 .

Alternatively, the projection image generation unit 33 may output to the computer P information on the distance of each object identified by the distance identification unit 32 together with the image of the PIC1. The computer P displays distance information of each object on the display together with the image of the PIC1. By referring to this distance information, the user selects the television T and the television stand S as objects to be moved. By executing the above-described processing, the projection image generation unit 33 can easily select an object scheduled to move from the PIC1 accurately.

Next, based on the information of RealDataObj1[x, y], the projected image generation unit 33 extracts GrayDataObj2[x, which is the image data of the area of the television T and the television stand S, from the image data of the PIC2 stored in the memory 20. , y] (second image data). Normally, it is difficult to imagine that the position of an object in the room R will change during the imaging of PIC1 and PIC2. Therefore, in PIC2, the projection image generation unit 33 selects the image data of the area having the same position information as the two-dimensional position information of RealDataObj1[x,y] as GrayDataObj2[x,y]. That is, RealDataObj1[x, y] and GrayDataObj2[x, y] are information indicating the areas of the same television T and television stand S in PIC1 and PIC2.

However, the projection image generator 33 may output the image of PIC2 to the computer P and allow the user to select GrayDataObj2[x, y]. The projection image generation unit 33 acquires information of GrayDataObj2[x, y] selected by the user from the computer P via the I/O unit 21 . The details of this process are the same as the process related to the image of the PIC1 described above, so description thereof will be omitted.

As described above, the projected image generation unit 33 selects RealDataObj1[x, y] (image data 1), which is the image data of the television T and the television stand S, from the image data of the PIC1 based on the user's operation. do. Similarly, the projection image generator 33 selects GrayDataObj2[x, y] (image data 2), which is the image data of the television T and the television stand S, from the image data of the PIC2 (step S14).

Below, the projection image generation unit 33 generates (A) a projection image projected at the planned movement position of the television T and the television stand S, and (B) a projection image projected onto the area where the television T and the television stand S are currently located. respectively. Then, the projection control unit 34 controls the projection unit 12 to project the projection images generated in (A) and (B).

Here, the process of generating the projection image related to (A) described above is described as process A, and the process of generating the projection image related to (B) is described as process B. Processing A and B will be described with reference to FIGS. 5B and 5C, respectively.

<Process A>
First, processing A will be described with reference to FIG. 5B. In step S14, when PIC1 is displayed on the display section of the computer P, the user uses the input section to specify the destination of the selected television T and television stand S. FIG. For example, the user uses the mouse of the computer P to drag and drop the selected TV T and TV stand S on the display unit, thereby moving the area of the PIC1 to the TV T and TV stand S. You can specify the destination.

FIG. 6C shows a destination area specified by dragging and dropping the television T and the television stand S on PIC1. Regions T1 and S1 indicated by dashed lines in FIG. 6C are destinations of the television T and television stand S designated by the user, and partially overlap with the bookshelf B. FIG. The computer P outputs the positional information of the designated areas T1 and S1 of the PIC1 to the projection image generator 33 . The projected image generation unit 33 designates the destination area of the television T and the television stand S in PIC2 based on the position information of the areas T1 and S1 in PIC1.

FIG. 6D shows destination areas T2 and S2 in PIC2. As described above, it is unlikely that the position of the object in the room R will change during the imaging of PIC1 and PIC2. Therefore, the projection image generator 33 selects the image data of the regions T2 and S2 having the same position information as the position information of the regions T1 and S1 of PIC1 (step S21). The selected image data is expressed as GrayDataObj3[x,y] (image data 3). GrayDataObj3[x, y] is a data string of a two-dimensional array whose positions are represented by the x and y axes, and is a data string having two-dimensional data for each of RGB color components.

Returning to FIG. 5B, the description continues. Next, the projection image generation unit 33 uses RealDataObj1 [x, y], GrayDataObj2 [x, y], and GrayDataObj3 [x, y] to generate a projection image of the destination of the television T and the television stand S ( step S22). Specifically, the projected image generator 33 performs the following calculations.

First, the projection image generation unit 33 subtracts 50% gray light data used for photographing in the third mode from GrayDataObj3[x, y]. Note that "subtraction" or "addition" in two image data means subtracting or adding the color of a pixel at the same position of another image data to the color of a pixel of one image data. It means to execute against As described above, the 50% gray light data is data in which the RGB components are R=50%, G=50%, and B=50%. Denoting the 50% gray light data as Gray50, the result of the subtraction is:
GrayDataObj3[x,y]-Gray50 (1)
(1) is difference data indicating colors that did not become gray in the destination area even when 50% gray light was irradiated. In other words, (1) indicates a color component other than gray in the destination area. (1) is difference data caused by the influence of the color of the bookshelf B and the wall W. FIG.

As a result of this calculation, in order to adjust the color of the projected image so that the color of the destination area is 50% gray, the following image data of the projected image is used as the complementary color for the destination area. It turns out that it should be projected to .
-1*(GrayDataObj3[x,y]-Gray50) (2)

Further, the projection image generation unit 33 subtracts 50% of the gray color light data from GrayDataObj2[x, y]. The result of this subtraction is:
GrayDataObj2[x,y]-Gray50 (3)
(3) is difference data indicating colors that did not become gray in the area of the television T and the television stand S (the movement source area) even when 50% of the gray light was irradiated. In other words, (3) indicates the color components other than gray in the movement source area. (3) is difference data caused by the influence of the colors of the television T and the television stand S. FIG.

As a result of this calculation, in order to adjust the color of the projected image so that the color of the source area is 50% gray, the following image data of the projected image is used as a complementary color for the source area. It turns out that it should be projected to .
−1*(GrayDataObj2[x,y]-Gray50) (4)
Normally, (2) and (4) are complementary color components to 50% gray light, and as shown below, they are factors with close values, but not the same values.
-1*(GrayDataObj2[x,y]-Gray50)
≈-1*(GrayDataObj3[x,y]-Gray50) (5)

The projection image generation unit 33 generates image data, which is a projection image onto the destination area, using the above calculation results. That is, the projection image generation unit 33 subtracts the complementary color data (4) in the movement source area from the image data of RealDataObj1[x, y] having the color information of the actual television T and the television stand S, By adding the complementary color data (3) in the area of , the image data of the projection image is generated. Assuming that the image data of the projection image is MoveDataObj[x, y], MoveDataObj[x, y] is as follows.
MoveDataObj[x,y]
=RealDataObj1[x,y]-(-1*(GrayDataObj2[x,y]-Gray50))+(-1*(GrayDataObj3[x,y]-Gray50))
=RealDataObj1[x,y]+GrayDataObj2[x,y]-GrayDataObj3[x,y] (6)

Based on the matter shown in (5), MoveDataObj[x, y] calculated in (6) has a value close to RealDataObj1[x, y], but they are not the same value. MoveDataObj[x,y] is image data of a projected image in which the color of the object (subject) at the destination is suppressed and the color of the television T and the television stand S is more reflected. The projection control unit 34 controls the projection unit 12 to project the projection image related to this MoveDataObj[x, y] onto the area near the bookshelf B, which is the movement destination area (step S23).

As a method of generating and projecting a projection image, in step S22, RealDataObj1[x, y] (image data of the television T and the television stand S in PIC1) is projected as it is by the projection unit 12 onto the regions T2 and S2. It is also possible. However, in this method, the color of the projected image data is affected by the color of the bookshelf B that originally existed in the room R and the color of the wall W nearby. color may not be reproduced correctly. Therefore, in the present invention, the projection image generation unit 33 corrects the color of the bookshelf B at the movement destination and the color of the wall W in the vicinity thereof so as to approach gray by performing the process of step S22.

In particular, the projection image generation unit 33 generates the image data of the projection image shown in (6), thereby suppressing the influence of the color of the bookshelf B and the color of the wall W near it on the color of the projection image. At the same time, the original colors of the television T and the television stand S are more reflected. Therefore, the projection system 1 can more accurately express the colors of the projected image.

Further, the projection unit 12 of the projection system 1 irradiates the television T and the television stand S with 100% white light (first light) when capturing RealDataObj1[x,y]. On the other hand, when capturing GrayDataObj2[x, y] and GrayDataObj3[x, y], the projection unit 12 projects 50% gray light (second light) onto the areas of the television T, the television stand S, and the projection destination. Irradiate. In other words, since the projection system 1 generates the projection image using the light from the projection unit 12 that irradiates the projection image, the projection image is more difficult to reproduce than when photographing is performed using light from another light source. Color accuracy can be improved.

Also, the projection system 1 irradiates the 100% white light that was used when capturing the RealDataObj1[x,y] with the white balance used when capturing the RealDataObj1[x,y] when capturing the RealDataObj1[x,y]. The projection system 1 captures the room R including the television T and the television stand S with light whose white balance has been adjusted, so that RealDataObj1[x, y] accurately captures the colors of the television T and the TV stand S. can be reflected. Therefore, it is possible to further improve the color accuracy of the projected image.

Also, in general, the spectral sensitivities of the camera 10 and the projection unit 12 are different. Therefore, if there is no presetting, the white balance set by the camera 10 may not correspond to the color temperature set by the projection unit 12 . Therefore, in the process described above, the projection system 1 uses 100% white light emitted by the projection unit 12 in the first mode to set the white balance of the camera. Therefore, the projection system 1 makes the white balance set by the camera 10 correspond to the color temperature set by the projection unit 12, so that the colors captured by the camera 10 are more accurately projected by the projection unit 12. be able to.

Also, the projection system 1 emits 50% gray light as reference light when capturing GrayDataObj2[x, y] and GrayDataObj3[x, y]. A 50% gray light is an achromatic color containing 50% each of three RGB components and has no saturation. Therefore, GrayDataObj2[x, y] and GrayDataObj3[x, y] tend to reflect the color characteristics of the TV T and the TV stand S, and the color characteristics of the shelf B and the wall W. Therefore, it is easy to suppress the influence of the color of the bookshelf B and the color of the nearby wall W on the color of the projected image, and the original colors of the television T and the television stand S are more likely to be reflected.

<Process B>
Next, processing B will be described with reference to FIG. 5C. As described above, in step S14, when PIC1 is displayed on the display section of the computer P, the user uses the input section to designate the television T and the television stand S as objects to be moved. In this case, the projection system 1 projects a dedicated projection image onto the area where the television T and the television stand S actually exist, thereby making the television T and the television stand S less visible (inconspicuous) to the user. Thereby, the projection system 1 displays that the television T and the television stand S have virtually moved. By generating a dedicated projection image based on the surrounding colors of the television T and the television stand S, it is possible to provide a display that does not give the user a sense of discomfort.

First, the user uses the input section of the computer P to specify the center point of the area of the television T and the television stand S (moving object) based on the display of PIC1 on the display section of the computer P. FIG. For example, the user can operate the mouse to specify the center point of the area with the pointer. The computer P transmits the information of the central point of the designated area to the projected image generator 33 of the projection system 1 .

However, the projection image generator 33 may specify the center point of the area of the television T and the television stand S based on the information of RealDataObj1[x, y] output from the computer P. FIG. For example, the projection image generation unit 33 obtains the center point (geometric center or center of gravity) of the area of the television T and the center point of the area of the television stand S, and divides the line connecting the two center points equally. and the center point of the area of the TV stand S may be set. Note that when the user selects one object, the projection image generation unit 33 can designate the geometric center or the center of gravity of the area as the center point. Furthermore, when the user selects three or more objects, the center of gravity of the center point obtained in each area can be specified as the center point of the area related to the selected object.

Next, the projected image generator 33 determines the color of the central point of the areas of the television T and the television stand S (step S31). Details of this will be described with reference to FIG. 6E.

FIG. 6E shows an excerpt of the areas of the television T and the television stand S in FIG. 6C. The vertical direction of each object is represented by the y-axis direction and the −y-axis direction in FIG. 6E, and the horizontal direction is represented by the x-axis direction and the −x-axis direction in FIG. 6E. The projection image generation unit 33 generates pixels in the y-axis direction and -y-axis direction and pixels in the x-axis direction and -x-axis direction when viewed from the pixel at the center point C of the area of the television T and the television stand S. Among them, the pixels that are one pixel outside the boundary line between the television T and the television stand S are specified. FIG. 6E shows these pixels as points Pu, Pd, Pr and Pl, respectively. In other words, these four points Pu to Pl are areas located just outside the areas of the television T and the television stand S. FIG. The projection image generation unit 33 specifies each color represented by RGB of these four pixel points Pu to Pl, and calculates the average color ObjRoundAve of the four points. The projection image generator 33 determines this ObjRoundAve as the color of the center point C (pixel color).

Next, the projection image generator 33 determines the color of pixels other than the center point C in the areas of the television T and the television stand S (step S32). Details of this will be described again with reference to FIG. 6E.

Point In in FIG. 6E is an example of a pixel located within the area of television T and television stand S. FIG. When calculating the color of the point In, the projected image generation unit 33 first assumes a line L connecting the point In and the center point C, and on the extension of the line L, the television T and the television stand S Select one pixel that is one pixel outside the border. Note that the pixels that are on the extension line of line L and are one pixel outside the boundary line between the television T and the television stand S are the pixels closer to the point In (point Pn) and the pixels farther from the point In (point Pf) are present. In the case of FIG. 6E, the projection image generation unit 33 selects a pixel (point Pn) that is one pixel outside the boundary line that exists on the extension line of the line L in the direction from the center point C toward the point In. Selecting pixels in this way is preferable in that natural colors can be determined according to the background of the television T and the television stand S as the projected image.

The projection image generation unit 33 selects a point Pn as a pixel located on the extension of the line L and outside the boundary line between the television T and the television stand S by one pixel in the process described above. Then, the projection image generation unit 33 calculates the color of the point In based on the colors of the center point C and the point Pn, the distance from the center point C to the point In, and the distance from the point In to the point Pn. . For example, if the color of point Pn is expressed as ColorPn, and the ratio of the distance from center point C to point In to the distance from point In to point Pn is X:Y, let the color of point In be ColorIn: The projection image generator 33 can obtain ColorIn as follows.
ColorIn=(ObjRoundAve*Y+ColorPn*X)/(X+Y) (7)
ColorIn is the color displayed by each value of RGB. The linear interpolation method described above is an example, and the projection image generation unit 33 may calculate ColorIn based on another method.

The projection image generation unit 33 applies the same method as that for calculating ColorIn to all pixels other than the center point C within the areas of the television T and the television stand S to calculate the color of each pixel. The calculated color data of each pixel (RGB data in two-dimensional array) is hereinafter referred to as ObjRoundPaint[x,y] (first image data). This ObjRoundPaint[x, y] is color data (virtual image data) for bringing the color of the area of the television T and the television stand S closer to the color of the surrounding area.

Next, the projection image generation unit 33 generates GrayDataObj2[x,y] (image data 2) obtained by photographing the television T and the television stand S to be moved, selected in step S14, with 50% gray light. are used to generate projection image data for the television T and the television stand S (step S33). The details are as follows.

First, the projected image generation unit 33 subtracts 50% of the gray color light data from ObjRoundPaint[x,y], and obtains ObjRoundPaintDiff[x,y], which is the color that is the difference from the reference color of ObjRoundPaint[x,y]. ] is obtained as follows.
ObjRoundPaintDiff[x,y]=ObjRoundPaint[x,y]-Gray50 (8)

Further, the projection image generation unit 33 subtracts 50% of the gray light data from GrayDataObj2[x,y], and obtains GrayDataObj2Diff[x,y], which is the color that is the difference from the reference color of ObjRoundPaint[x,y]. ] is obtained as follows.
GrayDataObj2Diff[x,y]=GrayDataObj2[x,y]-Gray50 (9)

By multiplying this GrayDataObj2Diff[x, y] by (-1), a correction value for changing the color of the area of the television T and the television stand S to 50% gray is obtained. The projection image generation unit 33 calculates ObjRoundPaintCari[x, y] by adding this correction value to ObjRoundPaintDiff[x, y] as follows.
ObjRoundPaintCar[x, y]
=ObjRoundPaint[x,y]-Gray50-(GrayDataObj2[x,y]-Gray50)
=ObjRoundPaint[x,y]-GrayDataObj2[x,y] (10)

ObjRoundPaintCari[x, y] is image data of a projection image for suppressing the influence of the color of an originally existing object (subject). The projection control unit 34 controls the projection unit 12 to project the projection image related to this ObjRoundPaintCari[x, y] onto the areas of the television T and the television stand S (step S34).

Note that the projection system 1 performs the projection processing in step S23 and the projection processing in step S35 at the same time, so that a virtual state in which the television T and the television stand S have moved to the vicinity of the bookshelf B in the room R can be displayed by the user. can be visually recognized.

As a method of projecting and generating a projection image, it is conceivable that ObjRoundPaint[x, y] calculated in step S32 is projected onto the regions T and S as it is by the projection unit 12 . However, in this method, although the color of the area of the television T and the television stand S can be approximated to the color of the surrounding area, the original color of the television T and the television stand S appears as the color of the area. Therefore, there is a limit to the effect of assimilating the color of the area of the television T and the television stand S with the color of the surrounding area.

Therefore, in the present invention, the projection image generation unit 33 projects the television T and the television stand S (projection area) based on the PIC1 photographed with 100% white light (first light). ObjRoundPaint[x, y] (first image data) is generated assuming that T and TV stand S do not exist. Then, each pixel color of GrayDataObj2[x, y] (second image data), which is the image data of the television T and the television stand S photographed with 50% gray light, is converted to ObjRoundPaint[x, Image data ObjRoundPaintCari[x, y] is calculated by subtracting each pixel color from [x, y]. The projection system 1 projects this ObjRoundPaintCari[x, y] onto the area of the television T and the television stand S, so that the color of the area of the television T and the television stand S is assimilated with the colors around the area. Thereby, it is possible to more naturally express that the television T and the television stand S have virtually moved.

In addition, the projection image generation unit 33 calculates the projection area of PIC1 based on PIC1 photographed in a state where the projection area where the television T and the television stand S are present and the surrounding area thereof is illuminated with 100% white light. Determine the color (pixel color) of the center point C of the projection area assuming that the television T and the television stand S do not exist in . Then, based on the color of the center point C and the pixel color of the surrounding area of the television T and the television stand S in PIC1, the center of the projection area assuming that the television T and the television stand S do not exist in the projection area Determine pixel colors other than point C. Thereby, the projection system 1 can naturally express the color of the area of the television T and the television stand S using the surrounding colors.

By performing the processing shown in FIGS. 5A to 5C as described above, when rearranging a room, the user can project an image of the furniture using the projection system 1 before actually moving the furniture. can be simulated. In particular, according to the present invention, by correcting the color of the projected image, it is possible to reduce the discomfort felt by the user even when projecting a projected image of an object having a different shape from the original object.

It should be noted that the present invention is not limited to the above embodiments, and can be modified as appropriate without departing from the scope of the invention. For example, the projection system 1 may use light that is not 100% white light for illumination in the first and second modes. For example, when the projection system 1 treats the HDR (High Dynamic Range) signal standard HLG (Hybrid Log Gamma) as a reference for the illumination light, 75% white light is used for illumination in the first mode and the second mode. You can use it. This 75% white light is the same color as the 100% white light described above. In this case, the headroom of the signal level is increased, and the clipping of the upper limit of the signal level calculated in (6) and (10) above is alleviated. In the first embodiment, the calculation result exceeds 100%, and the signal level of the light actually irradiated by the projection unit 12 becomes 100% (clip occurs, that is, the calculation result cannot be fully reflected). ), the projection system 1 can emit a projection image that directly reflects the calculation result.

The projection system 1 may also use non-50% gray light for illumination in the third mode. For example, the projection system 1 may select each component of RGB included in the gray light within a predetermined range (eg, 30% to 70%). When the projection system 1 shoots the room R with gray light having certain RGB components (%) and displays the projection image obtained by calculating the calculation results of (6) and (10) with the gray light. , the user may not be satisfied with the quality of the projected image. Such a case may occur depending on the color of the furniture in the room R. In such a case, it is possible to change the gray light, re-capture the PIC 2, and cause the projection system 1 to calculate the calculation results of (6) and (10) again and project a projection image. Become.

The projection control unit 34 controls the projection unit 12 to project projected images other than the projected images generated in the processes A and B to areas other than the area of the television T, the television stand S, and the movement destination areas of these objects. can be projected onto

The TOF sensor 11 in the projection system 1 may measure the distance to an object using radio waves, ultrasonic waves, etc. instead of infrared light. Also, the projection system 1 may measure the distance to the object using other mechanisms such as triangulation instead of TOF.

Instead of the computer P, the projection system 1 may be capable of processing for selecting an object that the user wants to move. In this case, the projection system 1 includes a display unit such as a display and an input unit such as buttons and a touch panel. For example, the television T and the television stand S can be specified by the user confirming the captured image on the display section and inputting the input section.

In Embodiment 1, the virtual movement of room furniture such as the television T and the television stand S has been exemplified and explained, but it is needless to say that the applicable example of projection mapping is not limited to this.

As described above, the one or more processors included in the projection apparatus of the above-described embodiments execute one or more programs containing instructions for causing a computer to execute the algorithms described with reference to the drawings. . By this processing, the processing described in the embodiment can be realized.

The program can be stored and delivered to the computer using various types of non-transitory computer readable media. Non-transitory computer-readable media include various types of tangible storage media. Examples of non-transitory computer readable media include magnetic recording media (e.g. hard disk drives), CD-ROMs (Read Only Memory), CD-Rs, CD-R/Ws, semiconductor memories (e.g. mask ROMs, PROMs, EPROMs). , flash ROM, RAM). The program may also be delivered to the computer on various types of transitory computer readable medium. Examples of transitory computer-readable media include electrical signals, optical signals, and electromagnetic waves. Transitory computer-readable media can deliver the program to the computer via wired channels, such as wires and optical fibers, or wireless channels.

Although the present invention has been described with reference to the embodiments, the present invention is not limited to the above. The configuration of the present invention includes substantially the same configuration that can be assumed by those skilled in the art, and various changes that can be understood can be made.

1 Projection System 10 Camera 11 TOF Sensor 12 Projection Unit 13 Apparatus Main Body 20 Memory 21 I/O Unit 30 Information Processing Unit 31 Shooting Control Unit 32 Distance Identification Unit 33 Projection Image Generation Unit 34 Projection Control Unit

Claims (5)

Based on the image data of the projection area in which the object exists and the surrounding area photographed in a state where the first light is applied to the projection area and the surrounding area, the object is projected onto the projection area in the image data. determining a pixel color at the center of the projection area assuming that no object exists in the projection area, and assuming that the object does not exist in the projection area based on the center pixel color and the pixel color of the peripheral area. Virtual image data, which is image data of the projection area assuming that the object does not exist in the projection area, is generated by determining the color of pixels other than the center pixel of the projection area in the case of a projection image generation unit that generates image data of a projection image projected onto the projection area based on the virtual image data;
a projection unit that projects the projection image onto the projection area;
A projection device comprising: The projection image generation unit subtracts each pixel color of second image data, which is image data of the projection area photographed under irradiation with the second light, from each pixel color of the virtual image data. generating image data for the projection image by
A projection device according to claim 1 . A state in which the object does not exist in the projection area based on image data of the projection area in which the object exists and the surrounding area and the image data of the projection area and the surrounding area photographed in a state in which the first light is applied to the projection area and the surrounding area. generating first image data, which is image data of the projection area assuming the above, and from the first image data, image data of the object in the projection area photographed under irradiation with the second light a projection image generation unit that generates image data obtained by subtracting the second image data as image data of a projection image;
a projection unit that projects the projection image onto the projection area;
A projection device comprising: Based on the image data of the projection area in which the object exists and the surrounding area photographed in a state where the first light is applied to the projection area and the surrounding area, the object is projected onto the projection area in the image data. determine the pixel color at the center of the projection area assuming the absence of
Determining a pixel color other than the central pixel in the projection area assuming that the object does not exist in the projection area, based on the central pixel color and the peripheral area pixel color; generating virtual image data that is image data of the projection area assuming that the object does not exist in the projection area;
generating image data of a projection image to be projected onto the projection area based on the virtual image data;
projecting the projection image onto the projection area;
A projection method performed by a projection device. A state in which the object does not exist in the projection area based on image data of the projection area in which the object exists and the surrounding area and the image data of the projection area and the surrounding area photographed in a state in which the first light is applied to the projection area and the surrounding area. generating first image data that is image data of the projection area assuming
Image data obtained by subtracting, from the first image data, second image data, which is image data of the object in the projection area photographed under irradiation with the second light, is used as image data of the projected image. generate and
projecting the projection image onto the projection area;
A projection method performed by a projection device.

Images