JP2016128870A - Image processing device, image processing system, and lighting device - Google Patents

Abstract

PROBLEM TO BE SOLVED: To make it possible to easily perform an operation for designating a region.SOLUTION: An image processing system comprises: a lighting device; an imaging device; and an image processing device, the lighting device emits invisible light, the imaging device is provided to set the invisible light as an object to be imaged and to image a screen onto which the invisible light emitted from the lighting device is projected, and the image processing device detects a mask region which corresponds to the invisible light which has been projected onto the screen in a picked-up image obtained by the imaging device, synthesizes a plurality of images on the basis of the detected mask region, and outputs a synthesized image in such a manner that the synthesized image is projected, as image light, onto the screen by a projector.SELECTED DRAWING: Figure 8

Description

  The present invention relates to an image processing device, an image processing system, and an illumination device.

It is known to use a pointing device such as a mouse or a motion controller that can be operated in space when inputting coordinates according to the position on the display screen.
Also, as an example, there is known a presentation control system that captures an image on a screen irradiated with light from a laser pointer, which is a pointing device, and inputs the position of the laser pointer light in the captured image as coordinates.

Japanese Patent Laid-Open No. 2001-125738

As an operation for the computer, there is a case where it is desired to specify a certain area instead of specifying a point as coordinates. In such a case, when an area is to be designated using a device for inputting coordinates as described above, an operation for forming an area by making coordinates continuous as points is required.
Specifically, in the case of a mouse, an operation of drawing a coordinate movement locus by a drag operation and forming an outline of the region is required. For example, when only an area needs to be designated as an operation for the computer, the operation for forming the area by making the coordinates continuous as described above is troublesome and may not be rational.

  The present invention has been made in view of such circumstances, and it is an object of the present invention to easily perform an operation of designating an area.

  In one embodiment of the present invention for solving the above-described problem, a mask region corresponding to invisible light projected on the screen in a captured image obtained by an imaging device that captures the screen with invisible light is captured. An image processing apparatus includes: a mask area detecting unit for detecting; and an image combining unit for combining a plurality of images based on the mask area detected by the mask area detecting unit.

  One aspect of the present invention is the above-described image processing device, wherein the image composition unit synthesizes an image of a region corresponding to the mask region in a sub image with a region corresponding to the mask region in a main image. May be.

  One aspect of the present invention is the above-described image processing device, wherein the image composition unit is configured to display barycentric coordinates of an area corresponding to the mask area in the main image when the size of the sub-image is different from that of the main image. And a region corresponding to the mask region in the sub-image based on the ratio of the size of the sub-image and the main image.

  One aspect of the present invention is the above-described image processing device, wherein the mask area detection unit may detect, as the mask area, an area in which pixels having a pixel value equal to or greater than a first threshold value in the captured image.

  One aspect of the present invention is the above-described image processing device, wherein the mask region detection unit includes pixels in a range of the pixel values included in the mask region that are in the range from the first threshold value to the second threshold value. The value is converted from zero corresponding to the pixel value to an output pixel value less than the maximum value, and the pixel value greater than the second threshold is converted to the maximum output pixel value. Pixels included in a region corresponding to the mask region in the image may be combined with the pixels of the main image based on the output pixel value.

  One aspect of the present invention is the above-described image processing device, wherein the mask area detection unit is common among a plurality of captured images obtained by a plurality of imaging devices that capture the screen from different predetermined positions. Then, an area where the invisible light that exists is projected may be detected as the mask area.

  One embodiment of the present invention is the above-described image processing device, wherein the imaging device performs imaging for each invisible light in a different wavelength region, thereby a plurality of wavelength regions corresponding to the invisible light in the different wavelength region. A corresponding captured image is acquired, the mask region detection unit detects a mask region for each of the plurality of wavelength region-capable captured images, and the image composition unit detects the mask detected for each of the plurality of wavelength region-captured images. Based on the region, a plurality of images may be individually synthesized for each of the plurality of captured images corresponding to the wavelength regions.

  One embodiment of the present invention is an image processing system including an illumination device, an imaging device, and an image processing device, wherein the illumination device includes an invisible light emitting unit that emits invisible light, and the imaging device includes: The invisible light is set as an imaging target, and is provided so as to capture a screen on which the invisible light emitted from the illumination device is projected. The image processing device is projected on the screen in a captured image obtained by the imaging device. A mask region detecting unit that detects a mask region corresponding to the invisible light, and an image combining unit that combines a plurality of images based on the mask region detected by the mask region detecting unit. And an output unit that outputs the synthesized image so that the projected image is projected as image light on the screen by the projection device. It is a non.

  One aspect of the present invention is the above-described image processing system including a plurality of the imaging devices that capture the screen from different predetermined positions, and the mask region detection unit is obtained by the plurality of imaging devices. Alternatively, an area projected with invisible light that exists in common among a plurality of captured images may be detected as the mask area.

  One aspect of the present invention is the above-described image processing system, including a plurality of the illumination devices that emit invisible light in different wavelength ranges, and the imaging device performs imaging for each invisible light in the different wavelength ranges. By obtaining a plurality of wavelength region corresponding captured images corresponding to each invisible light of the different wavelength region, the mask region detection unit detects a mask region for each of the plurality of wavelength region corresponding captured images, The image synthesizing unit may synthesize a plurality of images individually for each of the plurality of wavelength region-capable captured images based on the mask region detected for each of the plurality of wavelength region-captured images.

  One embodiment of the present invention is a lighting device including an invisible light emitting portion that has an outer shape corresponding to carrying and emits invisible light.

  One aspect of the present invention is the above-described illumination device, which emits information that indicates whether the invisible light is emitted from the invisible light emitting unit or the invisible light is not emitted. You may further provide a state alerting | reporting part.

  One embodiment of the present invention is the illumination device described above, and may further include an irradiation range adjustment unit that adjusts an irradiation range of the invisible light emitted from the invisible light emission unit.

  As described above, according to the present invention, it is possible to easily perform an operation for designating a region.

It is a perspective view which shows the construction example of the image processing system in 1st Embodiment. It is a block diagram which shows the structural example of the image processing system in 1st Embodiment. It is a perspective view which shows the example of an external appearance of the illuminating device in 1st Embodiment. It is a block diagram which shows the structural example of the illuminating device in 1st Embodiment. It is a block diagram which shows the structural example of the image processing apparatus in 1st Embodiment. It is a figure explaining the example of the procedure of the process in which the mask area | region detection part in 1st Embodiment detects a mask area | region from a captured image. It is a figure which shows typically the pixel value conversion algorithm in 1st Embodiment. It is a figure explaining the example of a procedure of the image composition process which the image composition part in a 1st embodiment performs according to the 1st display mode. It is a flowchart which shows the example of a procedure of the image composition process which the image composition part in 1st Embodiment performs according to 1st display mode. It is a figure explaining the example of a procedure of the image composition process which the image composition part in a 1st embodiment performs according to the 3rd display mode. It is a flowchart which shows the example of a procedure of the image composition process which the image composition part in 1st Embodiment performs according to 3rd display mode. In 1st Embodiment, it is a figure explaining the size and shape of the infrared light projection area | region designated on a screen changing according to the method of the projection of the infrared light from the illuminating device with respect to a screen. It is a figure explaining an example of the state by which infrared light as direct light is imaged with the infrared light diffused by the screen. It is a figure which shows the structural example of the image processing system in 2nd Embodiment. In 2nd Embodiment, it is a figure for demonstrating an example of the image process for removing the infrared light corresponding | compatible area | region according to the infrared light as a direct light as a noise.

<First Embodiment>
[Image processing system configuration example]
The image processing system in the present embodiment is an interactive system in which an image displayed on the screen 100 changes in accordance with an operation of the lighting device 300 as an input device by a user.
Hereinafter, an overall configuration example and application example of the image processing system of the present embodiment will be described with reference to FIGS. 1 and 2. FIG. 1 is a perspective view showing an example of construction of an image processing system in the present embodiment. FIG. 2 is a block diagram illustrating a configuration example of the image processing system according to the present embodiment.

As shown in FIGS. 1 and 2, the image processing system of this embodiment includes a screen 100, a projector 200, a lighting device 300, an imaging device 400, and an image processing device 500.
The screen 100 is provided on the front surface of the housing 10 so that the user Y can observe it. The screen 100 is provided with the projector 200, the illumination device 300, the imaging device 400, and the image processing device 500 in the housing 10. However, the image processing apparatus 500 may be provided outside the housing 10.

On the screen 100, an image PI projected by the projector 200 (projection device) is displayed. In this case, the projector 200 is a rear projection type that projects image light from the back surface of the screen 100.
As shown in FIG. 1, the user Y possesses a lighting device 300 as an input device. The lighting device 300 emits infrared light (an example of invisible light). The user Y is located on the front side of the screen 100 and uses the illumination device 300 to project infrared light emitted from the illumination device 300 to a desired position on the screen 100 on which an image is displayed. When the illumination device 300 is projected onto the screen 100, it is invisible to the user Y, but the projected infrared light is scattered on the screen 100, so that an invisible image region is formed as the infrared light projection region IRA. It is formed.

  The imaging apparatus 400 uses infrared light as an imaging target. That is, the imaging device 400 is configured to be able to capture infrared light. Specifically, the imaging apparatus 400 capable of imaging infrared light can be configured by providing an optical filter that transmits infrared light and attenuates visible light in a path through which light enters the light receiving element.

The imaging device 400 is installed so as to capture the screen from the back of the screen 100. At this time, the imaging range of the imaging apparatus 400 is corrected so as to correspond to the size of the image PI displayed on the screen 100, for example.
When the imaging apparatus 400 is installed as described above, the imaging apparatus 400 images the screen 100, so that, for example, an infrared light projection area formed on the screen 100 in a screen having a size corresponding to the image PI. A captured image including the IRA is obtained.

  The captured image obtained by the imaging apparatus 400 is output to the image processing apparatus 500 as shown in FIG. Note that FIG. 1 illustrates an aspect in which the imaging apparatus 400 and the image processing apparatus 500 are connected by a cable. That is, the captured image in this case is transmitted from the imaging device 400 to the image processing device 500 via a wired signal line or a data interface. However. The imaging apparatus 400 and the image processing apparatus 500 may be connected by wireless communication so that a captured image is transmitted.

The image processing apparatus 500 is configured by a computer and performs image processing so that an image displayed on the screen 100 changes based on an infrared light projection area IRA included in a captured image input from the imaging apparatus 400.
Here, the infrared light projection area IRA is formed when the user Y performs an operation of irradiating the screen 100 with infrared light by the illumination device 300 as an input device. That is, the input result by the operation of the user Y is reflected in the captured image of this embodiment. Therefore, the image displayed on the screen 100 by the image processing apparatus 500 is an interactive image that changes according to a user operation.
More specifically, the image processing apparatus 500 generates a composite image obtained by combining the sub image with the main image corresponding to the image PI and the region corresponding to the position where the infrared light projection region IRA included in the captured image is formed. Generate.

  The image processing apparatus 500 outputs an image signal based on the composite image to the projector 200 as shown in FIG. 2 so that the composite image is displayed on the screen 100. FIG. 1 shows a state in which the image processing apparatus 500 and the projector 200 are connected by wire. However, the image processing apparatus 500 and the projector 200 may be connected by wireless communication.

  The projector 200 receives a composite image from the image processing apparatus 500 and projects image light of the input composite image on the screen 100. As a result, the composite image is displayed on the screen 100. That is, on the screen 100, an image PI is displayed in which the sub-image is synthesized and displayed in the area where the infrared light projection area IRA is formed on the screen of the image content corresponding to the main image.

[Configuration Example of Lighting Device 300]
FIG. 3 is a perspective view showing an example of the appearance of the lighting device 300. A lighting device 300 shown in the figure includes a main body 301 having a shape suitable for carrying and a head 302 irradiated with invisible light. As described above, the lighting device 300 has an outer shape corresponding to carrying.

An on / off switch 303 and an indicator 304 are provided on the upper surface of the main body 301 of the lighting device 300.
The on / off switch 303 is a switch for performing an operation of switching between a state in which the illumination device 300 emits invisible light and a state in which the illumination device 300 does not emit invisible light.
When the lighting device 300 is turned off, the power is turned on by operating the on / off switch 303, and invisible light is emitted from the head portion 302 of the lighting device 300. At this time, the invisible light is emitted radially along the direction indicated by the arrow AX in FIG. In addition, when the on / off switch 303 is operated while the power of the lighting device 300 is on, the power is turned off and emission of invisible light is stopped.

  The indicator 304 (an example of an emission state notification unit) notifies whether the invisible light is emitted from the invisible light emission unit of the illumination device 300 or a state where the invisible light is not emitted. Specifically, the indicator 304 operates so as to emit light when, for example, invisible light is emitted, and not emit light when invisible light is not emitted. The indicator 304 can be configured to include, for example, an LED (Light Emitted Diode) corresponding to visible light.

  Invisible light is invisible to humans. For this reason, in the case of the illumination device 300, the presence or absence of light emission cannot be visually confirmed, such as a flashlight that emits visible light. Therefore, in the present embodiment, by providing the indicator 304, the user can visually confirm whether or not the illumination device 300 is emitting invisible light.

  The indicator 304 may be configured to indicate whether or not invisible light is emitted by a mode other than light emission. For example, a slide switch or the like is employed for the on / off switch 303. In addition, the indicator 304 may be configured such that the color appearing from the window formed in the main body 301 is switched in conjunction with the slide switch being physically slid.

FIG. 4 is a block diagram illustrating a configuration example of the illumination device 300. In the figure, the same parts as those in FIG.
As shown in the figure, the illumination device 300 includes an on / off switch 303, an indicator 304, a control circuit 311, a battery 312, an invisible light emitting unit 313, and an irradiation range adjusting unit 314.
The control circuit 311 is a circuit that controls emission of invisible light from the invisible light emitting unit 313 and light emission of the indicator 304.
The battery 312 is a power source built in the lighting device 300. The battery 312 may be a primary battery or a secondary battery.

The invisible light emitting part 313 is a part that emits infrared light. The invisible light emitting unit 313 includes an infrared LED 313a and an optical system unit 313b.
The infrared LED 313a emits infrared light in response to the supply of drive current. The optical system unit 313b is a part that includes, for example, a lens, a reflector, and the like, and is provided in the head unit 302 (FIG. 3), for example. Infrared light emitted from the infrared LED 313a is emitted radially along the optical axis direction indicated by the arrow AX as shown in FIG. 3 through the optical system unit 313b.

The on / off switch 303 is provided so as to be inserted into a line for supplying power from the battery 312 to the control circuit 311 as shown in FIG. As the on / off switch 303 is turned on, power is supplied from the battery 312 to the control circuit 311.
The control circuit 311 supplies a drive current to the infrared LED 313a in response to the supply of power. As described above, when the drive current flows through the infrared LED 313a in response to the supply of power, infrared light is emitted from the invisible light emitting unit 313. The control circuit 311 also supplies a drive current to the indicator 304 in response to power supply. The LED in the indicator 304 emits light by the drive current.
By operating the control circuit 311 in this way, infrared light is emitted and the indicator 304 emits light and infrared light is emitted as the power is turned on by operating the on / off switch 303. Indicates a state.

Further, the irradiation range adjustment unit 314 adjusts the irradiation range of infrared light emitted from the optical system unit 313b, for example, by adjusting the angle of the reflector in the optical system unit 313b. For example, as shown by an arrow F in FIG. 3, for example, a mechanism as the irradiation range adjustment unit 314 operates by rotating the head unit 302, and the irradiation range of infrared light is changed according to the rotation angle of the head unit 302. Be changed.
Note that the irradiation range adjustment unit 314 may be omitted. When the irradiation range adjustment unit 314 is omitted, the irradiation range of invisible light is fixed.

[Configuration example of image processing apparatus]
A configuration example of the image processing apparatus 500 will be described with reference to the block diagram of FIG. An image processing apparatus 500 shown in the figure includes an input / output unit 501, a control unit 502, and a storage unit 503.

  The input / output unit 501 inputs the captured image output from the imaging device 400. Further, the input / output unit 501 outputs an image to be displayed on the screen 100 to the projector 200. The image to be displayed on the screen 100 includes a composite image in which the sub image is combined with the main image as described above. That is, the input / output unit 501 (an example of an output unit) of the present embodiment outputs a composite image so that the composite image is projected as image light on the screen by the projector 200.

  The control unit 502 executes various controls in the image processing apparatus 500. The control unit 502 shown in the figure includes a mask area detection unit 521 and an image composition unit 522.

The mask area detection unit 521 detects a mask area corresponding to the infrared light projected on the screen 100 in the captured image obtained by the imaging apparatus 400 imaging the screen 100.
The image composition unit 522 composes a plurality of images based on the mask area detected by the mask area detection unit 521. Specifically, the image composition unit 522 synthesizes the image of the area corresponding to the mask area in the sub-image with the area corresponding to the mask area in the main image.
The image composition unit 522 outputs the generated composite image from the input / output unit 501 to the projector 200. As a result, the composite image is projected onto the screen 100 by the projector 200 and displayed as an image on the screen 100.

  The storage unit 503 stores various information used by the control unit 502. The storage unit 503 shown in the figure includes a main image storage unit 531 and a sub image storage unit 532 as parts for storing information used by the image composition unit 522.

The main image storage unit 531 stores a main image. The main image may be a still image or a moving image. The main image is an image displayed as the main screen on the entire screen 100.
The sub image storage unit 532 stores the sub image. The sub image may be a still image or a moving image. The sub-image is an image that is combined with the main image.

  Here, a specific example of the contents of the main image and the sub-image will be described. Further, a specific example here is a case where interactive display in any one of a first display mode to a third display mode described below is possible by the image processing system of the present embodiment.

  In the first display mode, an old map of a certain area created at a certain past time is displayed on the screen 100 as a main image. Then, when the user Y projects infrared light onto an arbitrary place on the old map displayed on the screen 100 by the lighting device 300, the old image corresponding to the area where the infrared light projection area IRA is formed. The current map (modern map) is displayed as a comparison target at a position on the map. That is, under the first display mode, the user can compare an arbitrary place on the old map with the modern map.

  Also in the second display mode, an old map of a certain area created at a certain past time is displayed on the screen 100 as a main image. Then, when the user Y projects infrared light onto an arbitrary place on the old map displayed on the screen 100 by the lighting device 300, the old image corresponding to the area where the infrared light projection area IRA is formed. An image in which the old map at the corresponding position is enlarged is displayed at a position on the map. That is, under the second display mode, an old map at an arbitrary position in the main image can be enlarged by a user operation.

  In the third display mode, an old map of a certain area created in a certain past time is displayed on the screen 100 as a main image. Then, when the user Y projects infrared light onto an arbitrary place on the old map displayed on the screen 100 by the lighting device 300, the old image corresponding to the area where the infrared light projection area IRA is formed. A trajectory (surveying trajectory) when a surveyor performs surveying for creating an old map is displayed at a position on the map. That is, the survey trajectory can be displayed on the old map under the third display mode.

In the first display mode to the third display mode, the main image stored in the main image storage unit 531 is a common old map image having a predetermined size (the number of horizontal pixels × the number of vertical pixels).
On the other hand, the sub-image differs as follows according to each of the first display mode to the third display mode.
That is, in the first display mode, the sub-image is a modern map image having the same size as the old map image as the main image.
In the second display mode, the sub-image is, for example, an image of an old map having a predetermined size with respect to the image of the old map as the main image.
In the case of the third display mode, the sub-image is an image representing a survey trajectory having the same size as the image of the old map as the main image, for example.

In the image processing apparatus 500 of the present embodiment, the sub-image storage unit 532 stores sub-images corresponding to the display modes from the first display mode to the third display mode. Accordingly, the image processing apparatus 500 can arbitrarily select and switch the display mode between the first display mode and the third display mode.
Specifically, the image processing apparatus 500 can switch the display mode every time a predetermined time elapses, for example. Alternatively, when the main image is a moving image, the image processing apparatus 500 is configured to repeat the reproduction of the main image as a moving image and switch the display mode at the timing when the reproduction of the main image is started. it can.

[Mask detection processing example]
Next, with reference to FIG. 6, an example of a mask processing procedure executed by the mask area detection unit 521 in the image processing apparatus 500 will be described. In the following description, an example in which the image processing apparatus 500 in the present embodiment is configured to perform image processing corresponding to a frame rate of 60 fps (frame per second) will be described.
FIG. 6A shows a captured image P1 for one frame captured by the imaging device 400 and input via the input / output unit 501. The captured image P1 in the figure includes an infrared light corresponding area AR1 corresponding to the infrared light projection area IRA projected on the screen 100.

  The captured image P1 captured by the imaging apparatus 400 of the present embodiment is distorted as a whole according to the distortion of an optical system such as a lens, as shown in FIG. In the case of the configuration of FIG. 1, since the imaging device 400 captures an image from the back side of the screen 100, a captured image obtained by capturing the screen 100 is an image observed from the front side of the screen 100. The left and right are reversed.

Therefore, the mask area detection unit 521 performs distortion correction on the captured image P1. Since how the captured image P1 is distorted is known according to, for example, the characteristics of the optical system of the imaging apparatus 400, it may be corrected by a predetermined algorithm.
By the distortion correction, the captured image P1 is obtained as a captured image P2 in which the distortion of the shape of the image is corrected as shown in FIG. 6B.
Here, the aspect ratio of the captured image P2 matches the aspect ratio of the main image. For example, in the process of correcting the distortion of the captured image P1, a correction process (size correspondence correction) for matching the aspect ratio of the captured image P2 with the aspect ratio of the main image may be performed.
In addition, as a correction process, you may perform distortion correction after performing left-right inversion correction.

  Next, the mask area detection unit 521 performs correction (horizontal inversion correction) that inverts the left and right of the captured image P2. By performing the left-right reversal correction, the captured image P2 becomes a captured image P3 shown in FIG. 6C, and is in the same direction as the image obtained by observing the screen 100 from the front.

Next, the mask area detection unit 521 performs processing for detecting a mask area in the captured image P3. The mask region to be detected is a region corresponding to the invisible light projected on the screen 100 in the captured image P3, that is, the infrared light corresponding region AR1.
The mask area detection unit 521 detects a mask area as follows. Here, the captured image P3 is an image in gray scale, and the region of the infrared light corresponding region AR1 has higher luminance (pixel value) than the surroundings. Therefore, the mask area detection unit 521 detects, as a mask area, an area in which pixels having a pixel value equal to or greater than a predetermined threshold are gathered in the captured image P3.

Specifically, the mask area detection unit 521 performs a process of converting each input pixel value into an output pixel value according to a predetermined pixel value conversion algorithm, with the pixel value for each pixel forming the captured image P3 as an input pixel value. Thus, the mask area is detected.
FIG. 7 schematically shows a pixel value conversion algorithm. According to the figure, the mask area detection unit 521 converts an input pixel value less than the first threshold th1 into an output pixel value of “0 (zero)”. Then, the mask area detection unit 521 converts the input pixel value of the first threshold th1 into a predetermined output pixel value larger than “0”. The mask area detection unit 521 detects, as a mask area, an area formed by a set of pixels having an output pixel value greater than “0”, that is, an area formed by a set of pixels having an input pixel value equal to or greater than the first threshold th1.
In addition, the mask area detection unit 521 converts an input pixel value larger than the second threshold th2 into a maximum output pixel value.
In addition, the mask area detection unit 521 converts an input pixel value in the range from the first threshold th1 to the second threshold th2 into an output pixel value from “0” to the maximum value corresponding to the input pixel value.

  By performing the pixel value conversion algorithm in FIG. 7, the captured image P3 in FIG. 6C is converted into the reference image P4 in FIG. 6D. In the reference image P4, a region corresponding to the captured image P3 other than the infrared light corresponding region AR1 is black due to an output pixel value of “0”. In the reference image P4, output pixel values larger than “0” are gathered in the region corresponding to the infrared light corresponding region AR1 of the captured image P3. As described above, in the reference image P4, a region where pixels having an output pixel value larger than “0” (an input pixel value greater than or equal to the first threshold th1) is a mask region MSK.

Further, the amount of light in the infrared light projection area IRA decreases from the center to the periphery. Accordingly, the brightness (pixel value) of the infrared light corresponding area AR1 of the captured image P3 decreases as it approaches the edge side from the center. That is, on the edge side of the infrared light corresponding area AR1, there are many pixels having input pixel values corresponding to the range from the first threshold th1 to the second threshold th2. For this reason, in the edge BD in the reference image P4, there are many pixels having an intermediate output pixel value between “0” and the maximum value.
The mask area detection unit 521 of the present embodiment detects the mask area by generating a reference image as shown in FIG.

[Example of image composition processing]
Next, referring to FIG. 8, in accordance with the first display mode in which an image of a modern map at the same location on the map is combined and displayed on an area where invisible light is projected on the image of the old map. A procedure example of the image composition process executed by the image composition unit 522 in the image processing apparatus 500 will be described.
The image composition processing described below is image processing corresponding to one frame using the reference image P4 obtained by the mask area detection unit 521 by the processing described in FIGS.

  As shown in FIG. 8A, the image composition unit 522 inputs a reference image P4 that is a detection result of the mask region by the mask region detection unit 521. Further, the image composition unit 522 acquires the sub image SB1 corresponding to the first display mode from the sub image storage unit 532, as shown in FIG. 8B. As described above, the aspect ratio of the reference image P4 is corrected so as to coincide with the main image at the stage of the correction process (size corresponding correction). Note that the size of the sub-image SB1 is the same as the size of the main image.

The image composition unit 522 performs mask processing by using the sub-image SB1 as a target image, masking a region other than the mask region MSK in the reference image P4, and extracting a region included in the mask region MSK.
As a result of such mask processing, as shown in FIG. 8C, the sub-image SB1 is displayed in an area corresponding to the mask area MSK, and the area other than the mask area MSK is not displayed (for example, white level). Become.

  Next, the image composition unit 522 inputs the main image MN from the main image storage unit 531 as illustrated in FIG. Then, the sub-image SB1 shown in FIG. 8C in which only the area corresponding to the mask area MSK is displayed is superimposed on the main image MN. As a result, as shown in FIG. 8E, a composite image PSH is generated in which the extracted image EX of the sub-image SB1 at the same position as the mask area MSK is superimposed on the main image MN.

Further, the pixels in the edge BD of the extracted image EX in which intermediate pixel values between “0” and the maximum value exist densely are changed in the composition ratio at the same position in accordance with the pixel value. Synthesis with the pixel is performed. For this reason, in the extracted image EX, the content of the sub-image clearly appears at the center, but the vicinity of the edge BD is displayed with transparency so that the background main image can be seen through.
Further, the pixel value at the edge BD of the reference image P4 tends to be smaller as the outer circumference is reached. For this reason, the transparency of the extracted image EX in the composite image PSH increases as it goes outside the edge BD.

When the composite image PSH of FIG. 8E is displayed on the screen 100, the user Y replaces the old map in the area on the screen 100 where the lighting device 300 projects the invisible light. You can observe a state where a modern map of a place on the same map is displayed. That is, display as the first display mode is realized.
In addition, the extracted image EX in the composite image PSH displayed in this way has higher transparency as it goes out of the edge BD, so that the boundary with the main image MN appears to be ambiguous. As described above, the boundary between the extracted image EX and the main image MN is expressed so as to be ambiguous, so that the appearance of the extracted image EX displayed on the main image displayed on the screen 100 is made natural. can do.

  A processing procedure example executed by the image processing apparatus 500 corresponding to the mask area detection described with reference to FIGS. 6 and 7 and the image composition processing described with reference to FIG. 8 will be described with reference to the flowchart of FIG. Note that the processing shown in the figure is image processing corresponding to one frame.

First, the mask area detection unit 521 inputs a captured image of a frame as a processing target (step S101).
Next, the mask area detection unit 521 corrects the captured image (distortion correction, left / right inversion correction, and size correspondence correction) (step S102).

  Next, the mask area detection unit 521 detects the mask area MSK using the captured image corrected in step S102 (step S103). Specifically, as described with reference to FIG. 6, the mask area detection unit 521 converts the pixel value (input pixel value) of each pixel of the captured image P3 into the output pixel value by the pixel value conversion algorithm illustrated in FIG. The reference image P4 is generated by the conversion.

  Next, as described with reference to FIGS. 8A and 8B, the image composition unit 522 inputs the reference image P4 and the sub-image SB1 obtained as a result of detection of the mask area in step S103. . Then, as described with reference to FIG. 8C, the image composition unit 522 performs a mask process on the sub-image SB1 so as to extract an image in an area corresponding to the mask area MSK of the reference image P4 (Step S104). ).

Next, as described in FIG. 8E, the image composition unit 522 composes the sub-image SB1 subjected to the mask process in step S104 with the main image MN, and generates a composite image PSH (step S105). ).
The mask area detection unit 521 and the image composition unit 522 generate the composite image PSH by executing the processes of steps S101 to S105 for each frame as described above. Then, the image composition unit 522 causes the projector 200 to output an image signal using the composite image PSH as a frame image from the input / output unit 501.

  The image composition unit 522 performs image processing according to the same procedure as in FIGS. 8 and 9 by using the sub-image of the surveying trajectory corresponding to the third display mode, so that the display as the first display mode is performed. Can be realized.

Further, in the description with reference to FIGS. 8 and 9 described above, a case where one mask area is detected from the captured image in accordance with the fact that one infrared light projection area IRA is formed on the screen 100 is taken as an example. ing.
However, the first display mode and the third display mode are realized by a process of combining the sub-image with the main image by the mask process. Therefore, in the first display mode and the third display mode, even when a plurality of mask areas are detected from the captured image, the sub-image is combined with the main image according to each mask area by the process shown in FIG. A composite image is generated. Therefore, the number of users who operate the lighting device 300 is not particularly limited.

Next, referring to FIG. 10, in response to the second display mode in which an enlarged view of the old map at the same map location is synthesized and displayed on the area where the invisible light on the old map image is projected. A procedure example of the image composition processing executed by the image composition unit 522 will be described.
The image composition processing described below is image processing corresponding to one frame using the reference image P4 obtained by the mask area detection unit 521 by the processing described in FIGS.

As shown in FIG. 10A, the image composition unit 522 inputs a reference image P4 that is a detection result of the mask region by the mask region detection unit 521.
In addition, the image composition unit 522 acquires the sub image SB2 corresponding to the second display mode from the sub image storage unit 532, as illustrated in FIG. The sub-image SB2 is used for displaying an image obtained by enlarging the main image as described above. In the figure, an example is shown in which a sub-image in which the vertical direction and the horizontal direction of the main image are enlarged at a magnification of 200% is displayed. In this case, the sub-image SB2 has a size of 2X × 2Y, where X is the vertical size (number of pixels) of the main image and Y is the horizontal size of the main image.
The magnification of the sub image with respect to the main image in the second display mode is not particularly limited. Further, the sub-image magnification may be a numerical value larger than 1 or a numerical value smaller than 1. That is, a reduced image may be displayed as a sub image in the second display mode.

Here, the reference image P4 in FIG. 10A has a size of X × Y with the main image. On the other hand, the sub-image SB2 has a size of 2X × 2Y. Therefore, the image composition unit 522 in the present embodiment specifies the position corresponding to the mask area MSK in the sub-image SB2 in order to display an enlarged view of the position on the same map at the position corresponding to the mask area MSK in the main image. To do.
In specifying the position corresponding to the mask area MSK in the sub-image SB2, the image composition unit 522 calculates the barycentric coordinates GT1 of the mask area MSK in the reference image P4 as shown in FIG.
Next, as shown in FIG. 10B, the image composition unit 522 calculates auxiliary centroid coordinates GT2 corresponding to the centroid coordinates GT1 in the sub-image SB2. The auxiliary center-of-gravity coordinate GT2 is a coordinate on the sub-image SB2 that coincides with the center-of-gravity coordinate GT1 when the reference image P4 is doubled to have the same size as the sub-image SB2.

The image composition unit 522 obtains a position corresponding to the mask region MSK in the sub-image SB2 for which the auxiliary barycentric coordinates GT2 has been obtained. For this purpose, the image composition unit 522 extracts the mask region MSK from the reference image P4 together with the barycentric coordinates GT1. Then, as shown in FIG. 10C, the image composition unit 522 superimposes the extracted mask area MSK on the sub-image SB2 so that the center-of-gravity coordinates GT1 and the auxiliary center-of-gravity coordinates GT2 coincide.
In addition, as shown in FIG. 10C, the image composition unit 522 masks a region other than the mask region MSK for the sub-image SB2, and performs mask processing so as to extract an image of the region where the mask region MSK is superimposed. I do. By such mask processing, an image portion included in the extracted image EX on which the mask area MSK is superimposed is extracted from the sub-image SB2.

Next, the image composition unit 522 acquires the main image MN corresponding to the second display mode from the main image storage unit 531 as illustrated in FIG. As described above, the size of the main image MN is X × Y like the reference image P4.
When the acquired main image MN and the reference image P4 in FIG. 10A are superimposed, the image composition unit 522 applies the mask described in FIG. 10C to the region corresponding to the mask region MSK in the main image MN. The extracted image EX extracted by the processing is synthesized.
As a result of the above-described combining process, the image combining unit 522 displays the extracted image EX of the sub-image SB2 at the same position as the mask area MSK superimposed on the main image MN, as shown in FIG. A composite image PSH is generated.

  Also in the second display mode, the composition ratio at the same position of the pixels in the edge BD of the extracted image EX where there are many intermediate pixel values between “0” and the maximum value is changed according to the pixel value. After that, composition with the pixels of the main image is performed.

When the composite image PSH of FIG. 10E is displayed on the screen 100, the user Y himself / herself is the lighting device 300 on the screen 100 on which the standard scale old map based on the main image is displayed. Thus, in the region where invisible light is projected, it is possible to observe a state in which an old map enlarged by a scale of 2 times the standard is displayed. That is, display as the second display mode is realized.
In addition, the extracted image EX in the composite image PSH displayed in this way also has higher transparency as it goes outside the edge BD, and the boundary with the main image MN is made ambiguous.

  With reference to the flowchart of FIG. 11, an example of a processing procedure executed by the image processing apparatus 500 corresponding to the image composition processing described with reference to FIG. 10 will be described. Note that the processing shown in the figure is image processing corresponding to one frame. Further, in FIG. 10, for convenience of understanding the description, the basic image composition processing according to the case where one mask area is detected from the captured image has been described. However, in the flowchart of FIG. A procedure example of image composition processing corresponding to a case where a region is detected will be described.

  In the same figure, the process of step S201-S203 is a process performed in order that the mask area | region detection part 521 detects a mask area | region similarly to step S101-S103 of FIG. That is, the processing procedure for detecting the mask area is common to the first display mode, the third display mode, and the second display mode. Depending on steps S201 to S203, a plurality of mask regions may be detected from the captured image.

  Next, as described with reference to FIG. 10A, the image composition unit 522 calculates the barycentric coordinates GT1 of the mask region MSK in the reference image P4 obtained as the detection result of the mask region in step S203 (step S204). . Here, when there are a plurality of mask regions MSK in the reference image P4 (when a plurality of mask regions MSK are detected), the image composition unit 522 calculates the barycentric coordinates GT1 for each of the plurality of mask regions MSK.

Further, the image composition unit 522 performs labeling for each mask region MSK present in the reference image P4 (step S205). When there are a plurality of mask regions MSK in the reference image P4 by labeling, the plurality of mask regions MSK are discriminated. In the labeling process, for example, the reference image P4 is binarized. Further, in the labeling process, for example, a process of reducing the size (resolution) may be performed in order to reduce the processing load.
In addition, the image composition unit 522 substitutes “0” as an initial value for a variable n indicating a label number assigned to each mask area MSK labeled in step S206 (step S206). In this case, the label numbers are assigned in ascending order from “0”.

Next, as described with reference to FIG. 10B, the image composition unit 522 calculates auxiliary centroid coordinates GT2 corresponding to the centroid coordinates GT1 of the mask area with the label number n in the sub-image SB2 (step S207).
Next, as described with reference to FIG. 10C, the image composition unit 522 superimposes the mask region MSK with the label number n on the sub-image SB2 using the auxiliary barycentric coordinates GT2 calculated in step S207 as a reference. Then, the image composition unit 522 extracts the image portion of the area corresponding to the mask area MSK with the label number n in the sub-image SB2 as the extracted image EX (Step S208).
Next, as described with reference to FIG. 19E, the image composition unit 522 synthesizes the extracted image EX extracted in step S208 with the region corresponding to the mask region MSK with the label number n in the main image MN. (Step S209).
One extracted image EX corresponding to the nth mask region is synthesized with the main image MN by calculating the auxiliary barycentric coordinates in step S207 and the mask processing in steps S208 and S209.

When the process of step S209 ends, the image composition unit 522 increments the variable n (step S210). The image composition unit 522 determines whether or not the variable n incremented in step S210 is larger than the maximum value (step S211).
If the variable n is less than or equal to the maximum value (step S211-NO), there remains a mask area MSK that has not yet been subjected to mask processing (combination of the extracted image EX with respect to the main image MN). Therefore, the image composition unit 522 in this case returns the process to step S207. As a result, mask processing is performed according to the label number n corresponding to each mask region MSK.

When the mask processing corresponding to all the mask areas MSK is completed, it is determined that the variable n has become larger than the maximum value (step S211—YES). When it is determined in step S211 that the variable n has become larger than the maximum value, the extracted image EX extracted from the sub-image SB2 is combined for every mask area MSK detected in step S203. A broken composite image PSH is generated.
Thereafter, the mask area detection unit 521 and the image composition unit 522 execute the processes in steps S201 to S211 according to the timing for each frame. Then, the image synthesis unit 522 causes the projector 200 to output an image signal using the synthesized image PSH as a frame image from the input / output unit 501 as a display image signal.

As can be understood from the above description, in the present embodiment, the user projects the infrared light onto the screen 100 using the illumination device 300 as an input device to form the infrared light projection area IRA. Thus, an operation for designating an area can be performed.
In other words, in this embodiment, when designating a region, the user only needs to perform a simple operation of projecting invisible light onto the screen 100 by the illumination device 300, and form the contour of the region by drawing a coordinate movement trajectory. There is no hassle of performing operations such as.

In the present embodiment, as illustrated in FIG. 12, the size and shape of the designated area can be easily changed.
In FIG. 12A, a state in which infrared light is projected from the three illumination devices 300-1, 300-2, and 300-3 onto the screen 100 is shown from the planar direction. In the description of the drawing, the lighting devices 300-1, 300-2, and 300-3 are described as the lighting device 300 unless otherwise distinguished.

FIG. 12B shows the screen 100 in a state where an infrared light projection region is formed by infrared light projected from each of the lighting devices 300-1, 300-2, and 300-3 from the front side. FIG.
In the drawing, an infrared light projection area IRA-1 is formed by infrared light projected from the illumination device 300-1. The infrared light projection area IRA-2 is formed by infrared light projected from the illumination device 300-2. The infrared light projection area IRA-3 is formed by infrared light projected from the illumination device 300-3. In the description of the figure, the infrared light projection areas IRA-1, IRA-2, and IRA-3 are described as infrared light projection areas IRA unless otherwise distinguished.

Here, when the distance ds1 from the lighting device 300-1 to the screen 100 is compared with the distance ds2 from the lighting device 300-2 to the screen 100, the distance ds1 is longer. Then, the invisible light emitted from the illumination device 300 spreads radially according to a certain irradiation angle.
Therefore, as shown in FIG. 12B, the infrared light projection area IRA-1 corresponding to the illumination device 300-1 is compared with the infrared light projection area IRA-2 corresponding to the illumination device 300-2. In this case, the infrared light projection area IRA-1 is larger.
That is, in this embodiment, the size of the area to be specified can be easily changed by the user moving in the front-rear direction in front of the screen 100 and changing the distance from the lighting device 300 to the screen 100.

In addition, as illustrated in FIG. 12A, the illumination device 300-1 and the illumination device 300-2 project infrared light from the front almost on the screen 100, whereas the illumination device 300- 3 is projected with a certain tilt angle with respect to the screen 100.
For this reason, both the infrared light projection areas IRA-1 and IRA-2 corresponding to the illumination device 300-1 and the illumination device 300-2 are substantially circular. On the other hand, the infrared light projection area IRA-3 corresponding to the illumination device 300-3 is flattened from a circle and becomes an ellipse extending in the lateral direction.
That is, in the present embodiment, the shape of the infrared light projection area IRA can be easily changed by changing the irradiation angle of the infrared light from the illumination device 300 to the screen 100.
Further, the shape of the infrared light projection area IRA can be simplified by, for example, blocking a part of the opening from which infrared light is emitted in the head unit 302 (FIG. 3) of the illumination device 300 with a hand or a plate. Can be changed.

Second Embodiment
Next, a second embodiment will be described.
The infrared light projection area IRA formed on the screen 100 is formed by the infrared light projected on the screen 100 diffusing on the screen 100. Since the screen 100 in this embodiment corresponds to the rear projection type projector 200, the screen 100 has a certain degree of light transmission. For this reason, infrared light as direct light transmitted through the screen 100 may become strong. In this case, the imaging apparatus 400 captures infrared light that has directly diffused to the imaging apparatus 400 without being diffused by the screen 100 together with infrared light that has been diffused by the screen 100.

An example of a state in which infrared light as direct light is imaged together with infrared light diffused by the screen 100 as described above will be described with reference to FIG.
FIG. 13A shows a state in which infrared light is radiated from the lighting device 300 to the screen 100 at a position on the right side on the front side of the screen 100.
In this case, in the infrared light beam emitted from the illumination device 300, the range of infrared light that is diffused by the screen 100 is indicated by the light range L 1, and red that is transmitted through the screen 100 without being diffused by the screen 100. The range of outside light is indicated by the light range L2.
The light range L1 does not include the imaging device 400 installed on the back surface of the screen 100, but the light range L2 includes the imaging device 400. In such a case, the infrared light received by the imaging device 400 is relatively stronger than the infrared light diffused by the screen 100 as the direct light.

When the infrared light as the direct light is relatively stronger than the infrared light diffused by the screen 100 as described above, the captured image P3 obtained by the imaging device 400 imaging the screen 100 is: The state shown in FIG.
That is, in the captured image P3, in addition to the infrared light corresponding area AR11 according to the infrared light diffused on the screen 100, there is a state in which an infrared light corresponding area AR12 corresponding to infrared light as direct light exists. Become.

Here, of the infrared light corresponding area AR11 and the infrared light corresponding area AR12, the infrared light corresponding area AR11 corresponds to the intention of the user who operates the illumination device 300. Therefore, in this case, it is only necessary to display a combined image in which images are combined according to the mask area corresponding to the infrared light corresponding area AR11.
However, when the mask area is detected for the captured image P3 shown in FIG. 13B, in addition to the mask area corresponding to the infrared light corresponding area AR11, the mask area corresponding to the infrared light corresponding area AR12 is also reached. It will be detected. In this case, the screen 100 has not only the area where the sub-image is synthesized based on the mask area corresponding to the infrared light corresponding area AR11 but also the sub image based on the mask area corresponding to the infrared light corresponding area AR12. The combined area will be displayed. That is, in this case, a malfunction occurs in which an image that does not correspond to the user's operation is displayed on the screen 100.

  One of the countermeasures against the above problems is to set the first threshold th1 described in FIG. 7 high. By setting the first threshold value high, an input pixel value having a somewhat large value can be set to an output pixel value of “0”. For this reason, if the input pixel value of the direct light is less than the first threshold th1, it is possible to prevent the image portion corresponding to the direct light from being detected as a mask region.

However, for example, when the direct light received by the imaging device 400 is considerably strong, it is difficult to prevent the image portion of the direct light from being detected as a mask region by adjusting the first threshold th1.
Also, increasing the first threshold th1 leads to a narrowing of the range from the first threshold th1 to the second threshold th2. If the range from the first threshold th1 to the second threshold th2 becomes narrow, it may be difficult to see because the region of the extracted image EX and the main image MN in the composite image PSH becomes clear.

Therefore, the image processing system according to the present embodiment is configured to be able to remove the infrared light corresponding region corresponding to the infrared light as the direct light as noise. If the infrared light corresponding area corresponding to the infrared light as the direct light is removed as noise, it is possible to prevent erroneous detection of the mask area corresponding to the infrared light corresponding area corresponding to the infrared light as the direct light. Is done.
Therefore, hereinafter, a configuration for removing an infrared light corresponding region corresponding to infrared light as direct light as noise will be described.

  FIG. 14 shows a configuration example of the image processing system in the present embodiment. As shown in the figure, in the present embodiment, two imaging devices 400-1 and 400-2 are provided so as to image the screen 100 from different positions. In the following description, the imaging devices 400-1 and 400-2 are referred to as the imaging device 400 unless otherwise distinguished.

In the imaging device 400-1 and the imaging device 400-2, when one imaging device 400 is included in the direct light range L2, the other imaging device 400 is not included in the direct light range L2. It has the positional relationship made. The distance Cds between the imaging device 400-1 and the imaging device 400-2 is set so as to obtain the above positional relationship.
In the drawing, the imaging device 400-1 is included in the light range L2 of direct light, but the imaging device 400-2 is not included in the light range L2.

With reference to FIG. 15, an example of image processing for removing, as noise, an infrared light corresponding region corresponding to infrared light as direct light will be described in the present embodiment.
In the figure, a captured image P3-1 and a captured image P3-2 are shown. The captured image P3-1 is a corrected captured image obtained by capturing with the imaging device 400-1 in FIG. The captured image P3-2 is a corrected captured image obtained by capturing with the imaging device 400-2.

The imaging device 400-1 is included in the direct light range L2 as shown in FIG. For this reason, in the captured image P3-1, an infrared light corresponding area AR11 corresponding to the infrared light diffused by the screen 100 and an infrared light corresponding area AR12 corresponding to the infrared light as direct light are included. include.
On the other hand, the imaging device 400-2 is not included in the light range L2 of direct light. For this reason, the captured image P3-2 includes the infrared light corresponding area AR11 corresponding to the infrared light diffused by the screen 100, but the infrared light corresponding to the infrared light as the direct light. The corresponding area AR12 is not included.

The mask area detection unit 521 in the present embodiment is common between the captured image P3-1 and the captured image P3-2 by performing, for example, multiplication processing on the captured image P3-1 and the captured image P3-2. A captured image P3 including the existing infrared light corresponding region and not including the infrared light corresponding region that does not exist in common is generated.
In this way, in the present embodiment, the captured image P3 in FIG. 6C can be configured not to include an infrared light corresponding region corresponding to infrared light as direct light. Then, the mask area detection unit 521 performs mask area detection for the captured image P3. Thereby, the mask area | region according to the infrared light as direct light is no longer detected, As a result, the synthesized image as the user who operates the illuminating device 300 intended can be displayed.

The configuration of the present embodiment illustrated in FIGS. 14 and 15 is also effective for disturbance light that is applied to the screen 100 from other than the illumination device 300 and passes through the screen 100.
FIG. 14 shows an example in which two imaging devices 400 are provided. However, three or more imaging devices 400 are provided, and the mask area detection unit 521 is present in common among the three or more captured images. A captured image including an infrared light corresponding region may be generated.

[Modification]
In the present embodiment, it is possible to perform different synthesis processing depending on each of the plurality of lighting devices 300 with the following configuration.
That is, about the some illuminating device 300, it comprises so that the infrared light by a respectively different wavelength range may be radiate | emitted. For this purpose, for example, a filter that allows infrared light in a different wavelength region to pass for each of the plurality of lighting devices 300 may be provided.

In addition, the imaging device 400 is configured so that infrared light for each wavelength range corresponding to the plurality of illumination devices 300 can be individually imaged.
For this purpose, a plurality of imaging devices 400 that perform imaging of infrared light corresponding to each wavelength range of infrared light emitted from the plurality of illumination devices 300 may be provided.
Alternatively, a spin wheel, a liquid crystal tunable filter, or the like may be applied to one imaging device 400 so that a captured image obtained by imaging infrared light in a different wavelength range for each field or frame, for example, may be obtained.

Then, the image processing apparatus 500 individually performs mask region detection and image synthesis processing of the sub-image (extracted image EX) for the main image for each of a plurality of captured images corresponding to different wavelength regions (wavelength region-capable captured images). . With such a configuration, a completely different extracted image EX can be displayed for each user who operates the lighting apparatus 300 according to the operation.
As an example, for example, when three users are operating the same screen 100 using the lighting device 300, the following display can be performed. In other words, the modern map is displayed in the first display mode for the first user, the enlarged view is displayed in the second display mode for the second user, and the third user is supported. In this case, the survey trajectory is displayed in the third display mode.

  Note that the image processing system in the embodiments so far has a rear projection type configuration in which an image is projected from the rear surface of the screen 100 by the projector 200. However, the image processing system of the present embodiment may be a front projection type in which an image is projected from the front surface of the screen by a projector. In the case of the front projection type, the imaging device is also provided so as to capture the screen from the front side of the screen.

  Note that a program for realizing the functions of the above-described image processing apparatus 500 is recorded on a computer-readable recording medium, and the program recorded on the recording medium is read into a computer system and executed, thereby executing the above-described process. Processing as the image processing apparatus 500 may be performed. Here, “loading and executing a program recorded on a recording medium into a computer system” includes installing the program in the computer system. The “computer system” here includes an OS and hardware such as peripheral devices. The “computer system” may include a plurality of computer devices connected via a network including a communication line such as the Internet, WAN, LAN, and dedicated line. The “computer-readable recording medium” refers to a storage device such as a flexible medium, a magneto-optical disk, a portable medium such as a ROM and a CD-ROM, and a hard disk incorporated in a computer system. As described above, the recording medium storing the program may be a non-transitory recording medium such as a CD-ROM. The recording medium also includes a recording medium provided inside or outside that is accessible from the distribution server in order to distribute the program. The code of the program stored in the recording medium of the distribution server may be different from the code of the program that can be executed by the terminal device. That is, the format stored in the distribution server is not limited as long as it can be downloaded from the distribution server and installed in a form that can be executed by the terminal device. Note that the program may be divided into a plurality of parts, downloaded at different timings, and combined in the terminal device, or the distribution server that distributes each of the divided programs may be different. Furthermore, the “computer-readable recording medium” holds a program for a certain period of time, such as a volatile memory (RAM) inside a computer system that becomes a server or a client when the program is transmitted via a network. Including things. The program may be for realizing a part of the functions described above. Furthermore, what can implement | achieve the function mentioned above in combination with the program already recorded on the computer system, what is called a difference file (difference program) may be sufficient.

10 housing, 100 screen, 200 projector, 300 illumination device, 301 main body, 302 head, 303 on / off switch, 304 indicator, 311 control circuit, 312 battery, 313 invisible light emitting unit, 313a infrared LED, 313b optical system unit 314 Irradiation range adjustment unit, 400 imaging device, 500 image processing device, 501 input / output unit, 502 control unit, 503 storage unit, 521 mask area detection unit, 522 image composition unit, 530 main image storage unit, 532 sub image storage Part

Claims (13)

A mask region detection unit that detects a mask region corresponding to the invisible light projected on the screen in a captured image obtained by the imaging device that captures the invisible light imaging the screen;
An image synthesis unit that synthesizes a plurality of images based on the mask area detected by the mask area detection unit;
An image processing apparatus comprising: The image processing apparatus according to claim 1, wherein the image synthesis unit synthesizes an image of an area corresponding to the mask area in a sub-image with an area corresponding to the mask area in the main image. When the size of the sub image is different from that of the main image, the image composition unit is based on the barycentric coordinates of the region corresponding to the mask region in the main image and the ratio of the size of the sub image and the main image. The image processing apparatus according to claim 2, wherein an area corresponding to the mask area is specified in the sub-image. The image processing apparatus according to any one of claims 1 to 3, wherein the mask area detection unit detects, as the mask area, an area in which pixels having a pixel value equal to or greater than a first threshold value in the captured image. The mask area detection unit has a pixel value in a range from the first threshold value to the second threshold value among pixel values of the pixels included in the mask area, which is from zero to less than a maximum value according to the pixel value. Converting to an output pixel value, and converting a pixel value greater than the second threshold value to a maximum output pixel value;
The image processing apparatus according to claim 4, wherein the image synthesis unit synthesizes pixels included in an area corresponding to the mask area in the sub-image with pixels of the main image based on the output pixel value. The mask area detection unit is configured to display an area in which invisible light that is present in common among a plurality of captured images obtained by a plurality of imaging devices that capture the screen from different predetermined positions is projected, as the mask area. The image processing apparatus according to any one of claims 1 to 5. The imaging device obtains a plurality of wavelength range-capable imaging images corresponding to the invisible light in the different wavelength regions by performing imaging for each invisible light in different wavelength regions,
The mask area detection unit detects a mask area for each of the plurality of captured images corresponding to wavelength ranges,
The said image synthetic | combination part synthesize | combines a several image separately for every said several wavelength range corresponding captured image based on the mask area | region detected for every said several wavelength range corresponding captured image. An image processing apparatus according to claim 1. An image processing system comprising an illumination device, an imaging device, and an image processing device,
The lighting device includes an invisible light emitting unit that emits invisible light,
The imaging device is provided to capture an invisible light to be imaged and to image a screen on which the invisible light emitted from the lighting device is projected,
The image processing apparatus includes:
A mask area detection unit that detects a mask area corresponding to invisible light projected on the screen in a captured image obtained by the imaging apparatus;
An image composition unit that composes a plurality of images based on the mask region detected by the mask region detection unit;
An image processing system comprising: an output unit that outputs the synthesized image so that the image synthesized by the image synthesizing unit is projected as image light on the screen by a projection device. A plurality of the imaging devices for imaging the screen from different predetermined positions,
The image according to claim 8, wherein the mask area detection unit detects, as the mask area, an area projected with invisible light that exists in common among a plurality of captured images obtained by the plurality of imaging devices. Processing system. A plurality of the illumination devices that emit invisible light in different wavelength ranges,
The imaging device obtains a plurality of wavelength-band-captured captured images corresponding to the invisible light in the different wavelength ranges by performing imaging for the invisible light in the different wavelength ranges,
The mask area detection unit detects a mask area for each of the plurality of captured images corresponding to wavelength ranges,
The said image synthetic | combination part synthesize | combines a several image separately for every said several wavelength range corresponding captured image based on the mask area | region detected for every said several wavelength range corresponding captured image. Image processing system. An illuminating device including an invisible light emitting unit that has an outer shape corresponding to a mobile phone and emits invisible light. The illuminating device according to claim 11, further comprising: an emission state notifying unit that notifies whether the invisible light is emitted from the invisible light emitting unit or the invisible light is not emitted. The illumination device according to claim 11, further comprising an irradiation range adjustment unit that adjusts an irradiation range of the invisible light emitted from the invisible light emission unit.

Images