JP2018042230A - Image projection system, program, and image projection method - Google Patents

Abstract

PROBLEM TO BE SOLVED: To provide an image projection system and the like, capable of acquiring a color or pattern of a contact surface of an object that comes into contact with an input panel.SOLUTION: An image projection system comprises: an input panel for making infrared light from a light source be transmitted inside; a projection screen arranged so as to be superposed on the back surface side of the input panel; an infrared camera for imaging scattered light generated by an object coming into contact with a front surface of the input panel, from the back surface side; a projector for projecting an image on the projection screen from the back surface side; a visible light camera for imaging the input panel from the back surface side; and a processing unit. The processing unit creates an original image for projection; binarizes an infrared image imaged by the infrared camera to generate a mask image; combines the original image with the mask image to generate a composite image; transmits the composite image to the projector to make the projector project the composite image; combines a visible light image imaged by the visible light camera with the mask image to generate a contact surface image obtained by extracting an image of an area corresponding to a contact portion; and generates a new original image on the basis of the contact surface image.SELECTED DRAWING: Figure 1

Description

  The present invention relates to an image projection system, a program, and an image projection method.

  Non-Patent Document 1 discloses a digital paint system that uses an arbitrary object for input. This system has an object contact shape acquisition function using FTIR and an output function using a rear transmission projector.

Mami Kosaka and Kaori Fujinami, “Improvement and User Evaluation of Input / Output Panel in Digital Paint System UnipaintPaint Using Arbitrary Objects”, Information Processing Society of Japan Interaction 2016, March 4, 2016, p. 1016-1021

  In the digital paint system disclosed in Non-Patent Document 1, although the contact shape of an object placed on the input panel (projection surface) can be acquired, the color and pattern of the contact surface of the object cannot be acquired.

  The present invention has been made in view of the above problems, and an object of the present invention is to provide an image projection system and the like that can acquire the color and pattern of the contact surface of an object that has contacted the input panel. It is to provide.

  (1) The present invention is caused by an input panel that guides infrared light from a light source therein, a projection screen that is placed on the back side of the input panel, and an object that is in contact with the surface of the input panel. Image projector comprising: an infrared camera that captures the scattered light from the back side; a projector that projects an image onto the projection screen from the back side; a visible light camera that captures the input panel from the back side; and a processing unit. In the system, the processing unit includes: an original image generation unit that generates an original image for projection; a mask image generation unit that generates a mask image by binarizing an infrared image captured by the infrared camera; A composite image generation unit that generates a composite image by combining the original image and the mask image, a projection control unit that transmits the composite image to the projector for projection, and a projection completion of the composite image An imaging control unit that causes the visible light camera to perform imaging later, and a visible light image captured by the visible light camera and the mask image are combined to contact the object and the input panel from the visible light image. A contact surface image generation unit that generates a contact surface image obtained by extracting an image of an area corresponding to a part, wherein the original image generation unit generates the original image based on the contact surface image, and generates the composite image. The generation unit relates to an image projection system that generates the composite image so that the contact portion becomes monochromatic when viewed from the visible light camera when the composite image is projected.

Further, the present invention is caused by an input panel that guides infrared light from a light source to the inside, a projection screen that is placed on the back side of the input panel, and an object that is in contact with the surface of the input panel. A program for an image projection system, comprising: an infrared camera that captures scattered light from the back side; a projector that projects an image on the projection screen from the back side; and a visible light camera that captures the input panel from the back side An original image generation unit that generates an original image for projection, a mask image generation unit that generates a mask image by binarizing an infrared image captured by the infrared camera, the original image, and the mask image A combined image generating unit that generates a combined image, a projection control unit that transmits the combined image to the projector for projection, and after completion of the projection of the combined image An imaging control unit that causes the visible light camera to perform imaging, a visible light image captured by the visible light camera, and the mask image, and a contact portion between the object and the input panel from the visible light image The computer functions as a contact surface image generation unit that generates a contact surface image obtained by extracting an image corresponding to the region, and the original image generation unit generates the original image based on the contact surface image, and the combined image The generation unit relates to a program for generating the composite image so that the contact portion becomes monochromatic when viewed from the visible light camera when the composite image is projected. The present invention also relates to a computer-readable information storage medium that stores the program.

  Further, the present invention is caused by an input panel that guides infrared light from a light source to the inside, a projection screen that is placed on the back side of the input panel, and an object that is in contact with the surface of the input panel. The image projection system includes an infrared camera that captures scattered light from the back side, a projector that projects an image on the projection screen from the back side, and a visible light camera that captures the input panel from the back side. An image projecting method, an original image generating step for generating an original image for projection, a mask image generating step for generating a mask image by binarizing an infrared image captured by the infrared camera, and the original image A composite image generating step of generating a composite image by combining the mask image, and a projection control step of transmitting the composite image to the projector for projection An imaging control step for causing the visible light camera to capture an image after the projection of the composite image is completed; and a visible light image captured by the visible light camera and the mask image are combined, and the object is obtained from the visible light image. And a contact surface image generating step for generating a contact surface image obtained by extracting an image of a region corresponding to a contact portion between the input image and the input panel. In the original image generating step, the original image is based on the contact surface image. And the composite image generation step generates the composite image so that the contact portion becomes monochromatic when viewed from the visible light camera when the composite image is projected.

  According to the present invention, an original image for projection and a mask image are synthesized, and a synthesized image is generated such that a region corresponding to a contact portion between an object and an input panel is monochromatic when viewed from a visible light camera at the time of projection. By extracting the image corresponding to the contact part from the visible light image captured by the visible light camera, the influence of the projection light of the projector is eliminated, and the color or pattern of the contact surface of the object is acquired. Can do.

  (2) In the image projection system, the program, and the image projection method according to the present invention, when the projection of the composite image is completed, the projector transmits notification information to the processing unit, and the imaging control unit ( In the imaging control step), when the notification information is received, the visible light camera may perform imaging.

  ADVANTAGE OF THE INVENTION According to this invention, it can prevent acquiring the visible light image imaged with the visible light camera before the projection of a synthesized image is completed.

  (3) In the image projection system, the program, and the image projection method according to the present invention, the imaging control unit (in the imaging control step) sets an area corresponding to the contact portion in the visible light image in the composite image. A difference may be calculated by comparing an image having the same color as the region and the composite image, and imaging with the visible light camera may be repeatedly performed until the calculated difference is within an allowable range.

  According to the present invention, even when the projector does not have a notification function or when the processing unit does not have a function to receive a notification, the image is captured by the visible light camera before the projection of the composite image is completed. It is possible to prevent a visible light image from being acquired.

  (4) In the image projection system, the program, and the image projection method according to the present invention, the imaging control unit (in the imaging control step) transmits the composite image to the projector after a predetermined time has passed. You may make an optical camera image.

  According to the present invention, even when the projector does not have a notification function or when the processing unit does not have a function to receive a notification, the image is captured by the visible light camera before the projection of the composite image is completed. It is possible to prevent a visible light image from being acquired.

  (5) In the image projection system, the program, and the image projection method according to the present invention, the original image generation unit (in the original image generation step) displays the movement trajectory of the object in contact with the input panel as the contact surface image. An image drawn using a color or pattern may be generated as the original image.

  ADVANTAGE OF THE INVENTION According to this invention, not only the contact shape of the object which contacted the input panel but the digital paint which can be drawn using the color and pattern of the contact surface of the said object are realizable.

  (6) In the image projection system, the program, and the image projection method according to the present invention, the image projection system, the program, and the image projection method further include a pressure sensor for detecting contact of the object with the input panel, and the mask image generation unit When the contact of an object is detected, the infrared image is binarized with a first threshold value to generate a mask image, and when the number of white pixels in the generated mask image is less than a predetermined value, the infrared image May be binarized with a second threshold value less than the first threshold value to generate a mask image again.

  According to the present invention, the shape, color, and pattern of the contact surface of an object that has contacted the input panel can be acquired with high accuracy.

It is a figure which shows typically the structure of the image projection system which concerns on 1st Embodiment. It is a side view which shows an example of a structure of the input panel in 1st Embodiment. It is an example of the functional block diagram of the image projection system which concerns on 1st Embodiment. It is a figure for demonstrating a comparative example. It is a flowchart which shows the flow of a process of the image projection system in 1st Embodiment. It is a figure which shows typically an example of an original image, a mask image, the image for correction | amendment, the mask image after correction | amendment, and a synthesized image. It is a figure which shows typically a projection image when a synthesized image is projected on a projection screen. It is a figure which shows typically an example of a visible light image and a contact surface image. It is a figure which shows typically the original image and synthetic | combination image which were produced | generated by drawing the movement locus | trajectory of an object with the color or pattern of the said object. It is a figure for demonstrating the timing of projection by a projector, and imaging | photography with a visible light camera. It is a figure for demonstrating the 2nd method of performing the imaging by a visible light camera after the projection by a projector is completed. It is a side view which shows an example of a structure of the input panel in 2nd Embodiment. It is a flowchart which shows the flow of a process of the image projection system in 2nd Embodiment. It is a figure which shows an example of the mask image which binarized the infrared image and the infrared image.

  Hereinafter, this embodiment will be described. In addition, this embodiment demonstrated below does not unduly limit the content of this invention described in the claim. In addition, all the configurations described in the present embodiment are not necessarily essential configuration requirements of the present invention.

(First embodiment)
FIG. 1 is a diagram schematically illustrating a configuration of an image projection system according to the first embodiment. The image projection system 1 includes an input panel 10 that guides infrared light from a light source therein, a projection screen 20 that is disposed on the back side of the input panel 10, and an object B that is in contact with the surface of the input panel 10. An infrared camera 30 that captures the scattered light generated from the back side, a projector 40 that projects an image on the projection screen 20 from the back side, a visible light camera 50 (RBG camera) that images the input panel 10 from the back side, And processing unit 100. The projection range of the projector 40 and the imaging ranges of the infrared camera 30 and the visible light camera 50 are approximately the same. When the user U places (contacts) an arbitrary object B on the input panel 10, an image corresponding to the shape, color, and pattern (including characters) of the contact surface of the object B is projected onto the projection screen 20. U can observe the projected image from the surface side of the input panel 10.

  FIG. 2 is a side view showing an example of the configuration of the input panel 10 according to the first embodiment. The input panel 10 is a transparent panel using an infrared light total reflection phenomenon (FTIR: Frustrated Total Internal Reflection), and includes a transparent plate 11 (for example, a transparent acrylic plate) having a refractive index higher than that of air. . An infrared light source 12 (for example, an infrared LED) is disposed at a position facing the side surface of each side of the transparent plate 11. Further, in order to improve the sensitivity of FTIR, a sheet-like transparent silicone rubber 13 is stuck on the surface of the transparent plate 11. The infrared light emitted from the infrared light source 12 is guided in the transparent plate 11 by a total reflection phenomenon. When the object B is brought into contact with the surface of the input panel 10 (transparent silicone rubber 13), infrared rays are scattered at the contact portion, and scattered light is generated. The infrared camera 30 images the infrared scattered light from the back side of the input panel 10.

  The projection screen 20 is a nearly transparent transmission film for rear projection. In the example shown in FIG. 2, the projection screen 20 is sandwiched between two transparent plates 11 (affixed to the surface of the lower transparent plate 11 and placed on the back side of the upper transparent plate 11. However, you may comprise so that the projection screen 20 may be affixed on the back surface of the transparent plate 11 of 1 sheet.

  FIG. 3 is an example of a functional block diagram of the image projection system 1 according to the first embodiment. The image projection system 1 includes a storage unit 110. The storage unit 110 stores a program and various data for causing the computer to function as each unit of the processing unit 100, and also functions as a work area of the processing unit 100. The function can be realized by a hard disk, a RAM, or the like.

  The processing unit 100 performs processing for controlling the infrared camera 30, the projector 40, and the visible light camera 50, and various image processing. The functions of the processing unit 100 can be realized by hardware such as various processors (CPU, DSP, etc.), ASIC (gate array, etc.), and programs. The processing unit 100 includes an original image generation unit 101 that generates an original image for projection, a mask image generation unit 102 that generates a mask image by binarizing an infrared image captured by the infrared camera 30, and an original image and a mask. A composite image generation unit 103 that generates a composite image by combining the images, a projection control unit 104 that transmits the composite image to the projector 40 for projection, and controls the visible light camera 50 after the composite image has been projected to capture an image. An imaging control unit 105 to be performed, and a contact surface image generation unit 106 that combines the visible light image captured by the visible light camera 50 and the mask image to generate a contact surface image.

  In the image projection system 1 of this embodiment, the input panel 10 is imaged from the back side by the infrared camera 30, so that the shape of the contact surface of the object B in contact with the surface of the input panel 10 (the object B and the input panel 10 (Contact shape) and the input panel 10 is imaged from the back side by the visible light camera 50, thereby obtaining the color and pattern of the contact surface. However, since the projection screen 20 is disposed on the back side of the input panel 10, as shown in FIG. 4, when the image is directly projected onto the projection screen 20 by the projector 40, the projection image PP (projection light) is the object B. By superimposing on the contact surface CF, the appearance of the contact surface CF changes, and the original color and pattern of the contact surface CF cannot be acquired.

  Such a competition between projection and imaging can be avoided by alternately switching between projection by the projector 40 and imaging by the visible light camera 50, for example. However, in consideration of the burden of blinking projection light on the user's eyes and the effect on the visual process in the brain, it is necessary to switch between projection and imaging at high speed (70 to 80 Hz or more). Therefore, when this method is adopted, it is necessary to increase the speed of each device (the projector 40, the visible light camera 50, and the processing unit 100), resulting in an increase in cost.

  In the image projection system 1 of the present embodiment, an image is projected such that the contact surface CF of the object B (the contact portion between the object B and the input panel 10) is monochromatic when viewed from the visible light camera 50 during image projection. Resolve the conflict between projection and imaging.

  FIG. 5 is a flowchart showing the flow of processing of the image projection system 1 in the first embodiment.

  First, the original image generation unit 101 generates an original image FP for projection (step S10). FIG. 6A schematically shows an example of the original image FP.

  Next, the processing unit 100 controls the infrared camera 30 to capture an image and acquires an infrared image captured by the infrared camera 30 (step S11). Next, the mask image generation unit 102 binarizes the acquired infrared image to generate a mask image MP (step S12). Since the binarized image may contain noise, it is desirable to generate a mask image MP by performing noise removal processing on the binarized image. FIG. 6B schematically shows an example of the mask image MP. In the mask image MP, the area CA is an area corresponding to the contact surface CF of the object B, the pixel value of the area CA is 1, and the pixel values of the other areas are 0.

Next, the mask image generation unit 102 synthesizes the mask image MP and the correction image RP (for example, multiplies each corresponding pixel), corrects the mask image MP, and obtains a mask image MP ′ (step S13). FIG. 6C schematically shows an example of the correction image RP. The correction image RP is an image that is uniformly monochromatic when viewed from the visible light camera 50 when the image is projected onto the projection screen 20. The luminance of each pixel of the correction image RP is determined by the intensity of reflected light from the projection screen 20 (the positional relationship between the projector 40 and the projection screen 20 may be substituted). For example, the mask image generation unit 102 sets a portion of the correction image RP projected at a position close to the projector 40 (a strong reflection of projection light) as a color with low luminance (for example, dark gray) and is far from the projector 40. The portion projected at the position (the reflection of the projection light is weak) generates an image having a high luminance color (for example, light gray). Further, the color of the correction image RP is a color in which the appearance (color) of the contact surface CF does not easily change during projection, for example, a color other than white (luminance value is the maximum value) or black (luminance value is 0). The color is low in saturation (for example, gray). The optimum color of the correction image RP can be determined from the color characteristics of the projector 40 and the visible light camera 50, and the like. FIG. 6D schematically shows the corrected mask image MP ′. In the corrected mask image MP ′, the correction image RP is superimposed on the area CA corresponding to the contact surface CF in the mask image MP.

  Next, the synthesized image generation unit 103 synthesizes the original image FP and the corrected mask image MP ′ (for example, the pixel values of the corresponding pixels are compared and the pixel value of the mask image MP ′ is 0 (black)). The pixel value of the original image FP is adopted for a certain pixel, and the pixel value of the mask image MP ′ is a pixel value other than 0 (the pixel value of the area CA is adopted the pixel value of the mask image MP ′). SP is generated (step S14). FIG. 6E schematically shows the composite image SP. In the composite image SP, the correction image RP is superimposed on the area CA corresponding to the contact surface CF in the original image FP. Therefore, the composite image SP is an image in which the appearance of the contact surface CF (the appearance from the visible light camera 50) becomes monochromatic at the time of projection.

  Next, the projection control unit 104 performs control to transmit the composite image SP to the projector 40 and project the composite image SP onto the projection screen 20 (step S15). FIG. 7 schematically shows a projection image PP when the composite image SP is projected. As shown in FIG. 7, according to the present embodiment, by projecting the composite image SP such that the appearance of the contact surface CF becomes monochromatic at the time of projection, the influence of the projection image PP on the appearance of the contact surface CF is affected. It can be suppressed as much as possible.

  Next, the imaging control unit 105 controls the visible light camera 50 to capture an image, and obtains a visible light image OP captured by the visible light camera 50 (step S16). FIG. 8A schematically shows the visible light image OP.

  Next, the contact surface image generation unit 106 combines the visible light image OP and the mask image MP to generate a contact surface image CP obtained by extracting an image of the area CA corresponding to the contact surface CF from the visible light image OP ( Step S17). FIG. 8B schematically shows the contact surface image CP. The color or pattern of the area CA in the contact surface image CP represents the color or pattern of the contact surface CF of the object B. Thus, according to the present embodiment, it is possible to acquire the color and pattern of the contact surface CF of the object B by eliminating the influence of the projection light of the projector 40.

  Next, it transfers to step S10 and repeats the process after step S10. In step S10, the original image generation unit 101 generates a new original image FP based on the contact surface image CP generated in step S17 of the previous frame (processing unit time).

  For example, the original image generation unit 101 uses the area CA in the contact surface image CP generated in the previous frame (and the area CA in the contact surface image CP generated before that) as the color of the contact surface image CP (area CA) or By drawing with a pattern, an original image FP in which the movement locus of the object B on the input panel 10 is drawn with the color or pattern of the object B is generated. Thereby, not only the contact shape of the object B in contact with the input panel 10 but also digital paint that can be drawn using the color or pattern of the contact surface of the object B can be realized.

In the case of the example shown in FIG. 9, at time t1, an original image FP _{1 in} which the area CA ₀ in the contact surface image CP generated at time t0 (previous frame) is drawn with the color or pattern of the area CA ₀ is generated, and time t1 A composite image SP ₁ is generated by masking the area CA ₁ in FIG. Further, at time t2, an original image FP _{2 in} which the movement trajectory TR of the object B including the area CA ₁ at time t1 is drawn with the color or pattern of the area CA ₁ is generated, and a composite image in which the area CA ₂ at time t2 is masked. SP ₂ is generated. Similarly, at time t3, to generate the original image FP ₃ the movement trajectory TR was drawn in the color or pattern of the area CA ₂ including a region CA ₂ at time t2, the composite image SP ₃ masked regions CA ₃ at time t3 Is generated. In this way, the movement trajectory TR of the object B can be drawn with the color or pattern of the object B.

When the contact surface of the object B is provided with a pattern (for example, a barcode, a two-dimensional symbol, or a character string) obtained by encoding the identification information of the object B, the processing unit 100 displays the contact surface image CP ( The object B is identified by performing image recognition on the image of the area CA, and the original image generation unit 101 is an element for projecting an image related to the identified object B on the input panel 10 in the vicinity of the object B. You may comprise so that the image FP may be produced | generated.

  Here, as illustrated in FIG. 10A, the imaging control unit 105 starts imaging with respect to the visible light camera 50 in a state where the projection of the composite image SP (update of the projection screen) by the projector 40 has been completed. Need to be instructed.

  As shown in FIG. 10B, if imaging by the visible light camera 50 is started before the projection of the composite image SP by the projector 40 is completed, the composite image SP generated and projected in the previous frame is started. And a composite image SP generated / projected in the current frame (a projected image being updated) are captured, and the influence of the projection light is not excluded (superimposition of the projected image and the contact surface CF) In other words, the visible light image OP picked up in the state where the image is generated) is acquired.

  Hereinafter, three methods for performing imaging by the visible light camera 50 after the projection of the composite image SP by the projector 40 is completed will be described.

  In the first method, when the projection of the composite image SP is completed, the projector 40 transmits notification information to that effect to the processing unit 100, and the imaging control unit 105 receives this communication information. Then, the visible light camera 50 is controlled to start imaging. According to the first method, it is possible to reliably prevent the visible light camera 50 from starting imaging before the projection of the composite image SP is completed, and the visible light camera 50 after the projection of the composite image SP is completed. Shooting can be performed without delay.

  In the second method, the imaging control unit 105 sets the area CA in the synthesized image SP transmitted to the projector 40 and the visible light image OP received from the visible light camera 50 to the same color as the area CA in the synthesized image SP. A difference (total value of difference absolute values for each pixel) is calculated by comparing the visible light image OP and the corrected mask image MP ′, and the calculated difference is less than a predetermined threshold δ (allowable) The imaging instruction to the visible light camera 50 is repeatedly performed until it is within the range. In the example shown in FIG. 11, as a result of comparing the visible light image OP acquired by the first imaging with the composite image SP, the difference is equal to or greater than the threshold value δ. Yes. Then, as a result of comparing the visible light image OP acquired by the second imaging with the composite image SP, the difference is less than the threshold δ, so the visible light image OP is an image captured after the projection of the composite image SP is completed. It is determined that the process proceeds to the next step.

  In the third method, the total time (predetermined time) of the time required to transfer the composite image SP to the projector 40 and the time required to update the projection screen is estimated by a preliminary survey, and the imaging control unit 105 performs the composite image SP. Is transmitted to the projector 40, and after the total time estimated in advance has elapsed (after the time when it is estimated that the update of the projection screen has been completed), the visible light camera 50 is controlled to start imaging.

  According to the second method and the third method, the projection of the composite image SP is completed even when the projector 40 does not have a notification function or when the processing unit 100 does not have a function of receiving a notification. It is possible to prevent the imaging by the visible light camera 50 from being started before performing.

(Second Embodiment)
Depending on the material of the object B and how the user U touches the input panel 10 with the object B,
Due to the incomplete contact between the object B and the input panel 10, information on the contact surface CF of the object B (area CA in the mask image MP) is acquired even though the object B is in contact with the input panel 10. There are cases where it is not possible. In this case, if the threshold value for binarizing the infrared image (generating the mask image MP) is lowered, information on the contact surface CF can be acquired. However, even when the object B is not in contact with the input panel 10, if the infrared image is always binarized with a low threshold value to generate the mask image MP, it leaks from the surface of the input panel 10 (or is totally reflected). In the case where infrared rays that have not entered the corner are reflected on an object that is not in contact with the contact surface CF (for example, the hand or arm of the user U holding the object B) and reflected on the input panel 10. There is a possibility that reflection or the like may be erroneously detected as the contact surface CF.

Therefore, in the second embodiment, the contact of the object B with the input panel 10 is detected by a pressure sensor (pressure sensor), and when the contact of the object B with the input panel 10 is detected, an infrared image is acquired, the obtained infrared image by binarizing the first thresholds T ₁ and generates a mask image MP. If the number of white pixels in the generated mask image MP is less than a predetermined value, the acquired infrared image is binarized with a second threshold T ₂ that is less than the _first threshold T ₁ and the mask image MP is obtained. Generate again. That is, in the second embodiment, when it can be considered that the information on the contact surface CF could not be acquired even though the object B is in contact with the input panel 10, the threshold value for binarizing the infrared image Is reduced to generate a mask image MP.

  FIG. 12 is a side view illustrating an example of the configuration of the input panel 10 according to the second embodiment. 12, the same components as those shown in FIG. 2 are denoted by the same reference numerals, and the description thereof is omitted as appropriate.

  In the second embodiment, a pressure sensor 60 for detecting contact of the object B with the input panel 10 is provided between the upper transparent plate 11 and the projection screen 20. As the pressure sensor 60, for example, a resistive film force sensor can be used. A silicone rubber 14 that functions as a protrusion is provided between the upper transparent plate 11 and the pressure sensor 60. The pressure sensor 60 and the silicone rubber 14 are provided at the four corners in the plan view of the input panel 10.

  FIG. 13 is a flowchart showing the flow of processing of the image projection system 1 in the second embodiment. Note that steps S20 and S27 to S31 shown in FIG. 13 are the same as steps S10 and S13 to S17 shown in FIG.

  The processing unit 100 determines whether or not the object B is in contact with the input panel 10 based on an output signal from the pressure sensor 60 (an average value of output signals from the plurality of pressure sensors 60) (step S100). S21) When the object B is not in contact with the input panel 10 (N in step S21), the process proceeds to step S20. On the other hand, when the object B is in contact with the input panel 10 (Y in step S21), the processing unit 100 controls the infrared camera 30 to capture an image, and the infrared image captured by the infrared camera 30 is captured. Obtain (step S22).

Then, the mask image generation unit 102, by binarizing the infrared image obtained by the first thresholds _{T 1} generates a mask image MP (step S23). Next, the mask image generation unit 102 counts the number S of white pixels (pixels having a pixel value of 1) in the mask image MP generated in step S23 (step S24). Here, an image obtained by binarizing an infrared image captured in a state where the object B is not in contact with the input panel 10 is prepared as a background image, and the background image and the mask image MP are compared for each pixel. Thus, the number of pixels having a higher pixel value of the mask image MP than the background image may be counted as the number S of white pixels.

Next, the mask image generation unit 102 determines whether or not the counted number S of white pixels is less than the predetermined value P (step S25), and when the number S of white pixels is equal to or greater than the predetermined value P (step S25). In N) of S25, the process proceeds to step S27. When the number S of white pixels is less than the predetermined value P (Y in step S25), the mask image generation unit 102 uses the acquired infrared image as a binary value with a second threshold T ₂ (T ₂ <T ₁ ). To generate a mask image MP (step S26).

Next, in the case of false in step S25, the mask image generation unit 102 combines the mask image MP generated in step S23 (binarized with the _first threshold value T1) and the correction image RP to generate a mask. image MP 'generates, in the case of true in step S25, generated at step S26 (second binarized by the threshold T ₂₎ mask image MP by synthesizing the correction image RP and the mask image MP' Is generated (step S27).

FIG. 14 shows an example of an infrared image IP and a mask image MP obtained by binarizing the infrared image IP. The mask image MP ₁ shown in FIG. 14 is an image obtained by binarizing the infrared image IP with the first threshold value T _{1. The} mask image MP ₂ is obtained by binarizing the infrared image IP with the second threshold value T ₂ . It is an image. Here, the gradation of the infrared image IP is 8 bits, the first threshold T ₁ = 253, and the second threshold T ₂ = 175. The infrared image IP is an image captured when the user U touches the input panel 10 with the sponge-like object B. Since the close contact between the object B and the input panel 10 is insufficient, the infrared image IP is The mask image MP ₁ binarized with the _first threshold value T ₁ has almost no white pixel area (area CA). In this case, in the second embodiment, it is determined that the number S of white pixels in the mask image MP ₁ is less than the predetermined value P, and the mask image MP _{2 obtained} by binarizing the infrared image IP with the second threshold T ₂ is used. Is generated. Here, the number of pixels of the infrared image IP (and mask image MP) is 720 × 720 pixels, and the predetermined value P = 50. The mask image MP _2, it can be seen that the area of white pixels corresponding to the contact surface CF of the object B are present, and to obtain information of the contact surface CF.

As described above, according to the second embodiment, when the contact of the object B with the input panel 10 is detected, the acquired infrared image IP is binarized with the _first threshold value T ₁ to generate the mask image MP ₁ . and, if the number S of white pixels in the mask image MP ₁ is not less than the predetermined value P, it performs the subsequent process using the mask image MP _1, if the number S of white pixels is less than the predetermined value P Does not touch the input panel 10 by performing the subsequent processing using the mask image MP _{2 obtained} by binarizing the infrared image IP with the second threshold T ₂ lower than the first threshold T _1. To accurately and reliably acquire the shape, color, and pattern of the contact surface CF of the object B that is incompletely in close contact with the input panel 10 while preventing the B reflection from being erroneously detected as the contact surface CF. Can do.

  In addition, this invention is not limited to the above-mentioned embodiment, A various change is possible. The present invention includes configurations that are substantially the same as the configurations described in the embodiments (for example, configurations that have the same functions, methods, and results, or configurations that have the same objects and effects). In addition, the invention includes a configuration in which a non-essential part of the configuration described in the embodiment is replaced. In addition, the present invention includes a configuration that exhibits the same operational effects as the configuration described in the embodiment or a configuration that can achieve the same object. Further, the invention includes a configuration in which a known technique is added to the configuration described in the embodiment.

DESCRIPTION OF SYMBOLS 1 ... Image projection system, 10 ... Input panel, 11 ... Transparent board, 12 ... Infrared light source, 13 ... Transparent silicone rubber, 14 ... Silicone rubber, 20 ... Projection screen, 30 ... Infrared camera, 40 ... Projector, 50 ... Visible light Camera ... 60 ... Pressure sensor 100 ... Processing unit 101 ... Original image generation unit 102 ... Mask image generation unit 103 ... Composite image generation unit 104 ... Projection control unit 105 ... Imaging control unit 106 ... Contact surface image Generation unit, 110 ... storage unit

Claims (8)

An input panel for guiding infrared light from the light source to the inside;
A projection screen disposed on the back side of the input panel;
An infrared camera that images from the back side the scattered light generated by an object in contact with the surface of the input panel;
A projector that projects an image onto the projection screen from the back side;
A visible light camera that images the input panel from the back side;
An image projection system comprising a processing unit,
The processor is
An original image generation unit for generating an original image for projection;
A mask image generation unit that generates a mask image by binarizing an infrared image captured by the infrared camera;
A composite image generating unit that generates a composite image by combining the original image and the mask image;
A projection control unit for transmitting the composite image to the projector and projecting it;
An imaging control unit that causes the visible light camera to perform imaging after the projection of the composite image is completed;
A visible image captured by the visible light camera and the mask image are combined to generate a contact surface image obtained by extracting an image of an area corresponding to a contact portion between the object and the input panel from the visible light image. A contact surface image generation unit that
The original image generation unit
Generating the original image based on the contact surface image;
The composite image generation unit
An image projection system for generating the composite image such that the contact portion is monochromatic when viewed from the visible light camera when the composite image is projected. In claim 1,
The projector is
When the projection of the composite image is completed, notification information is transmitted to the processing unit,
The imaging control unit
An image projection system that causes the visible light camera to perform imaging when the notification information is received. In claim 1,
The imaging control unit
A difference is calculated by comparing an image corresponding to the contact portion in the visible light image with the composite image and an image in the same color as the region in the composite image, and the calculated difference is within an allowable range. An image projection system that repeatedly performs imaging by the visible light camera. In claim 1,
The imaging control unit
An image projection system that causes the visible light camera to perform imaging after a predetermined time has elapsed since the composite image was transmitted to the projector. In any one of Claims 1 thru | or 4,
The original image generation unit
The image projection system which produces | generates the image which drawn the movement locus | trajectory of the said object which contacted the said input panel using the color or pattern of the said contact surface image as said original image. In any one of Claims 1 thru | or 5,
A pressure sensor for detecting contact of the object with the input panel;
The mask image generation unit
When the contact of the object with the input panel is detected, the infrared image is binarized with a first threshold value to generate a mask image, and the number of white pixels in the generated mask image is less than a predetermined value In addition, the image projection system generates a mask image again by binarizing the infrared image with a second threshold value less than the first threshold value. An input panel that guides infrared light from the light source to the inside, a projection screen that is arranged on the back side of the input panel, and scattered light generated by an object that contacts the surface of the input panel from the back side A program for an image projection system comprising: an infrared camera for imaging; a projector that projects an image on the projection screen from the back side; and a visible light camera that images the input panel from the back side;
An original image generation unit for generating an original image for projection;
A mask image generation unit that generates a mask image by binarizing an infrared image captured by the infrared camera;
A composite image generating unit that generates a composite image by combining the original image and the mask image;
A projection control unit for transmitting the composite image to the projector and projecting it;
An imaging control unit that causes the visible light camera to perform imaging after the projection of the composite image is completed;
A visible image captured by the visible light camera and the mask image are combined to generate a contact surface image obtained by extracting an image of an area corresponding to a contact portion between the object and the input panel from the visible light image. The computer functions as a contact surface image generation unit that performs
The original image generation unit
Generating the original image based on the contact surface image;
The composite image generation unit
The program which produces | generates the said synthetic | combination image that the said contact part becomes single color seeing from the said visible light camera when the said synthetic | combination image is projected. An input panel that guides infrared light from the light source to the inside, a projection screen that is arranged on the back side of the input panel, and scattered light generated by an object that contacts the surface of the input panel from the back side An image projection method executed by an image projection system comprising: an infrared camera for imaging; a projector for projecting an image on the projection screen from the back side; and a visible light camera for imaging the input panel from the back side. ,
An original image generating step for generating an original image for projection;
A mask image generation step of generating a mask image by binarizing an infrared image captured by the infrared camera;
A composite image generation step of generating a composite image by combining the original image and the mask image;
A projection control step of transmitting the composite image to the projector and projecting it;
An imaging control step for causing the visible light camera to perform imaging after completion of the projection of the composite image;
A visible image captured by the visible light camera and the mask image are combined to generate a contact surface image obtained by extracting an image of an area corresponding to a contact portion between the object and the input panel from the visible light image. And a contact surface image generating step to perform,
In the original image generation step,
Generating the original image based on the contact surface image;
In the composite image generation step,
An image projecting method for generating the composite image such that the contact portion is monochromatic when viewed from the visible light camera when the composite image is projected.