JP2012226507A - Emitter identification device, emitter identification method and emitter identification program - Google Patents

Abstract

PROBLEM TO BE SOLVED: To define the emission of an emitter as input information, and to identify the input information with a simple procedure on the basis of the emission color of the emitter.SOLUTION: An emitter identification device 100 includes: an image acquisition part 112 for acquiring an IR image and a color image respectively obtained by simultaneously imaging the same object by an IR camera 150 and a color camera 152; a position information acquisition part 114 for detecting a first pixel whose luminance value is equal to or more than a predetermined threshold as the existing position of the emitter from the IR image, and for acquiring first position information showing the existing position; an emission color determination area selection part 116 for selecting the emission color determination area of the emitter including a second pixel at a position corresponding to the first position from the color image on the basis of the first position information acquired by the position information acquisition part 114; and an emission color determination part 118 for determining the emission color of the emitter on the basis of the color component information of the pixel included in the emission color determination area of the color image.

Description

  The present invention relates to a light emitter identification device, a light emitter identification method, and a light emitter identification program.

  In recent years, a technique has been developed in which a user draws on an electronic blackboard using a simulated pen, acquires the pen trajectory, and displays image data using a projector or the like.

  For example, Patent Document 1 (Japanese Patent Application Laid-Open No. 2002-079792) discloses a vibration transmission plate that transmits elastic wave vibration, vibration input means (vibration input pen) that inputs elastic wave vibration to the vibration transmission plate, and vibration. Coordinate positions on the vibration transmission plate of the vibration input means according to the vibration detection means provided on the outer edge of the transmission plate and the time for the elastic wave vibration from the vibration input means to reach the vibration detection means. There is described an electronic blackboard device provided with a coordinate calculating means for calculating. Here, it is described that the vibration input pen functions as a marker pen that can draw with ink.

  Japanese Patent Laid-Open No. 2005-135081 discloses a light pen used for a screen that acquires the coordinates of a pointer by capturing an image with an image sensor, and a light emitting means provided at the tip of the pointer, and the pointer. There is described a light pen that includes a pressure detection means for detecting that the tip of the light is being pressed by the screen, and a light emission control means for controlling the light emission means. Thus, the operation of the light pen can be known by the light receiving element, and image data can be generated for the projector.

  In addition, a controller is used that is operated by a user with a hand when playing a game. For example, Patent Document 3 (Japanese Patent Application Laid-Open No. 2007-313354) describes a configuration of a controller (game operation device) for the purpose of stable operation with one hand. Here, an example is described in which a game machine and a controller perform wireless communication by Bluetooth. In the technique described in the document, a plurality of controllers can be connected to the game machine by configuring the controller and the game machine to store each other's Bluetooth address in advance by initial setting or the like.

JP 2002-079792 A Japanese Patent Laying-Open No. 2005-135081 JP 2007-313354 A

  However, conventionally, when a plurality of users use input means such as a pen and a controller, there has been no method for distinguishing the plurality of input means by a simple procedure. For example, as described in Patent Document 3, it is possible to distinguish each input unit by using a configuration in which a Bluetooth address is stored in advance using Bluetooth wireless communication. However, in such a method, it is necessary to provide a wireless communication function on the input means side, and the configuration becomes complicated.

  The present invention has been made in view of the above circumstances, and an object of the present invention is to provide a technique for identifying input information in a simple procedure based on light emission color of a light emitter, using light emission of the light emitter as input information. There is.

According to the present invention,
An image acquisition unit for acquiring an IR image and a color image obtained by simultaneously imaging the same object with an IR camera and a color camera;
A position information acquisition unit that detects, from the IR image, a first pixel having a luminance value equal to or greater than a predetermined threshold as the presence position of the light emitter, and acquires first position information indicating the presence position;
Based on the first position information acquired by the position information acquisition unit, a light emission color determination for selecting a light emission color determination region of a light emitter including a second pixel at a position corresponding to the first position from the color image. An area selector;
A light emission color determination unit that determines a light emission color of the light emitter based on color component information of pixels included in the light emission color determination region of the color image selected by the light emission color determination region selection unit;
A light emitter identification device is provided.

According to the present invention,
Obtaining an IR image and a color image respectively obtained by simultaneously imaging the same object with an IR camera and a color camera;
Detecting, from the IR image, a first pixel having a luminance value equal to or greater than a predetermined threshold as an existing position of the light emitter, and obtaining first position information indicating the existing position;
Based on the first position information acquired in the step of acquiring the first position information, an emission color determination region of a light emitter including a second pixel at a position corresponding to the first position from the color image. A process of selecting
Determining a light emission color of the light emitter based on color component information of pixels included in the light emission color determination region of the color image selected in the step of selecting the light emission color determination region;
A light emitter identification method is provided.

  The IR image obtained by capturing the landscape including the illuminant with an IR camera acquires only near-infrared light. Therefore, the influence of ambient light such as illumination can be reduced, and the color image obtained with the color camera can be reduced. As compared with the above, noise can be reduced, and it is easy to determine the difference between the light emitter and other portions based on the luminance value of the pixel. According to the illuminant identification device and the illuminant identification method of the present invention, since the location of the illuminant is first detected from the IR image, the illuminant can be detected accurately and quickly. Thereafter, since the emission color determination process is performed based on the color component information at the corresponding position of the color image, the emission color determination of the light emitter can be easily performed with high accuracy. According to such a configuration, for example, when a plurality of light emitters having different light emission colors are prepared, the light emission color of each light emitter is determined, and the process of identifying each light emitter based on the light emission color is easily performed. Can do. Therefore, when the light emitter is used as an input device, a plurality of input devices can be easily identified.

  In the light emitter identification device of the present invention, the light emission color determination unit includes a second position indicating a light emission color of the light emitter and first position information of the IR image or a position of the second pixel of the color image. Information can be output in association with the information. In this way, the luminescent color for identifying the illuminant can be obtained together with the first position information or the second position information specified by the illuminant, so that, for example, output from the illuminant identified at the corresponding position on the display Can be done.

  Also, the light emitter identification device of the present invention is an input that is identified by the light emission color based on the light emission color of the light emitter output from the light emission color determination unit and the first position information or the second position information. An input receiving unit that receives an input from the device to the first position information or the second position information can be further included. In this way, input information from the input device can be received based on the position information of the light emitter, and the input device can be identified according to the light emission color determined from the color component information of the color image. Therefore, it is possible to identify input information input by a plurality of users using a plurality of input devices each including a light-emitting body having a different emission color.

In the light emitter identification device of the present invention,
The light emission color determination region selection unit determines whether or not a luminance value Yi of a pixel included in the selected light emission color determination region is equal to or greater than a predetermined necessary luminance value Yd, and the luminance value Yi is greater than the necessary luminance value Yd. When the luminance value Yi is not less than noise when the luminance value Yi is equal to or greater than the necessary luminance value Yd, it is possible to determine that the emission color determination region is not noise.
The light emission color determination unit, when the light emission color determination region selection unit determines that the light emission color determination region is not noise, based on the color component information of the pixels included in the light emission color determination region of the color image, The emission color of the light emitter can be determined,
The required luminance value Yd can be set such that the larger the distance Di is, the lower the value is, according to the distance Di between the center of the color image and the emission color determination area.

  When the illuminant is near the extension of the central axis of the camera lens and the incident angle to the camera lens is small, the luminance value is relatively large, but the illuminant is far from the extension of the central axis of the camera lens. If the incident angle to the camera lens is large, the luminance value becomes small. According to the above configuration, the required luminance value Yd used for the determination in the noise determination process of the emission color determination region of the color image has a large distance Di according to the distance Di between the center of the color image and the emission color determination region. On the other hand, that is, when the incident angle to the camera lens is large, the value is set to be low, so that noise is accurately removed in consideration of the characteristic of the luminance value at the place where the illuminant is detected. be able to.

  According to the present invention, there is provided a light emitter identification system including the light emitter identification device, a plurality of light emitters having different emission colors, an IR camera that images the light emitter, and a color camera.

  It should be noted that any combination of the above-described constituent elements and a conversion of the expression of the present invention between a method, an apparatus, a system, a recording medium, a computer program, etc. are also effective as an aspect of the present invention.

  According to the present invention, it is possible to identify input information in a simple procedure based on the light emission color of the light emitter using light emission of the light emitter as input information.

It is a figure which shows typically an example of a structure of the light-emitting body identification system in embodiment of this invention. It is a figure which shows typically an example of a structure of the input device in embodiment of this invention. It is a figure which shows the state where the locus | trajectory drawn with the input device in embodiment of this invention was output from the light-emitting body identification device, and was displayed on the drawing board. It is a block diagram which shows an example of a structure of the light-emitting body identification device in embodiment of this invention. It is a figure which shows an example of a structure of a corresponding | compatible memory | storage part. It is a flowchart which shows the process sequence of the light-emitting body identification device in embodiment of this invention. It is a schematic diagram for demonstrating the process by the dark imaging condition determination part in embodiment of this invention. It is a schematic diagram for demonstrating the process by the dark imaging condition determination part in embodiment of this invention. It is a flowchart which shows the process sequence of step S18 of FIG. 6 in detail. It is a figure which shows typically an example of the process to IR image and a color image. It is a figure which shows typically the other example of the process to IR image and a color image. It is a flowchart which shows the process sequence of step S20 of FIG. 6 in detail. It is a flowchart which shows an example of the process sequence of step S130 of FIG. 12 in detail. It is a flowchart which shows an example of the process sequence of FIG.12 S134 in detail. It is a flowchart which shows in detail an example of the process sequence of step S136 of FIG. It is a figure which shows an example of a structure of a setting value memory | storage part. It is a figure which shows an example of the screen which is projected on the projection film for projectors at the time of a pre-processing, and is displayed for the user who exists in the drawing surface side of a drawing board. It is a figure which shows an example of the screen shown to the user who is a setting person in the setting procedure of shutter speed. It is a flowchart which shows the modification of the process of step S16 to step S20 of the flowchart shown in FIG.

  Hereinafter, embodiments of the present invention will be described with reference to the drawings. In all the drawings, the same reference numerals are given to the same components, and the description will be omitted as appropriate.

First, the configuration of the light emitter identification system 10 according to the present embodiment will be briefly described. FIG. 1 is a diagram schematically illustrating an example of a configuration of a light emitter identification system 10 according to the present embodiment.
The light emitter identification system 10 includes a drawing board 160 supported by a support stand 162, an imaging unit 148, a projector 154, a plurality of input devices 170, and a light emitter identification device 100. FIG. 1A is a plan view of the drawing board 160 viewed from the drawing surface on which the user draws with the input device 170, and FIG. 1B is a side view of the drawing board 160.

  In the present embodiment, the imaging unit 148 may include an IR camera 150 (infrared camera) and a color camera 152. Here, the IR camera 150 and the color camera 152 each acquire a moving image. The IR camera 150 captures an object with infrared rays to obtain an IR image. The color camera 152 captures an object with visible light and obtains a color image. In the present embodiment, the IR image captured by the IR camera 150 is used to detect the position information (coordinates) of the light emitter. In the present embodiment, the color image captured by the color camera 152 is used to determine the emission color of the light emitter.

  Further, the IR camera 150 and the color camera 152 are arranged so that the same object can be imaged substantially simultaneously. That is, the IR camera 150 and the color camera 152 are arranged so as to face substantially the same direction. For example, the IR camera 150 and the color camera 152 can be arranged vertically and the IR camera 150 and the color camera 152 can be arranged side by side.

  In the present embodiment, the IR camera 150 and the color camera 152 are installed at a distance from the drawing board 160, and the drawing board 160 is within the imageable area, and drawing is performed on almost the entire area. The board 160 can be arranged to be imaged. For example, the IR camera 150 and the color camera 152 can be arranged at a position overlapping the center of the drawing board 160. As a result, the object on the extension of the central axis of the lens of the IR camera 150 or the color camera 152 is captured at the center of the image acquired by the IR camera 150 or the color camera 152.

  The IR camera 150 can be configured by cutting visible light with an optical filter, for example. The IR camera 150 can be configured to cut visible light of 920 nm or less, for example. The color camera 152 can be configured by cutting near infrared rays and infrared rays with an optical filter, for example. However, the IR camera 150 and the color camera 152 are configured to detect a light emitter such as an LED (Light Emitting Diode) used as the light emitter 172 of the input device 170.

  In the present embodiment, the IR camera 150 and the color camera 152 can be prepared by providing optical filters having different wavelengths of light to be cut using, for example, cameras having the same configuration. As an example, each of the IR camera 150 and the color camera 152 can use a VGA (Video Graphics Array) of 640 × 480 pixels, an XGA of 1280 × 960 pixels, or the like.

  In the present embodiment, the drawing board 160 can be made of a transparent acrylic plate, for example. Further, a projector projection film 164 for displaying data projected from the projector 154 can be provided on the surface opposite to the drawing surface of the drawing board 160. Data output from the light emitter identification device 100 is projected from the projector 154 onto the projector projection film 164 and displayed to the user on the drawing surface side of the drawing board 160. In the present embodiment, the user performs various input operations using the input device 170 while viewing the screen projected onto the projector projection film 164.

  FIG. 2 is a diagram schematically illustrating an example of the configuration of the input device 170. The input device 170 may have a configuration in which a light emitter 172 is attached to the tip. In addition, the input device 170 is provided with a switch 174 that controls on / off of light emission of the light emitter 172. Here, the switch 174 can be configured such that, for example, the light emitter 172 emits light only while the user presses the switch 174. As a result, when the user momentarily presses the switch 174 of the input device 170, the light emitter 172 can be caused to emit light instantaneously and can be used in the same manner as a mouse having a click button. In addition, the switch 174 can be configured to be switchable to a mode in which the light emitter 172 emits light continuously. As a result, the input device 170 can be easily used as, for example, a pen for a user to draw characters and pictures. In this embodiment mode, the light emitter 172 can be formed of an LED. The configuration of the input device 170 is not limited to this. For example, the switch 174 is provided between the light emitter 172 and the main body, and the switch 174 is turned on by pressing the light emitter 172 of the input device 170 against the drawing board 160. As a configuration, the light emitter 172 can emit light. In this way, the user can cause the light emitter 172 to emit light simply by pressing the light emitter 172 of the input device 170 against the drawing board 160, and the user can smoothly perform the drawing operation using the input device 170.

  Returning to FIG. 1, the light emitter identification system 10 according to the present embodiment may be configured to use a plurality of input devices 170 each including light emitters having different emission colors. For example, in the present embodiment, the light emitter identification system 10 includes, as the input device 170, a white input device 170a in which the light emitter 172 is a white LED, a blue input device 170b in which the light emitter 172 is a blue LED, and a light emitter 172. The green input device 170c, which is a green LED, and the red input device 170d, in which the light emitter 172 is a red LED, can be used.

  In the present embodiment, the light emitter identification device 100 determines the light emission color of each light emitter 172 and identifies the input device 170 according to the light emission color. As a result, the light emitter identification device 100 can identify the input device 170 based on the emission color even when a plurality of users perform input operations using the input device 170, respectively. Can be distinguished and understood.

  In such a state, the user presses the light emitter 172 of the input device 170 against the drawing board 160 and draws characters, figures, or the like while the light emitter 172 emits light, or blinks the light emitter 172. . The illuminant identification device 100 acquires the presence and position information of the illuminant 172 as an input operation based on the luminance value from an image obtained by imaging the illuminant 172 with the imaging unit 148. The light emitter identification device 100 outputs data controlled based on the detected input operation of the light emitter 172 to the projector 154. The data output from the projector 154 is projected onto the projector projection film 164 and displayed on the drawing board 160.

FIG. 3 is a diagram illustrating a state in which the locus 166 and the locus 168 drawn by the input device 170 are output from the light emitter identification device 100 and displayed on the drawing board 160.
Here, for example, the locus 166 is assumed to be a locus drawn by the white input device 170a, and the locus 168 is assumed to be a locus drawn by the blue input device 170b. In the present embodiment, the illuminant identification device 100 identifies these trajectories based on the emission color of the illuminant 172 of the white input device 170a and the emission color of the illuminant 172 of the blue input device 170b. Can be output.

FIG. 4 is a block diagram showing an example of the configuration of the illuminant identification device 100 in the present embodiment. The illuminator identification device 100 can be configured by a personal computer, for example.
The illuminant identification device 100 includes an IR image input unit 102, a color image input unit 104, a data output unit 106, an image acquisition unit 112, a position information acquisition unit 114, an emission color determination region selection unit 116, an emission color determination unit 118, a dark An imaging condition determination unit 120, an input reception unit 122, a program execution processing unit 124, a setting value storage unit 130, a light emission color determination information storage unit 132, a correspondence storage unit 134, and an image storage unit 136.

  The IR image input unit 102 is connected to the IR camera 150 via a connection interface such as a USB or a serial port, and inputs an IR image captured by the IR camera 150. The color image input unit 104 is connected to the color camera 152 via a connection interface such as a USB or a serial port, and inputs a color image captured by the color camera 152.

  The data output unit 106 outputs various data of the light emitter identification device 100 to an external display. In the present embodiment, the data output unit 106 is connected to the projector 154 via a connection interface such as an analog RGB connector (display port), and outputs data to the projector 154. Note that connection interfaces for image transmission / reception between the IR camera 150 and the IR image input unit 102, between the color camera 152 and the color image input unit 104, and between the data output unit 106 and the projector 154 are particularly limited. However, in another example, for example, it may be configured to perform wireless communication or the like.

  The image acquisition unit 112 acquires an IR image and a color image obtained by imaging the same object with the IR camera 150 and the color camera 152 substantially simultaneously. The image acquisition unit 112 acquires an IR image from the IR camera 150 via the IR image input unit 102 and a color image from the color camera 152 via the color image input unit 104. The IR image and color image acquired by the image acquisition unit 112 are stored in the image storage unit 136.

  The position information acquisition unit 114 detects the presence position of the light emitter 172 from the IR image acquired by the image acquisition unit 112. The light emission color determination region selection unit 116 is a light emission including a corresponding position of a color image acquired almost simultaneously with the IR image based on the position information of the position of the input device 170 detected by the position information acquisition unit 114 from the IR image. Select a color judgment area. The light emission color determination unit 118 determines the light emission color of the light emitter 172 based on the color component information in the light emission color determination region of the color image selected by the light emission color determination region selection unit 116. The dark imaging condition determination unit 120 determines imaging conditions for the IR camera 150 and the color camera 152.

  Based on the emission color of the illuminant 172 determined by the emission color determination unit 118 and its position information, the input reception unit 122 indicates that there has been an input operation to the position information from the input device 170 identified by the emission color. Accept.

  The correspondence storage unit 134 stores a correspondence between the emission color of the light emitter 172 of the input device 170 and the identification information of the input device. FIG. 5 is a diagram illustrating an example of the configuration of the correspondence storage unit 134. Here, the input device ID is associated with the light emission color of the light emitter 172. For example, the emission color “white” is the input device ID “1”, the emission color “blue” is the input device ID “2”, the emission color “green” is the input device ID “3”, and the emission color “red” is the input device ID. It is “4”. The input reception unit 122 accesses the correspondence storage unit 134 and acquires an input device ID corresponding to the emission color determined by the emission color determination unit 118. Based on the identified result, the input receiving unit 122 can output display data from the data output unit 106 so that a pointer or the like of the identified input device 170 is displayed at a corresponding position.

  The program execution processing unit 124 executes the program based on the input operation received by the input receiving unit 122.

  The setting value storage unit 130 is referred to when the position information acquisition unit 114, the emission color determination region selection unit 116, the dark imaging condition determination unit 120, and the like determine the luminance value of the image or perform other various determinations. The set value such as a reference value to be set and various settings are stored. The light emission color determination information storage unit 132 stores setting values such as a reference value and various settings that are referred to when the light emission color determination unit 118 determines the light emission color of the light emitter.

  Next, the processing procedure of the illuminant identification device 100 in the present embodiment will be described in detail. FIG. 6 is a flowchart showing a processing procedure of the illuminant identification device 100 in the present embodiment.

(Preprocessing: Darkening imaging condition determination)
First, as preprocessing, an image captured by the IR camera 150 and the color camera 152 in accordance with an environment (hereinafter referred to as an application environment) in which the illuminator identification device 100 is actually applied when receiving an input by light emission of the illuminant 172. Darkening imaging conditions for darkening the image are determined so that ambient light is not reflected on the screen (S4).

  When receiving an input by the light emission of the light emitter 172, not only the target light emitter 172 but also the data projected from the projector 154 to the projector projection film 164 in the image captured by the IR camera 150 and the color camera 152. When ambient light such as light is reflected, there is a high possibility that the ambient light is erroneously detected as light emission from the light emitter 172. Therefore, in the present embodiment, the pixel values of the images captured by the IR camera 150 and the color camera 152 when there is no light emission of the light emitter 172 in an application environment where ambient light is present are general image processing. Darkening imaging conditions for darkening the image are determined in advance so that the value determined to be black in the color determination. In the image captured by the IR camera 150 and the color camera 152 with the illuminant 172 emitted under the dark imaging conditions determined as described above, the pixel value other than the position where the illuminant 172 exists is black. Can be performed with high accuracy. The dark imaging condition can be, for example, the shutter speed, aperture amount, gain adjustment amount, or exposure amount of the IR camera 150 or the color camera 152. However, the luminous intensity of the illuminant 172 is sufficiently higher than the luminous intensity of the ambient light, and a certain luminance value is obtained even in an image captured under dark imaging conditions in which ambient light is not reflected in the images captured by the IR camera 150 and the color camera 152. And a pixel value such as an RGB value.

Hereinafter, a configuration for controlling the shutter speeds of the IR camera 150 and the color camera 152 as dark imaging conditions will be described as an example.
First, the procedure for determining the shutter speed of the color camera 152 will be described.
The illuminant identification system 10 is installed in an application environment where ambient light is present, and a state in which the target illuminant 172 does not emit light is sequentially imaged while appropriately changing the shutter speed of the color camera 152. Here, the shutter speed can be appropriately changed in the direction of increasing the speed. The image acquisition unit 112 acquires a first preprocessed image that is a color image captured by the color camera 152 via the color image input unit 104. The dark imaging condition determination unit 120 acquires the first preprocessed image acquired by the image acquisition unit 112 in association with the shutter speed of the color camera 152 when the image is captured. The shutter speed control unit 120 compares the pixel values at a plurality of positions included in the first preprocessed image with a predetermined value s ₁ for determining whether or not the pixel value is black. The predetermined value s ₁ can be stored in the setting value storage unit 130 in advance, and the dark imaging condition determination unit 120 accesses the setting value storage unit 130 and sets the pixel values at a plurality of positions to the predetermined value s _1. Compare with The predetermined value s _1, the color information of the ambient light is hardly state may be a value determined to be a black example, in color determination of the general image processing. The pixel value can be an RGB value. In this case, the predetermined value s ₁ can be set to a value at which all of the RGB values are substantially zero, for example, about 20 to 30 (when each of the RGB values takes a value of 0 to 255). The dark imaging condition determination unit 120 determines the first shutter speed at which the pixel values at a plurality of positions are all equal to or less than the predetermined value s ₁ as the dark imaging conditions of the color camera 152 in this application environment. Here, the plurality of positions included in the first preprocessed image include, for example, a central portion and a peripheral portion around the central portion.

Any of the pixel values at a plurality of positions included in the first preprocessed image obtained by imaging the state in which the target light emitter 172 does not emit light with the shutter speed of the color camera 152 being 1/125 seconds, for example, is predetermined. When the value is not less than ₁ , the shutter speed of the color camera 152 is further increased, for example, 1/250 seconds. Next, it is determined whether or not all the pixel values at a plurality of positions included in the first preprocessed image obtained by imaging with the shutter speed of the color camera 152 being 1/250 seconds are equal to or less than the predetermined value s _1. . In this way, the shutter speed at which all the pixel values at a plurality of positions are equal to or less than the predetermined value s ₁ is detected. Although the shutter speed may be changed continuously, for example, 1/60 seconds, 1/125 seconds, 1/250 seconds, 1/500 seconds, and the like, which are generally used as camera shutter speeds. You can change the speed step by step.

FIG. 7 is a diagram schematically showing a first preprocessed image 180 that is a color image of the drawing board 160 acquired by the image acquisition unit 112. Here, the image 182 of the drawing board 160 is displayed on almost the entire first preprocessed image 180. The darkening imaging condition determination unit 120 can acquire, for example, the respective RGB values of a total of nine locations from the center e of the image and the positions a to d of the periphery and a position i to f of the periphery. Further, the plurality of positions may be the center positions of the respective regions obtained by equally dividing the image 182, for example. For example, the plurality of positions may be the center positions of the respective regions obtained by dividing the image 182 into 9 equal parts of length × width 3 × 3. The RGB values at each position may be acquired sequentially, but may be acquired in substantially real time. The dark imaging condition determination unit 120 detects first shutter speeds that are equal to or less than the predetermined value s _{1 in} all positions a to i, and determines the first shutter speed as the dark imaging conditions.

Next, the procedure for determining the shutter speed of the IR camera 150 will be described.
This procedure can also be basically the same as the procedure for determining the shutter speed of the color camera 152. The illuminant identification system 10 is installed in an application environment where ambient light is present, and a state in which the target illuminant 172 does not emit light is sequentially imaged while appropriately changing the shutter speed of the IR camera 150. The image acquisition unit 112 acquires a second preprocessed image that is an IR image captured by the IR camera 150 via the IR image input unit 102. The dark imaging condition determination unit 120 acquires the second preprocessed image acquired by the image acquisition unit 112 in association with the shutter speed of the IR camera 150 when the image is captured. Shutter speed control unit 120, respectively the pixel values a plurality of positions of the pixel values included in the second pre-processing the image is compared with a predetermined value s ₂ for determining black or not. The predetermined value s ₂ can be stored in the setting value storage unit 130 in advance, and the dark imaging condition determination unit 120 accesses the setting value storage unit 130 and sets the pixel values at a plurality of positions to the predetermined value s _2. Compare with The predetermined value s ₂ can be a value that is determined to be black in a state in which there is almost no color information of ambient light, for example, color determination in general image processing. The pixel value can be an RGB value. In this case, the predetermined value s ₂ can be set to a value at which all of the RGB values are substantially zero, for example, about 25 to 40 (when each of the RGB values takes a value from 0 to 255). The dark imaging condition determining unit 120 determines the second shutter speed at which all the pixel values at a plurality of positions are equal to or less than the predetermined value s ₂ as the dark imaging condition of the IR camera 150 in this application environment. Here, the plurality of positions included in the second preprocessed image include, for example, a central portion and a peripheral portion around the central portion.
Note that the predetermined value s ₁ and the predetermined value s ₂ are not constant for each of the RGB values, and may be different values for each of the R value, the G value, and the B value. Further, the predetermined value s ₁ and the predetermined value s ₂ can be set to different values depending on whether the position to be determined is the central portion or the outer peripheral portion of the image. For example, as will be described later, when the same light source is imaged, the luminance is higher in the central portion of the image than in the peripheral portion of the image. Therefore, the values of the predetermined value s ₁ and the predetermined value s ₂ can be set to be stricter than the central portion, that is, a smaller value in the peripheral portion of the image.
In the above processing, the shutter speeds of the IR camera 150 and the color camera 152 can be manually controlled by the user, or can be automatically controlled by the dark imaging condition determining unit 120.

(Preprocessing: Threshold value, predetermined value determined)
Returning to FIG. 6, after the shutter speed, which is a dark imaging condition of the IR camera 150 and the color camera 152, is determined, threshold values and predetermined values used in various determinations described below are determined according to the determined shutter speed. Then, it is stored in the set value storage unit 130 and the emission color determination information storage unit 132 (S6). Specific examples and determination methods of each threshold value and predetermined value will be described later.
Thus, the preprocessing is completed.

(Darkening imaging condition setting)
Next, prior to the start of the process of accepting input by light emission of the light emitter 172, the IR camera 150 and the color camera 152 are set to the dark imaging condition determined in step S4 (S8). Specifically, the shutter speed of the IR camera 150 is set to the first shutter speed, and the shutter speed of the color camera 152 is set to the second shutter speed. The shutter speeds of the IR camera 150 and the color camera 152 can also be configured to be manually controlled by the user, or can be configured to be automatically controlled by the dark imaging condition determining unit 120.

(IR image and color image acquisition)
After the above setting process is completed, a process of accepting an input by light emission of the light emitter 172 is started (S10). The image acquisition unit 112 acquires an IR image and a color image obtained by imaging the same object with the IR camera 150 and the color camera 152 substantially simultaneously (S12). The image acquisition unit 112 can acquire the IR image and the color image obtained at substantially the same time in association with information indicating a correspondence relationship such as the number of frames, acquisition time, imaging time, or time code. Note that the IR camera 150 and the color camera 152 can be configured to acquire moving images at the same frame rate. The term “substantially simultaneous” means that the positions of the light emitters 172 exist at positions corresponding to each other when the IR camera 150 and the color camera 152 respectively capture the movement of the light emitter 172 when the user performs drawing with the input device 170. The state can be recognized. For example, the image acquisition unit 112 may sequentially assign the same number of frames to the IR image and the color image acquired from the IR image input unit 102 and the color image input unit 104 substantially simultaneously. Further, the image acquisition unit 112 may give the acquisition time when the respective images are acquired to the IR image and the color image acquired from the IR image input unit 102 and the color image input unit 104. As another example, the illuminant identification system 10 may be configured to include, for example, a synchronization generator, and an imaging time, a time code, and the like are captured in images captured by the IR camera 150 and the color camera 152 by the synchronization generator. Can also be given. In the following, a case will be described as an example where the image acquisition unit 112 sequentially assigns the same number of frames to the IR image and the color image acquired from the IR image input unit 102 and the color image input unit 104 substantially simultaneously. The image acquisition unit 112 stores the IR image and the color image in the image storage unit 136 in association with information indicating the corresponding relationship.

(Detecting the location of the light emitter from the IR image)
Subsequently, the position information acquisition unit 114, the image acquisition unit 112 acquires from the stored IR image in the image storage unit 136, detects a first pixel luminance value becomes a predetermined threshold value t ₁ or more as emitters Then, the first position information indicating the position of the first pixel is acquired (S14). Specifically, the coordinates (pixel coordinates) of the first pixel are acquired as the first position information. The luminance value of the IR image is based on the RGB value of each pixel. For example, “luminance value = 0.298912 × R + 0.586611 × G + 0.114478 × B (R, B, and G are R value, B value, and G value, respectively) It can be calculated from The threshold t ₁ can be set to a value of about 60 or more and 100 or less, for example. The threshold t ₁ can be determined in step S 6 of FIG. 6, and can be stored in the set value storage unit 130.

FIG. 10 is a diagram schematically illustrating an example of processing for an IR image and a color image. Hereinafter, description will be made with reference to FIG. 10 as appropriate.
As shown in FIG. 10A, the horizontal coordinate of the IR image 200 is taken as the x coordinate and the vertical coordinate is taken as the y coordinate. Position information acquisition unit 114, the luminance value of the sequential scanning to the pixels all the pixels of the IR image 200 sequentially determines whether the threshold value t ₁ or more. Here, if there is a first pixel whose luminance value is equal to or greater than the threshold value t ₁ , the first pixel is detected as the presence position of the light emitter 172. Then, the coordinates (xi, yi) are acquired as the first position information indicating the position of the first pixel. For example, when the IR camera 150 is configured with a VGA of 640 × 480 pixels and the resolution of the IR image is 640 × 480 pixels, xi is a real number from 0 to less than 640, and yi is a real number from 0 to less than 480. Next, the position information acquisition unit 114 notifies the light emission color determination region selection unit 116 of the first position information and the number of frames of the IR image. Thereby, the light emission color determination region selection unit 116 performs a light emission color determination region selection process which will be described later. The position information acquisition unit 114 sequentially performs this process on all the pixels of the IR image 200 acquired by the image acquisition unit 112.

(Select emission color judgment area from color image)
Returning to FIG. 6, when the first position information of the IR image and the number of frames of the IR image are notified from the position information acquisition unit 114, the light emission color determination region selection unit 116 accesses the image storage unit 136 to respond. The light emission color determination area of the light emitter including the second pixel at the position corresponding to the first position is selected from the color image of the number of frames to be processed (S16).

  As shown in FIG. 10B, the horizontal coordinate of the color image 202 is assumed to be m coordinate, and the vertical coordinate is assumed to be n coordinate. Based on the first position information (xi, yi) of the IR image acquired by the position information acquisition unit 114, the light emission color determination region selection unit 116 selects the second pixel at the position corresponding to the first position from the color image. The luminescent color determination area 204 of the illuminant to be included is selected. Here, the coordinates (pixel coordinates) of the second pixel of the color image corresponding to the coordinates (xi, yi) of the first pixel of the IR image are (mi, ni).

  Here, for example, in step S 6, the resolution of the IR image captured by the IR camera 150 and the resolution of the color image captured by the color camera 152 can also be stored in advance in the set value storage unit 130. The light emission color determination area selection unit 116, based on the ratio of the resolution of the IR image stored in the setting value storage unit 130 and the resolution of the color image, the second pixel of the color image corresponding to the first pixel of the IR image. Can be determined. For example, when the color camera 152 is configured with a VGA of 640 × 480 pixels and the resolution of the IR image is 640 × 480 pixels, mi is a real number from 0 to less than 640, and ni is a real number from 0 to less than 480. Further, when the resolution of the IR pixel acquired by the image acquisition unit 112 is equal to the resolution of the color image, mi = xi and ni = yi.

  As shown in FIG. 10B, the emission color determination area selected by the emission color determination area selection unit 116 is the second position at the position corresponding to the first pixel of the IR image acquired by the position information acquisition unit 114. Although it can be configured to include only pixels, it can also be configured to include more pixels including the second pixel. For example, the light emission color determination region selection unit 116 can select a wide range of pixels including the second pixel and a predetermined number of pixels around it as the light emission color determination region.

  FIG. 11 is a diagram schematically illustrating another example of the processing for the IR image 200 and the color image 202. As illustrated in FIG. 11B, the light emission color determination region selection unit 116 can also select a region including the second pixel and surrounding pixels as the light emission color determination region 204. A setting range as to which range is selected as the light emission color determination area 204 can be stored in the setting value storage unit 130 in advance. The emission color determination area selection unit 116 can select the emission color determination area 204 based on the setting range stored in the setting value storage unit 130.

  If the IR image acquired by the IR camera 150 and the color image acquired by the color camera 152 are misaligned, the light emission color determination region selecting unit 116 corrects the misaligned portion and selects the corresponding second pixel. Thus, the second position information can be obtained. For example, as a pre-process, the position of the drawing board 160 in the IR image and the color image obtained by imaging the drawing board 160 with the IR camera 150 and the color camera 152, respectively, is pre-processed. Similarly, by detecting the deviation, it can be set that the deviation has occurred. Misalignment of the drawing board 160 in the IR image and the color image is automatically corrected using a software program such as pattern matching, or the position of the drawing board 160 in the IR image and the color image is designated by the user. Then, it can be corrected by a software program.

(Noise removal for color images)
Returning to FIG. 6, the light emission color determination region selection unit 116 performs noise removal based on the luminance value of the selected light emission color determination region (S18). Specifically, the light emission color determination region selection unit 116 determines whether or not the luminance value Yi of the pixel included in the selected light emission color determination region is equal to or greater than a predetermined necessary luminance value Yd, and the luminance value is the necessary luminance value Yd. When the luminance value is lower, the emission color determination area is determined to be noise, and when the luminance value is equal to or greater than the necessary luminance value Yd, the emission color determination area is determined not to be noise.

  When the illuminant is imaged by the color camera 152, even if the illuminant has the same brightness, the luminance in the obtained image is different between the central portion and the outer peripheral portion of the light spot of the illuminant. In particular, when a light emitting body having high directivity such as LED is used, the light emitting body is in the vicinity of an extension of the central axis of the camera lens, and the light spot of the light emitting body is small when the incident angle to the camera lens is small. The central part is imaged and the luminance value increases. On the other hand, when the light emitter is located away from the extension of the center axis of the lens of the camera and the incident angle to the camera lens is large, the outer peripheral portion of the light spot is imaged and the luminance value becomes small. This tendency is particularly noticeable in the case of color images. That is, even when the same illuminant is imaged, if the illuminant is present at the center of the color image captured by the color camera 152, the center of the light spot of the illuminant will be imaged, so the light emission The brightness value of the body increases. On the other hand, when the illuminant is present in the outer peripheral part of the color image captured by the color camera 152, the outer peripheral part of the light spot of the illuminant is imaged, so that the luminance value of the illuminant decreases. For this reason, when the same required luminance value is used for the central portion and the outer peripheral portion of the color image, for example, it is difficult to detect the presence of the light emitter in the outer peripheral portion, but even the noise is determined to be the light emitter in the central portion. There is a high possibility that Therefore, in the present embodiment, the necessary luminance value Yd used for the determination by the light emission color determination region selection unit 116 is greater when the distance Di is larger according to the distance Di between the center of the color image and the light emission color determination region. Is set to be low.

FIG. 9 is a flowchart showing this procedure in detail.
First, the light emission color determination region selection unit 116 calculates the luminance value Yi of the light emission color determination region of the color image (S104). The luminance value of the color image is based on the RGB value of each pixel. For example, “luminance value = 0.298912 × R + 0.586611 × G + 0.114478 × B (R, G, B are R value, G value, B value, respectively) It can be calculated from Here, when a plurality of pixels are included in the emission color determination area, the emission color determination area selection unit 116 uses the average value of the luminance values of the plurality of pixels as the luminance value Yi of the pixels included in the emission color determination area. It is also possible to use the luminance value of the pixel located at the center of the plurality of pixels. Also, a weighted average value such that the contribution ratio increases as the luminance value of the pixel located at the center among the plurality of pixels can be used.

  Next, the light emission color determination area selection unit 116 calculates a distance Di between the center of the color image and the light emission color determination area (S106). Here, when a plurality of pixels are included in the emission color determination region, the emission color determination region selection unit 116 can use the distance between the center of the color image and the center of the emission color determination region as the distance Di. .

The distance Di is (mc, nc) when the center coordinates of the color image are
Di = sqrt ((mi-mc) ² + (ni-nc) ² )
(Where sqrt is a function for obtaining the square root).

  As shown in FIGS. 10C and 11C, the light emission color determination region selection unit 116 sets the distance Di from the center coordinates (mc, nc) of the color image 202 to the center coordinates of the light emission color determination region 204. Based on this, the necessary luminance value Yd is calculated and noise determination of the emission color determination area 204 is performed.

  Returning to FIG. 9, thereafter, the light emission color determination region selection unit 116 calculates the necessary luminance value Yd based on the distance Di (S108). The required luminance value Yd can be calculated based on the following equation, for example. Based on the following formula, the required luminance value Yd is set so that the value continuously decreases as the distance Di increases.

  Yd = (Yf−Yc) × Di / Df + Yc (Formula 1)

  Here, “Yc” is a required luminance value at the center of the color image, “Yf” is a required luminance value at the outermost periphery of the color image, and “Df” is a distance from the center of the color image to the outermost periphery. Yes (Yf <Yc). Yc can be set to a value of about 80 to 120, for example. Although the value of Yf depends on the value of Df, for example, when the resolution of the color image is 640 × 480 pixels and Df = 640 pixels, the value can be set to about 15 or more and 30 or less. The necessary luminance values Yc and Yf can be determined in step S6 of FIG. 6 and can be stored in the set value storage unit 130. Formula 1 and other values can also be stored in the set value storage unit 130 in advance.

  In the above example, the example in which the required luminance value Yd continuously changes in accordance with the distance Di between the center portion of the color image and the emission color determination region is shown. However, the center portion of the color image and the emission color determination region are shown. In other examples, the necessary luminance value Yd is changed stepwise according to the range of the distance Di, if the distance Di is set so as to be lower according to the distance Di. Can also be set.

  Subsequently, the light emission color determination region selection unit 116 determines whether or not the luminance value Yi is equal to or greater than the necessary luminance value Yd (S110). If the luminance value Yi is equal to or higher than the required luminance Yd (YES in S110), the light emission color determination area selection unit 116 determines that a light emitter is detected in this light emission color determination area (S112), and the light emission color determination part 118 Notify that. Thereafter, the light emission color determination unit 118 performs the light emission color determination processing in step S20 of FIG. On the other hand, if the luminance value Yi is not equal to or greater than the required luminance value Yd in step S110 (NO in S110), the light emission color determination region selection unit 116 determines that this light emission color determination region is noise (S114), and performs the process. finish.

(Determination of emission color)
Returning to FIG. 6, when the illuminant detection determination is performed in step S <b> 18 and a notification is received from the emission color determination region selection unit 116, the emission color determination unit 118 emits the color image selected by the emission color determination region selection unit 116. Based on the color component information (RGB values) of the pixels included in the color determination area, the light emission color of the light emitter is determined (S20). Here, the color component information of the pixel can be an RGB value.

FIG. 12 is a flowchart showing in detail the processing procedure of step S20 of FIG.
Here, the case where the luminescent color of the light emitter 172 is any one of white, blue, green, and red will be described as an example. In such a case, the light emission color determination unit 118 first determines whether or not the light emission color of the light emitter is white. When the luminescent color of the illuminant is white, there is little tendency for any of the RGB values to protrude, and it is determined whether the color is white in consideration of the balance of the RGB values. For this reason, the possibility of erroneous determination can be reduced even when variations in the output of the R value, G value, and B value occur, and the determination accuracy can be improved.

(White judgment)
When the illuminant is white, the RGB values are similarly averaged, and the average value is relatively high.

First, the emission color determination section 118, the color component information of the pixels contained in the luminescent color determination area based on (RGB value), the average value aveC RGB values is equal to or greater than a predetermined value _{s 3} (S122) . Here, assuming that the RGB values of the pixels included in the light emission color determination region are sR, sG, and sB, respectively, the average value aveC = (sR + sG + sB) / 3 of the RGB values. Note that, as in the example described with reference to FIGS. 11B and 11C, when the emission color determination region 204 includes a plurality of pixels, for example, sR, sG, and sB are a plurality of pixels. It can be an average value. In this case, for example, a weighted average value that increases the value of the pixel located at the center of the emission color determination region 204 may be used.

If the average value aveC RGB values is larger than the predetermined value _{s 3} (YES in S122), the emission color determination section 118, each component is equal to or average or (S124). Determination of whether each component is average can be performed as follows.

First, the minimum value minC among sR, sG, and sB is detected. Next, tR = sR-minC, tG = sG-minC, and tB = sB-minC obtained by subtracting minC from sR, sG, and sB, respectively. Next, an average value aveTC = (tR + tG + tB) / 3 of tR, tG, and tB is calculated. When each component is average, the values of tR, tG, and tB obtained by subtracting the minimum value minC in sR, sG, and sB from sR, sG, and sB, respectively, are small. The light emission color determination unit 118 can determine whether or not aveTC is smaller than the predetermined value s ₄ as a determination of whether or not each component is average.

In other words, the emission color determination section 118, when aveC> (YES in S122) a predetermined value _s 3 and aveTC of <(YES in S124) predetermined value _s 4, that the emission color of the light-emitting element is determined to be white it can. In this case, the light emission color determination unit 118 outputs white as the light emission color of the light emitter (S126). Here, the predetermined value s ₃ is the critical threshold value for determining that the white (white_color_limit), for example, may be a value at 50 or 70 below. The predetermined value s ₄ can be a value at for example 15 to 25.

  If the light emission color of the light emitter cannot be determined to be white by the above procedure (NO in S122 or NO in S124), the light emission color determination unit 118 determines that the light emission color of the light emitter is blue, green, or red other than white. It is determined whether or not.

(Blue, green, red judgment)
In the determination process for the emission colors other than white, the color determination of the component having the largest value is performed according to sR, sG, and sB of the pixels included in the emission color determination area. That is, when the B component (sB) is the largest (YES in S128), blue determination processing is performed (S130). If the G component (sG) is the largest (NO in S128, YES in S132), a green color determination process is performed (S134). If the R component (sR) is the largest (NO in S132), a red determination process is performed (S136). Next, a specific example of the determination process for each color will be described.

(A) Blue determination process First, an example of the blue determination process in step S130 will be described. FIG. 13 is a flowchart showing in detail an example of the processing procedure of step S130 of FIG.
First, the emission color determining unit 118 calculates the ratio perG = tG / tB of the green component for the blue component, perG = tG / tB is determined whether or not a predetermined value _{s 5} below (S160). Here, the predetermined value s ₅ can be set to a value of about 0.2 or more and 0.3 or less, for example. Perg = tG / if tB is in a predetermined value _{s 5} below (YES in S160), the emission color determination unit 118, a distance Di between the center portion and the luminescent color determination area of the color image whether within a predetermined value _{d 1} Judgment is made (S162).

As a result of studies by the present inventors, it has been clarified that when the emission color is blue, sB is greatly increased when the emission color determination region is close to the center of the color image. For this reason, in the present embodiment, the determination procedure is different depending on whether or not the pixel to be determined is located at the center of the color image. Predetermined value d _1, for example the number of pixels of the long side of the color image when the n _1, can be set to a value of n _1/20 or more n _1/5 or less. For example, when the resolution of the color camera 152 is a VGA size of 640 × 480 (horizontal × vertical) pixels, the number of horizontal sides n ₁ = 640 pixels, which is the long side of the color image, is given as an example. ₁ can be a 64 pixel to be _n 1/10. That is, in this case, the determination procedure can be made different depending on whether or not the determination target pixel is within a radius of 64 pixels from the center of the color image. The predetermined value d ₁ is set by actually acquiring a color image using a blue input device, measuring the value of sB at each position of the color image, and detecting in which range the sB is greatly increased. Can do.

When the distance Di is not within the predetermined value _{d 1} in step S162 (NO in S162), and blue determined (S164). On the other hand, (YES in S162) when the distance Di is within a predetermined value _{d 1,} sB is determined whether the predetermined value _{s 6} or more (S166). Here, the predetermined value s ₆ can be set to a value of about 96 to 120, for example. sB often higher than the predetermined value _{s 6} (YES in S166), but the blue determination (S164), if not the predetermined value _{s 6} or more (NO in S166), the noise determination (S168).

As another example, even when the distance Di is not within the predetermined value d ₁ in step S162, that is, even when the emission color determination region is not in the center of the color image, a reference value is provided and sB is equal to or greater than the reference value. It is also possible to make a blue determination only when it is greater than or equal to a reference value, and to make a noise determination when it is smaller than the reference value. As the reference value in this case, a predetermined value s ₇ having a value smaller than the predetermined value predetermined value s ₆ used when the emission color determination region is in the center of the color image can be used.

If perG = tG / tB is not equal to or smaller than the predetermined value s ₅ in step S160 (NO in S160), whether or not perG = tG / tB is equal to or larger than the predetermined value s ₈ (however, the predetermined value s ₈ > the predetermined value s ₅ ). Is determined (S170). Here, the predetermined value _{s 8} may be a value at for example 0.7 to 0.9.

Perg = If tG / tB is a predetermined value or more _{s 8} (YES in S170), the emission color of the light emitter is likely to be green. In this case, emission color determination unit 118, the value of tB used for the determination in step S170 is equal to or a predetermined value _{s 9} or higher (S172). Here, the predetermined value s ₉ may be a value at for example 30 or more and 50 or less. If the value of tB is too small, the tG by noise is very small amount change, because perG fluctuates extremely, if tB is smaller than the predetermined value s ₉ is for the noise process. tB cases the predetermined value _{s 9} or more (YES in S172), the green judgment (S173).

However, even if tB at step S172 is smaller than the predetermined value _{s 9} (NO in S172), sB predetermined value _{s 10} (although the predetermined value _{s 10>} predetermined value _{s 9)} For more (YES in S174), step S164 Proceed to step 4 to make blue determination. This is a process that takes into account that when the value of sB is large, the color separation capability of the camera is exceeded because blue is too bright, and the G component may be output in a saturated state. is there. Here, the predetermined value _{s 10} may be a value at for example 200 to 220 or less.

If sB is not the predetermined value _{s 10} or more in step S174 (NO in S174), the noise determination (S176). If perG = tG / tB is not the predetermined value _{s 8} or more in step S170 (NO in S170) is also a noise determination (S178).

(B) Green Determination Process Next, an example of the green determination process will be described. FIG. 14 is a flowchart showing in detail an example of the process in step S134 of FIG.
Again, the emission color determining unit 118 calculates the ratio perR = tR / tG of the red component to the green component, when PERR is equal to or less than the predetermined value _{s 11} (YES of S180) is the green determined (S182). Here, the predetermined value _{s 11} may, for example, may be a value at 0.4 to 0.6. On the other hand, if the perR is not less than a predetermined value _{s 11} at step S180 (NO in S180), the noise determination (S184).

(C) Red determination process Next, an example of a red determination process will be described. FIG. 15 is a flowchart showing in detail an example of the processing procedure of step S136 of FIG.
Again, the emission color determining unit 118 calculates the ratio perG = tG / tR of the green component to red component, if Perg is a predetermined value _{s 12} or less (YES in S190) is a red determined (S192). Here, the predetermined value _{s 12} may be a value at for example 0.7 to 0.9. On the other hand, if the perG is not less than a predetermined value _{s 12} at step S190 (NO in S190), the noise determination (S194).
Thus, the determination of the emission color is completed. The light emission color determination unit 118 notifies the position information acquisition unit 114 of data indicating the light emission color of the determined light emitter. The position information acquisition unit 114 receives the data indicating the emission color acquired from the emission color determination unit 118 and the first position information (xi, yi) indicating the position of the first pixel of the IR image. Notify In another example, as the position information, second position information (mi, ni) indicating the position of the second pixel of the color image may be used.

(Input operation reception)
Returning to FIG. 6, the input receiving unit 122 is identified by the emission color based on the emission color of the light emitter 172 and the first position information (or second position information) output from the emission color determination unit 118. It is accepted that there is an input from the device 170 to the first position information (or second position information) (S22). Specifically, the input receiving unit 122 refers to the correspondence storage unit 134 and identifies which input device 170 the input information is based on, based on the data indicating the emission color of the light emitter 172, and the IR image. An input to the coordinates (xi, yi) (or the coordinates (mi, ni) of the color image) is accepted. For example, when the luminescent color output from the luminescent color determination unit 118 is white, the input receiving unit 122 accesses the correspondence storage unit 134 and uses the input device with ID “1” of the input device corresponding to white to The fact that there is an input to the position information of 2 is accepted. The input receiving unit 122 notifies the program execution processing unit 124 of the received input.

  Thereafter, the program execution processing unit 124 executes the program according to the input operation received by the input receiving unit 122 (S24). Data formed by the program executed by the program execution processing unit 124 is output from the data output unit 106. The data output from the data output unit 106 can be projected on the projector projection film 164 by the projector 154 and displayed on the drawing surface of the drawing board 160. As a result, the user who performs input using the input device 170 can check the data reflecting his / her input while looking at the drawing board 160.

  Here, the program executed by the program execution processing unit 124 may be, for example, a drawing program for the user to draw a figure or the like using the input device 170. In this case, for example, the program execution processing unit 124 can output, from the data output unit 106, a locus corresponding to the locus of the illuminant in the IR image or color image acquired by the image acquisition unit 112. In addition, the program execution processing unit 124 can display a color palette or the like for setting a color of a graphic or the like drawn by each user on the drawing board 160, for example. When the user selects any color from the color palette, the program execution processing unit 124 displays the position information of the displayed color palette, the position information of the input device 170, and the light emitter 172 that the user performs using the input device 170. Based on the control operation of the light emission timing (corresponding to the click) and the light emission color of the input device 170, the color selected by each user is judged, and the color of the locus of the light emitter 172 output from the data output unit 106 is selected. Can be colored.

Next, effects of the light emitter identification device 100 in the present embodiment will be described.
According to the light emitter identification device 100 in the present embodiment, input information from the input device is received based on the position information of the light emitter, and the input device is identified according to the light emission color determined from the color component information of the color image. be able to. Therefore, it is possible to identify input information input by a plurality of users using a plurality of input devices each including a light-emitting body having a different emission color. In addition, since the position of the light emitter is first detected from the IR image and the light emission color determination process is performed based on the color component information at the corresponding position of the color image, the light emitter can be detected and the light emission color can be determined quickly and accurately. Can do.

  In this embodiment, an LED is used as the light emitter of the input device. As described above, the incident angle of light from the light emitter to the camera lens varies depending on the position of the light emitter. In the above embodiment, in the noise determination processing of the emission color determination area of the color image, the required luminance value Yd used for determination has a large distance Di according to the distance Di between the center of the color image and the emission color determination area. The value is set to be lower. For this reason, noise removal is performed in consideration of the difference in luminance value of the light emitter due to the difference in the incident angle of light from the light emitter to the camera lens, so that noise can be removed with high accuracy.

  The illuminant identification system 10 including the illuminant identification device 100 having the above-described configuration is configured to input an image by illuminating the illuminant 172 with an imaging unit 148 such as a camera and recognizing the coordinates in the operation of computer software. It is configured to accept. At that time, by using the light emitters 172 having different emission colors, a plurality of light emitters 172 are recognized in real time so that the computer software can be operated simultaneously by a large number of people. As a result, for example, in the operation of application software such as entertainment software such as games, event software, and educational software, a large number of people can be operated simultaneously. This also allows the input device to perform operations similar to free and intuitive operations such as a mouse, touch panel, and tablet.

  Each component of the light emitter identification device 100 shown in FIG. 1 is not a hardware unit configuration but a functional unit block. Each component of the illuminant identification device 100 includes an arbitrary computer CPU, memory, a program that realizes the components shown in the figure loaded in the memory, a storage unit such as a hard disk that stores the program, and a network connection interface. It is realized by any combination of hardware and software. It will be understood by those skilled in the art that there are various modifications to the implementation method and apparatus. In addition, each component of the light emitter identification device 100 does not need to be provided in one device, and may be configured to be distributed in a plurality of devices and exchange data with each other via a network.

  As mentioned above, although embodiment of this invention was described with reference to drawings, these are the illustrations of this invention, Various structures other than the above are also employable.

(Modification 1)
In the above embodiment, as described in step S6 of FIG. 6, according to the dark imaging condition determined after determining the dark imaging condition in the application environment where ambient light exists in step S4. An example is shown in which threshold values and predetermined values used in various determinations are determined and stored in the set value storage unit 130 and the emission color determination information storage unit 132. However, as another example, threshold values and predetermined values used in various determinations are determined in advance for each of a plurality of imaging conditions prior to the process of step S4 in FIG. It can also be stored in the information storage unit 132. An example is shown in FIG. FIG. 16 is a diagram illustrating an example of a part of the internal configuration of the set value storage unit 130. Here, a case where the shutter speed is controlled as the dark imaging condition is shown. For example, the shutter speed is assumed to be 1/500 seconds, 1/250 seconds, 1/125 seconds, 1/60 seconds, etc., and the threshold value t ₁ , necessary luminance values Yf and Yc used for each are stored. Has been. In the case of such a configuration, various determinations of the corresponding set values are made according to the first shutter speed of the color camera 152 and the second shutter speed of the IR camera 150, which are the darkening imaging conditions determined in step S4. Can be used. For example, in the illustrated example, when the second shutter speed which is the dark imaging condition of the IR camera 150 is 1/250 seconds, the threshold t ₁ used in step S14 is 70, and the second shutter speed is 1/125. for seconds, the threshold t ₁ may be a 75 or the like.

  In this way, it is not necessary to perform the process of determining the set value in step S6 each time the dark imaging condition is determined in FIG. 6 and step S4. Further, even if the ambient light changes after the light emitter identification system 10 is installed in a certain environment, it is not necessary to re-determine the set value. Therefore, it is possible to flexibly cope with changes in ambient light around the light emitter identification system 10. Therefore, each time the illuminator identification device 100 is activated, the dark imaging conditions of the IR camera 150 and the color camera 152 are determined according to the state of the surrounding ambient light, and setting values corresponding to the determined dark imaging conditions are set. Various determinations can be made using this method.

FIG. 8 shows the projection image 164 projected on the projector when determining the dark imaging conditions of the IR camera 150 and the color camera 152 according to the ambient ambient light condition, and is provided to the user on the drawing surface side of the drawing board 160. It is a figure which shows an example of the screen displayed.
Here, in order to prevent the user from causing the light emitter 172 of the input device 170 to emit light during the preprocessing, for example, a message such as “Preparing. Do not switch on the input device” may be displayed. it can. Thereby, it is possible to determine the dark imaging condition in consideration of the ambient light at the place where the light emitter identification device 100 is actually used.

(Modification 2)
In the above embodiment, in the procedure for determining the dark imaging condition described in step S4 of FIG. 6, the first preprocessed image captured by the dark imaging condition determination unit 120 with the IR camera 150 and the color camera 152 is used. In the example, the dark imaging conditions are determined by determining whether or not the pixel values at a plurality of positions in the second preprocessed image are equal to or smaller than the predetermined value s ₁ or the predetermined value s _2, respectively. However, the user determines whether or not ambient light is not reflected in the first preprocessed image and the second preprocessed image captured by the IR camera 150 and the color camera 152. You can also. In this case, the darkening imaging condition determination unit 120 uses the current shutter speed of the IR camera 150 or the color camera 152 and the RGB values at a plurality of positions of the IR image or color image acquired by the image acquisition unit 112 as a data output unit. It can output from 106 and can be made to show to a user (setting person). FIG. 17 is a diagram illustrating an example of a screen 184 presented to the user. On the screen 184, an image 188 (color image or IR image) including the image 186 of the drawing board is displayed. Also, RGB values at each of a plurality of positions a to i in the image 188 are displayed. The user looks at the RGB values at each of a plurality of positions a to i displayed on the screen 184, and determines the current shutter speed as a dark imaging condition depending on whether or not the RGB values at all positions are less than or equal to a predetermined value. It can be determined whether or not. As still another example, the RGB values at each position can be graphed and presented to the user.

(Modification 3)
When an LED is used as the light emitter of the input device 170, the LED has a problem that manufacturing variation tends to occur. For example, even for a blue light emitter, RGB values may vary greatly due to manufacturing variations. Therefore, for example, as a preprocessing, the illuminant identification device 100 captures, with the color camera 152, the state in which the illuminant of the input device of the luminescent color that is scheduled to be used by instructing the luminescent color is used by the user. Color component information of the color illuminant can be acquired and stored in the luminescent color determination information storage unit 132 as a reference value. Then, when the emission color determination unit 118 determines the emission color, the emission color determination unit 118 refers to the reference value stored in the emission color determination information storage unit 132 and determines the emission color based on the similarity to the reference value. You can also

FIG. 18 is a diagram illustrating an example of a screen that is projected onto the projector projection film 164 and displayed to the user on the drawing surface side of the drawing board 160 during the above preprocessing.
Here, for example, a message such as “Please switch on the red input device and paint in this circle” so that the user emits the illuminant 172 of the input device 170 of each emission color during the preprocessing. Can be displayed. The image acquisition unit 112 can store the color component information of the illuminant acquired in this state in the luminescent color determination information storage unit 132 as a reference value of the red illuminant. In addition, reference values can be obtained in the same procedure for other emission colors other than red.

(Modification 4)
In the above embodiment, the drawing board 160 is provided, and the user presses the light emitter 172 of the input device 170 against the drawing board 160 to perform drawing. However, as another example, the user does not press the light emitter 172 of the input device 170 against the drawing board 160, but the user can freely hold the input device 170 in his / her hand and perform drawing or the like. The body identification system 10 may not include the drawing board 160, and the imaging means 148 of the IR camera 150 and the color camera 152 may be configured to capture a state in which the user can freely operate the input device 170 in his / her hand.

(Modification 5)
Moreover, in the above embodiment, the example which the information output from the data output part 106 is projected on the projection film 164 for projectors by the projector 154 was shown. In order to play with a plurality of people, it is preferable to use such a relatively large display device. However, the display device is not limited to projection by a projector, and a normal display or the like can also be used.

(Modification 6)
The modification of the process of step S16 to step S20 of FIG. 6 is shown. FIG. 19 shows a flowchart of this processing procedure.
In the process of step S14 in FIG. 6, the illuminant is detected from the IR image, and the number of frames (for example, 10) and the first position information (for example, (x1, y1)) of the IR image are detected from the position information acquisition unit 114. When notified (YES in S202), the light emission color determination region selecting unit 116 stores the notified number of frames (10) and the first position information (x1, y1) in a storage unit (not shown) (S204). ). Thereafter, the light emission color determination region selection unit 116 and the light emission color determination unit 118 first position of the color image associated with the number of frames (10) corresponding to the number of frames (10) stored in the storage unit. An emission color determination process is performed for a position corresponding to the information (for example, (m1, n1)) (S206). When the light emission color of the light emitter is determined from the color image of the number of frames (10) (YES in S208), this process ends and the process proceeds to step S22 in FIG. On the other hand, when the emission color of the light emitter is not determined from the color image of the number of frames (10) (NO in S208), the emission color determination area selection unit 116 increases the number of frames by 1 (S210), and returns to step S204. To update the number of frames (11) in the storage unit. Thereafter, the emission color determination process is performed for the position (m1, n1) of the color image associated with the updated number of frames (11) (S206), and the emission color of the light emitter is determined from the color image. The same processing can be performed until

  However, if the emission color of the illuminant is not determined from the color image even if the number of frames is increased by a predetermined number or more, the processing for this position (m1, n1) can be terminated. Further, the light emitter is detected from the new IR image in the process of step S14 in FIG. 6, and the number of frames (for example, 15) and the first position information (for example, (x2, y2)) of the new IR image are the positions. When notified from the information acquisition unit 114 (YES in S202), the process for the position (m1, n1) of the color image is limited to the number of frames (14), and the process is performed even when the emission color of the light emitter is not determined from the color image. Can be terminated.

DESCRIPTION OF SYMBOLS 10 Illuminant identification system 100 Illuminant identification device 102 IR image input unit 104 Color image input unit 106 Data output unit 112 Image acquisition unit 114 Position information acquisition unit 116 Luminescent color determination region selection unit 118 Luminous color determination unit 120 Dark imaging condition Determination unit 122 Input reception unit 124 Program execution processing unit 130 Setting value storage unit 132 Luminescent color determination information storage unit 134 Corresponding storage unit 136 Image storage unit 148 Imaging unit 150 IR camera 152 Color camera 154 Projector 160 Drawing board 162 Support base 164 Projector Projection film 166 Trajectory 168 Trajectory 170 Input device 170a White input device 170b Blue input device 170c Green input device 170d Red input device 172 Luminescent body 174 Switch 182, 186, 188 Image 180 1 preprocessed image 184 screen 200 IR image 202 color image 204 emission color determination area

Claims (8)

An image acquisition unit for acquiring an IR image and a color image obtained by simultaneously imaging the same object with an IR camera and a color camera;
A position information acquisition unit that detects, from the IR image, a first pixel having a luminance value equal to or greater than a predetermined threshold as the presence position of the light emitter, and acquires first position information indicating the presence position;
Based on the first position information acquired by the position information acquisition unit, a light emission color determination for selecting a light emission color determination region of a light emitter including a second pixel at a position corresponding to the first position from the color image. An area selector;
A light emission color determination unit that determines a light emission color of the light emitter based on color component information of pixels included in the light emission color determination region of the color image selected by the light emission color determination region selection unit;
A light emitter identification device including: The light emitter identification device according to claim 1,
The light emission color determination unit associates and outputs the light emission color of the light emitter and the first position information of the IR image or the second position information indicating the position of the second pixel of the color image. Body identification device. The light emitter identification device according to claim 2,
Based on the emission color of the light emitter and the first position information or the second position information output by the emission color determination unit, the first position information or the first position information is input from the input device identified by the emission color. The light-emitting body identification device which further contains the input reception part which receives that there existed the input to 2 positional information. In the light-emitting body identification device in any one of Claim 1 to 3,
The light emission color determination region selection unit determines whether or not a luminance value Yi of a pixel included in the selected light emission color determination region is equal to or greater than a predetermined necessary luminance value Yd, and the luminance value Yi is greater than the necessary luminance value Yd. When the luminance value Yi is not less than noise when the luminance value Yi is equal to or greater than the necessary luminance value Yd, the emission color determination region is determined as noise when the emission color determination region is low.
The light emission color determination unit, when the light emission color determination region selection unit determines that the light emission color determination region is not noise, based on the color component information of the pixels included in the light emission color determination region of the color image, Determining the emission color of the illuminant;
The required luminance value Yd is a light emitter identification device set such that the larger the distance Di is, the lower the value is according to the distance Di between the center of the color image and the light emission color determination region. Obtaining an IR image and a color image respectively obtained by simultaneously imaging the same object with an IR camera and a color camera;
Detecting, from the IR image, a first pixel having a luminance value equal to or greater than a predetermined threshold as an existing position of the light emitter, and obtaining first position information indicating the existing position;
Based on the first position information acquired in the step of acquiring the first position information, an emission color determination region of a light emitter including a second pixel at a position corresponding to the first position from the color image. A process of selecting
Determining a light emission color of the light emitter based on color component information of pixels included in the light emission color determination region of the color image selected in the step of selecting the light emission color determination region;
A light emitter identification method including: The light emitter identification method according to claim 5, wherein:
Further including the step of setting the color camera to a dark imaging condition in which environmental light other than light emission of the light emitter is not reflected in a color image captured by the color camera,
In the step of selecting the emission color determination region from the color image, a light emitter identification method of selecting the emission color determination region from the color image captured by the color camera set in the dark imaging condition. In the light-emitting body identification method of Claim 5 or 6,
Further including the step of setting the IR camera to a dark imaging condition in which ambient light other than light emission of the light emitter is not reflected in an IR image captured by the IR camera;
In the step of acquiring the first position information from the IR image, the first pixel is detected as the presence position of the light emitter from the IR image captured by the IR camera set to the dark imaging condition. A light emitter identification method for acquiring first position information indicating the existence position. Computer
Image acquisition means for acquiring an IR image and a color image obtained by simultaneously imaging the same object with an IR camera and a color camera;
Position information acquisition means for detecting, from the IR image, a first pixel having a luminance value equal to or greater than a predetermined threshold value as the presence position of the light emitter, and acquiring first position information indicating the presence position;
Based on the first position information acquired by the position information acquisition means, a light emission color determination for selecting a light emission color determination region of a light emitter including a second pixel at a position corresponding to the first position from the color image. Area selection means,
A light emission color determination unit that determines a light emission color of the light emitter based on color component information of a pixel included in the light emission color determination region of the color image selected by the light emission color determination region selection unit;
Illuminant identification program to function as.

Images