Classifications

• • G—PHYSICS
  • G01—MEASURING; TESTING
  • G01S—RADIO DIRECTION-FINDING; RADIO NAVIGATION; DETERMINING DISTANCE OR VELOCITY BY USE OF RADIO WAVES; LOCATING OR PRESENCE-DETECTING BY USE OF THE REFLECTION OR RERADIATION OF RADIO WAVES; ANALOGOUS ARRANGEMENTS USING OTHER WAVES
  • G01S11/00—Systems for determining distance or velocity not using reflection or reradiation
  • G01S11/12—Systems for determining distance or velocity not using reflection or reradiation using electromagnetic waves other than radio waves
• • G—PHYSICS
  • G06—COMPUTING OR CALCULATING; COUNTING
  • G06F—ELECTRIC DIGITAL DATA PROCESSING
  • G06F21/00—Security arrangements for protecting computers, components thereof, programs or data against unauthorised activity
  • G06F21/30—Authentication, i.e. establishing the identity or authorisation of security principals
  • G06F21/31—User authentication
  • G06F21/32—User authentication using biometric data, e.g. fingerprints, iris scans or voiceprints
• • G—PHYSICS
  • G06—COMPUTING OR CALCULATING; COUNTING
  • G06F—ELECTRIC DIGITAL DATA PROCESSING
  • G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
  • G06F3/01—Input arrangements or combined input and output arrangements for interaction between user and computer
  • G06F3/011—Arrangements for interaction with the human body, e.g. for user immersion in virtual reality
• • G—PHYSICS
  • G06—COMPUTING OR CALCULATING; COUNTING
  • G06F—ELECTRIC DIGITAL DATA PROCESSING
  • G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
  • G06F3/01—Input arrangements or combined input and output arrangements for interaction between user and computer
  • G06F3/017—Gesture based interaction, e.g. based on a set of recognized hand gestures
• • H—ELECTRICITY
  • H04—ELECTRIC COMMUNICATION TECHNIQUE
  • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
  • H04N23/00—Cameras or camera modules comprising electronic image sensors; Control thereof
  • H04N23/20—Cameras or camera modules comprising electronic image sensors; Control thereof for generating image signals from infrared radiation only
  • H04N23/23—Cameras or camera modules comprising electronic image sensors; Control thereof for generating image signals from infrared radiation only from thermal infrared radiation

Definitions

• Embodiments of this invention are directed to user interfaces for control of computer systems and more specifically to user interfaces that use thermal imaging to provide or augment control input to a computer program.
• interfaces have been developed for use in conjunction with video games that rely on other types of input.
• Microphone-based systems are used for speech recognition systems that try to supplant keyboard inputs with spoken inputs.
• FIG. 1 is a schematic diagram illustrating a thermal imaging interface method for control of a computer program according to an embodiment of the present invention.
• FIGs. 2A-2B are schematic diagrams illustrating use of a thermal imaging interface according to an embodiment of the present invention.
• FIG. 3 is a schematic diagram illustrating use of a thermal imaging interface according to an alternative embodiment of the present invention.
• FIG. 4 is a schematic diagram illustrating use of a thermal imaging interface according to another alternative embodiment of the present invention.
• FIG. 5 is a block diagram depicting an example of a computer implemented apparatus that uses a thermal imaging interface in accordance with an embodiment of the present invention.
• FIG. 6 is a block diagram illustrating a non-transitory computer readable medium containing computer-readable instructions for implementing a thermal imaging interface method according to an embodiment of the present invention.
• Embodiments of the present invention implement a new user interface for computer programs based on the detection of various user characteristics through thermal imaging to provide a control input.
• Embodiments of the present invention can overcome the disadvantages associated with the prior art through use of thermal imaging in conjunction with a user interface to provide or augment input to a computer system.
• Thermal imaging cameras can provide information about the temperature or variation in temperature of objects within an image.
• the variation in temperature can distinguish objects that might otherwise appear identical in an image taken with a conventional (visible light) camera.
• the variation in temperature may also be used to detect certain characteristics of each object that may be used to control a computer program.
• FIG. 1 is a schematic diagram illustrating a possible system for controlling a computer program using a thermal imaging interface.
• the system 100 generally includes a
• thermographic camera 103 coupled to a computer processing system 105.
• the thermographic camera 103 may be positioned proximate a video display 107 that is coupled to the processing system 105 such that the user 101 faces the camera 103 when facing the display 107.
• thermographic camera 103 to track various user characteristics (e.g., heart rate, respiration rate) is particularly advantageous because it avoids the need to put sensors on the user's body. Additional information about the user may be obtained with much less intrusion. Tracking the heart rate of a user through thermographic imaging avoids the need for attaching a pulse monitor to the body. Likewise, tracking the respiration rate through thermographic imaging avoids the need for attaching a headset or microphone to the user's face. Ultimately, the functionality of the computer program can be further optimized through processing of this additional information without compromising the user's mobility and comfort. Determining user characteristics using a thermographic camera 103 is also advantageous because it avoids the need for the user to hold a controller device by hand to ascertain those characteristics. This frees the user's hands for other tasks, or allows the controller to perform other tasks.
• user characteristics e.g., heart rate, respiration rate
• thermographic camera refers to a type of camera sometimes called a Forward Looking Infrared (FLIR) or infrared camera that forms an image using infrared radiation.
• FLIR Forward Looking Infrared
• the thermographic camera 103 forms an image using one or more lenses and a sensor in a fashion similar to a camera that forms an image using visible light.
• an infrared camera can operate in wavelengths as long as 14,000 nm (14 ⁇ ).
• thermographic camera 103 can take advantage of the fact that all objects emit a certain amount of black body radiation as a function of their temperatures. Generally speaking, the higher an object's temperature, the more infrared radiation the body emits as black-body radiation. An infrared camera can detect this radiation in a way similar to the way an ordinary camera detects visible light. However, since bodies emit infrared radiation even in total darkness, the ambient light level does not matter.
• Embodiments of the present invention may use more than one thermographic camera. For example, two thermographic cameras may be used in a side-by-side configuration to provide stereoscopic (3D) images that can provide three-dimensional information. An equivalent thermographic "stereo" camera may be implemented in a single device having side-by-side lenses that image different views of the some scene onto a sensor or array of sensors.
• Multiple cameras might be used to create a three-dimensional representation of any number of user characteristics (e.g., breath volume and rate of flow).
• the thermographic camera 103 may use a cooled thermal sensor or an uncooled thermal sensor operating at ambient temperature, or a sensor stabilized at a temperature close to ambient using small temperature control elements.
• Modern uncooled detectors use sensors that detect changes in resistance, voltage or current when heated by infrared radiation. These changes can then be measured and compared to values at the operating temperature of the sensor.
• Uncooled infrared sensors can be stabilized to an operating temperature to reduce image noise, but they are not cooled to low temperatures and do not require bulky, expensive cryogenic coolers. This makes infrared cameras smaller and less costly.
• Uncooled detectors are mostly based on pyroelectric and ferroelectric materials or microbolometer technology. These materials are used to form a detector having an array of pixels with highly temperature-dependent properties. These pixels can be thermally insulated from the surrounding environment and read electronically.
• Ferroelectric detectors operate close to a phase transition temperature of the sensor material.
• the pixel temperature of the detector is calibrated to a highly temperature-dependent polarization charge on the detector material.
• the achieved noise equivalent temperature difference (NETD) of ferroelectric detectors with f/1 optics and 320x240 sensors can be 70- 80 mK.
• An example of a possible ferroelectric sensor assembly consists of barium strontium titanate bump-bonded by polyimide thermally insulated connection.
• Other possible phase- change materials that can be used in infrared detectors include lanthanum barium manganite (LBMO), a metal insulator phase change material.
• microbolometer There is another possible detector that can detect small changes in the electrical resistance of a sensor material. Such a device is sometimes called a microbolometer. Microbolometers can reach NETD down to 20 mK. A typical microbolometer may include a thin film vanadium (V) oxide sensing element suspended on silicon nitride bridge above the silicon- based scanning electronics. The electric resistance of the sensing element can be measured once per frame.
• V vanadium
• thermographic camera 103 is somewhat arbitrary in the example shown in the FIG. 1.
• the thermographic camera 103 can be placed on top of a video display 107 facing a user 101.
• the thermographic camera can be mounted to a hand-held game controller or a portable game device.
• embodiments of the present invention are not limited to such configurations.
• the values of temperature (or temperature difference) at each pixel of the image sensor in the thermographic camera can be stored in a memory of the computer system 105 as an array.
• a program executed on the computer system 105 can analyze the temperature patterns in the array to distinguish a user from the user's surroundings, or the distance from the
• thermographic camera 103 to the user. Additionally, the temperature patterns in the array may be used to determine a user's respiration rate 115 as well as a user's heart rate 113.
• thermographic camera or a conventional (i.e., visible light) camera
• auxiliary sensor 109 may be used as an additional source of information, e.g., thermographic, acoustic, or video image information, that additionally characterizes the user.
• a separate visible light camera 109 may be coupled with the thermographic camera 103. The separate
• thermographic camera 103 can provide additional thermographic information that may be combined with the information from the visible light camera 109 to more accurately define the location of objects in the room and the location of a specific area of an object.
• thermographic information can be used in conjunction with acoustic information, e.g., from a microphone or microphone array, to identify which individual within a given thermographic image is a source of sounds, such as vocal sounds, whistling, coughing, or sneezing.
• FIGs. 2A and 2B illustrate examples of locating a user's position in relation to the user's surroundings using thermal infrared imaging according to an embodiment of the present invention.
• FIG. 2A illustrates an example of a system that uses thermal infrared imaging to differentiate objects from their surroundings according to an embodiment of the present invention.
• a thermographic camera 203 can be coupled to a computer processing system 205.
• the thermographic camera 203 may be positioned proximate a video display 207 that is coupled to the processing system 205 such that the user 201 faces the thermographic camera 203 when facing the display 207.
• a visible light camera 209A can be positioned to face the same direction as the thermographic camera 203.
• the visible light camera 209A can be used as an auxiliary sensor.
• the user 201 can be positioned directly in front of the background (e.g., wall) 217.
• the user 201 is wearing clothing with a similar color to that of the background 217.
• the visible light camera 209A is used to determine the physical location of all objects in its field of view.
• the software analyzing images from the visible light camera 209A is used to determine the physical location of all objects in its field of view.
• image analysis software executed by the computer processor 205 can determine that two unique objects are depicted in an image rather than just one.
• the computer processor 205 may then tag each unique object with a segmentation cue for future identification and processing.
• thermographic camera 203 Once the thermographic camera 203 has enabled the computer processor 205 to identify the location of objects within its field of view, the computer processor 205 may then be configured to continue tracking movement of the objects (e.g., hand and arm movement of a user) using the previously determined segmentation cue.
• the thermographic camera 203 When coupled with a visible light camera 209A, the thermographic camera 203 provides an additional source for tracking movement that will increase the overall accuracy of the system.
• thermographic camera 203 can be coupled to a computer processing system 205.
• the thermographic camera 203 is positioned proximate a video display 207 that is coupled to the processing system 205 such that the user 201 faces the camera 203 when facing the display 207.
• an infrared emitter and sensor 209B that is positioned to face the same direction as the thermographic camera 203, is used as an auxiliary sensor.
• the computer processing system 205 may then compare depth data obtained from the infrared emitter and sensor 209B with data obtained from thermal imaging. Finally, the computer processing system 205 may interpolate the depth of the missing parts of the object using the thermal imaging data.
• depth d of an object may be determined by use of the thermographic camera alone.
• cool air emanating from the location of the thermographic camera may be blown toward the object.
• the distance from the thermographic camera to the object could then be gauged by measuring the time needed for the thermal perturbation caused by the cool air to reach the object.
• the information extracted may include determining when a breath starts or stops, the duration of a breath, or the duration of not breathing, e.g., an interval between breaths, the orientation of the breath, and whether the breath involves inhaling or exhaling.
• determining when a breath starts or stops, the duration of a breath, or the duration of not breathing e.g., an interval between breaths, the orientation of the breath, and whether the breath involves inhaling or exhaling.
• a user's respiration rate 315 may be then used to identify many different characteristics associated with that particular user.
• the respiration rate 315 as determined by the thermographic camera 303 can be used to identify whether a particular user is speaking.
• the computer processing system 305 may recognize when a user is speaking 314 by locating irregularities in that user's respiration pattern 315.
• a particularized change in a user's respiration pattern 315 may indicate that the user is in the process of speaking 314.
• the thermographic camera 303 may be particularly useful in identifying which of those users is in the process of speech 314.
• thermographic camera 303 to detect a user's respiration rate avoids the need for more intrusive detection mechanisms (e.g., headsets, microphones). Eliminating the need for a user to strap on sensors to his body in order to provide respiratory information to a computer program will greatly enhance the user's experience and comfort level.
• thermographic camera Use of a user's respiration rate as determined by a thermographic camera to control a computer program is not limited to speech detection.
• a thermal imaging interface There are a number of different computer applications that can use a thermal imaging interface.
• computer applications that judge a user's singing by capturing sounds of that person singing with a microphone and analyzing the sounds to determine the pitch, timing, and lyrics of the singing. These characteristics can be compared to some reference parameters for the pitch, timing, and lyrics of a particular song.
• the criteria forjudging the user's singing ability may be greatly enhanced by introducing the user's breathing as a judgment parameter.
• a computer program may use information from thermographic images to determine the timing of the user's breath during the duration of the song. This may then be compared to reference parameters that will result in a more accurate determination of the user's singing ability.
• a user's respiration pattern may be used for speech recognition and security.
• a thermographic camera can be used to detect a user's respiration rate/pattern based on a user's breath.
• a computer processing system may then use this information to identify whether an image of a user's face actually corresponds to a real user by comparing the observed breathing pattern to a reference pattern.
• a particular user may be identified based on the user's individualized respiratory pattern during speech.
• the thermographic camera may act as a security barrier to prevent unauthorized access to a computer program.
• Identification of the user can be based on a combination of face identification from analysis of a visible light image, speaker identification by voice, and respiration pattern identification by analysis of the user's breath as determined from thermographic images.
• the use of combinations of different modes of user identification provide a good mechanism to counter attempts at counterfeit user identification through use of a picture and recording to cheat systems that identify the user by face and voice only.
• Analysis of the respiration pattern when one is speaking can be of particular importance to improve the identification performance.
• the thermographic camera identifies an authorized user based in whole or in part on his respiratory pattern, access to the computer program will be granted.
• the thermographic camera fails to recognize a particular thermographic image as corresponding to a real user or fails to recognize as user based on his respiratory pattern, access to the computer program will be denied.
• Such speech recognition enhancement may be implemented as a part of a first line of security or perhaps as an additional security mechanism that strengthens the existing security system in place.
• thermographic camera may use a thermographic camera to determine a user's heart rate.
• the detected heart rate can then be used as a control for a computer program.
• Fig. 4 illustrates an example of a system that can use thermal infrared imaging to identify a user's heart rate according to an embodiment of the present invention.
• thermographic camera 403 is coupled to a computer processing system 405.
• the thermographic camera 403 can be positioned proximate a video display 407 that is coupled to the processing system 405 such that the user 401 faces the camera 403 when facing the display 407.
• a visible light camera 409 that is positioned to face the same direction as the thermographic camera 403, may be used as an auxiliary sensor.
• the thermographic camera 403 obtains thermal infrared images of the user 401. These images record the blackbody radiation pattern associated with a user's body heat. From these images, the computer processing system 405 may measure a one or more vital signs 413 for one or more users, such as a user's heart beat rate, respiration rate, body temperature, or blood pressure.
• a user's heart beat rate may be measured by determining the rate or period of periodic temperature changes in the user's skin that result from the user's heart beat.
• a user's blood pressure might be estimated, e.g., from the amplitude of such periodic temperature changes.
• Respiration rate may be measured by determining the period of periodic temperature changes near the user's mouth and/or nose that result from the user's breathing.
• a user's body temperature may be determined from a long term average of temperature measurements over some relevant portion of the user's body. The temperature measurements may be subject to filtering, e.g., low pass filtering to remove rapid fluctuations over some relevant period of time.
• An additive or multiplicative offset may be applied to the long term average to account for differences between skin and core temperature.
• Embodiments of the invention are not limited to just those implementations that use the particular vital signs mentioned in the foregoing examples. It is noted that certain embodiments may use vital signs other than heart beat rate, respiration rate, body
• vital signs may be extrapolated from measurements obtained by the thermographic camera 403.
• vital signs may be extrapolated from temporal or spatial variation of temperature at specific locations or average temperature or average temperature variation over some region of a thermographic image.
• a user's vital sign 413 as a means for differentiation will act to supplement the above described methods.
• one or more users' vital signs 413 such as heart beat rate, respiration rate, body temperature, blood pressure, and the like, may be utilized where a video game or computer program requires distinguishing living objects from inanimate objects 417. Because inanimate objects 417 are not prone to periodic surface temperature changes, they can be easily distinguished from living objects 401. As such, in a room full of objects (both living and inanimate), a computer processing system 405 will be able to distinguish between the two categories by using the thermal infrared images taken by the thermographic camera 403.
• the processing system 405 can be configured to adjust a rate at which actions take place during execution of a program according to one or more users' vital signs.
• a user's heart rate may be used to control the animation of graphics 419 on the visual display 407.
• the graphics 419 on the visual display 407 may be a virtual representation (i.e., avatar) of the user 401.
• the avatar's facial expressions, emotions, or other visual manifestations of the avatar may change accordingly.
• An excited user may have a correspondingly excited avatar that acts the part (e.g., frantic movements, exaggerated facial expressions, etc.).
• a subdued user may have a correspondingly despondent avatar (e.g., muted movements, somber facial expressions, etc.).
• a user's respiration rate, blood pressure, body temperature, or other thermographically derived vital sign may be used to control computer animation in a similar fashion.
• a user's vital sign 413 may alternatively be used to control more than the animation of graphics 419 in the context of a video game.
• the vital sign may control other aspects of the video game as well, such as the pace at which the game is played.
• the game could increase or decrease in difficulty in response to the user's heart rate, respiration rate, body temperature, blood pressure, and the like.
• the game could increase in difficulty to provide a greater challenge to the user 401.
• the game may decrease in difficulty to allow for the user to catch his breath and recover.
• the processor 405 may be configured to determine a health condition of a user from results of analysis of the one or more vital signs 413 and modify a state of execution of a program according to the determined health condition.
• a health condition of a user may be determined from results of analysis of the one or more vital signs 413 and modify a state of execution of a program according to the determined health condition.
• a user's body temperature may rise during the course of exercise as the user's metabolism increases.
• the processing system 405 may be able to identify users with uncharacteristically high or low body temperature (gauged, e.g., by the user's normal temperature or by expected
• temperatures for a typical person which could indicate a fever or other health condition. That information could be used as an input and/or used to modulate the action going on in the game.
• the system 400 may modify a state of execution of a program running on the processing system 405 in other ways in response to the determined health condition of the user 401.
• the scene displayed on the display 407 e.g., a game scene, television program, or video
• the volume and nature of sound presented in conjunction with the scene displayed may similarly be modified.
• Vital signs such as heart beat or respiration rate can also be used to control how background sounds or music is played.
• the firing rate can be increased or decreased with changes in the user's heart beat or respiration rate.
• a fixed number of bullets e.g., three
• the vital sign can be mapped to trigger an action of a virtual object such as a missile or projectile that appear during game play.
• the vital sign can determine how fast a missile rotates while moving forward.
• the processing system 405 can synchronize the tempo of background music to the user's vital sign, such as heart beat or respiration rate.
• the thermal characteristic detector unit 518 can include a sensor that is sensitive to spatial and temporal variations in temperature sufficient to distinguish a user's heart rate, respiration rate, and location from measured temperature variations.
• the thermal characteristic detector unit 518 includes a thermographic camera, which may operate as described above with respect to FIG. 1.
• the values of temperature (or temperature difference) at each pixel of an image sensor in the thermographic camera can be stored in the memory 505 as an array.
• the program 503 can analyze the temperature patterns in the array to identify a user's location and to identify thermal patterns that correspond to a user's heart rate and respiration rate. This information may then be used as a control input.

Landscapes

• Engineering & Computer Science (AREA)
• Theoretical Computer Science (AREA)
• General Engineering & Computer Science (AREA)
• Physics & Mathematics (AREA)
• General Physics & Mathematics (AREA)
• Human Computer Interaction (AREA)
• Computer Security & Cryptography (AREA)
• Multimedia (AREA)
• Toxicology (AREA)
• Signal Processing (AREA)
• Health & Medical Sciences (AREA)
• Computer Hardware Design (AREA)
• Software Systems (AREA)
• Electromagnetism (AREA)
• Radar, Positioning & Navigation (AREA)
• Remote Sensing (AREA)
• Measuring And Recording Apparatus For Diagnosis (AREA)
• Radiation Pyrometers (AREA)
• Position Input By Displaying (AREA)

Priority Applications (3)

Applications Claiming Priority (6)

Publications (1)

Family

ID=45870267

Family Applications (1)

Country Status (5)

Cited By (1)

Families Citing this family (82)

Citations (8)

Family Cites Families (38)

• 2010
  • 2010-12-15 US US12/969,501 patent/US8638364B2/en active Active
• 2011
  • 2011-09-19 EP EP11827310.1A patent/EP2606638A4/en not_active Withdrawn
  • 2011-09-19 WO PCT/US2011/052182 patent/WO2012040114A1/en not_active Ceased
  • 2011-09-19 CN CN201310454386.9A patent/CN103500008B/zh active Active
  • 2011-09-19 CN CN201180047201.1A patent/CN103141080B/zh active Active
  • 2011-09-19 JP JP2013530212A patent/JP5797757B2/ja active Active

Patent Citations (8)

Non-Patent Citations (1)

Also Published As

Similar Documents

Legal Events