KR102418369B1 - Device attachment with dual band imaging sensor - Google Patents

Abstract

Various techniques are disclosed for providing an attachment device configured to removably attach and provide infrared imaging functionality to a mobile phone or various portable electronic devices. Such an attachment device may include an infrared imaging module and a non-thermal imaging module that cooperates with one or more of an infrared imaging module and a non-thermal imaging module in the attachment device and a light source in the attachment device to capture and process an image.

Description

DEVICE ATTACHMENT WITH DUAL BAND IMAGING SENSOR

The present application for the present invention is based on the priority claim of U.S. Provisional Application No. 61/923,732 filed on January 5, 2014 under the title of "DEVICE ATTACHMENT WITH DUAL BAND IMAGING SENSOR" use the

The present application is a CIP application of U.S. Application No. 4/281,883 filed on May 19, 2014 under the title "DEVICE ATTACHMENT WITH INFRARED IMAGING SENSOR", the entirety of which is incorporated herein by reference.

The present application is a CIP application of international application PCT/US2013/062433 filed on September 27, 2013 under the name of "DEVICE ATTACHMENT WITH INFRARED IMAGING SENSOR", and the entirety of which is incorporated herein by reference.

International application PCT/US2013/062433 is based on the priority claim of U.S. Provisional Application No. 61/880,827, filed on September 20, 2013 under the title "DEVICE ATTACHMENT WITH INFRARED IMAGING SENSOR," use the whole

International application PCT/US2013/062433 is a CIP application of US Application No. 13/901,428 filed on May 23, 2013 under the title of "DEVICE ATTACHMENT WITH INFRARED IMAGING SENSOR", the entirety of which is incorporated herein by reference. do.

U.S. Application No. 13/901,428 is based on U.S. Provisional Application No. 61/652,075, filed May 25, 2012, entitled "DEVICE ATTACHMENT WITH INFRARED IMAGING SENSOR," as the basis for claiming priority, and use the whole

U.S. Application No. 13/901,428 is a CIP application of U.S. Application No. 29/423,027, filed May 25, 2012 under the title "DEVICE ATTACHMENT WITH CAMERA," and is incorporated herein by reference in its entirety.

The present application is a CIP application of PCT/US2013/78551 filed on December 31, 2013 under the name of "INFRARED IMAGING DEVICE HAVING A SHUTTER", and the entirety of which is incorporated herein by reference.

International application PCT/US2013/78551 is based on the priority claim of US Provisional Application 61/747,789 filed on December 31, 2013 under the title "INFRARED IMAGING DEVICE HAVING A SHUTTER" wish to

International application PCT/US2013/78551 is a CIP application of US application 13/966,052 filed on August 13, 2013 under the title "INFRARED CAMERA SYSTEM HOUSING WITH METALIZED SURFACE" use the whole

U.S. Application No. 13/966,052 claims priority to U.S. Provisional Application No. 61/683,124, filed on August 14, 2012, entitled "INFRARED CAMERA SYSTEM HOUSING WITH METALIZED SURFACE" , and use it in its entirety.

The present application is a CIP application of PCT/US2014/59200 filed on October 3, 2014 under the name of "DURABLE COMPACT MULTISENSOR OBSERVATION DEVICES", the entirety of which is incorporated herein by reference.

International application PCT/US2014/59200 is a CIP application of US Application No. 14/101,245, filed December 9, 2013, entitled "LOW POWER AND SMALL FORM FACTOR INFRARED IMAGING," use the whole

U.S. Application 14/101,245 is a continuation of International Application PCT/US2012/041744, filed June 8, 2012, entitled "LOW POWER AND SMALL FORM FACTOR INFRARED IMAGING," which use the whole

International application PCT/US2012/041744 provides the basis for claiming priority to U.S. Provisional Application No. 61/656,889, filed on June 7, 2012, entitled "LOW POWER AND SMALL FORM FACTOR INFRARED IMAGING" and use the whole.

International application PCT/US2012/041744 gives priority to U.S. Provisional Application No. 61/545,056, filed on October 7, 2011 under the title "NON-UNIFORMITY CORRECTION TECHNIQUES FOR INFRARED IMAGING DEVICES" It serves as the basis for the argument, and uses it in its entirety.

International application PCT/US2012/041744 is based on U.S. Provisional Application No. 61/495,873, filed on June 10, 2011 under the title "INFRARED CAMERA PACKAGING SYSTEMS AND METHODS," use the whole

International application PCT/US2012/041744 is based on US Provisional Application No. 61/495,879 filed on June 10, 2011 under the name of "INFRARED CAMERA SYSTEM ARCHITECTURES" as the basis for claiming priority, and is incorporated herein by reference in its entirety. .

International application PCT/US2012/041744 is based on U.S. Provisional Application No. 61/495,888 filed on June 10, 2011 under the title of "INFRARED CAMERA CALIBRATION TECHNIQUES" as the basis for claiming priority, and is incorporated herein by reference in its entirety. .

International Application PCT/US2014/59200 is a CIP Application of U.S. Application No. 14/099,818, filed December 6, 2013, entitled "NON-UNIFORMITY CORRECTION TECHNIQUES FOR INFRARED IMAGING DEVICES" , and use it in its entirety.

U.S. Application 14/099,818 is a continuation of International Application PCT/US2012/041749, filed June 8, 2012, entitled "NON-UNIFORMITY CORRECTION TECHNIQUES FOR INFRARED IMAGING DEVICES" , and use it in its entirety.

International application PCT/US2012/041749, entitled "NON-UNIFORMITY CORRECTION TECHNIQUES FOR INFRARED IMAGING DEVICES," gives priority to U.S. Provisional Application No. 61/545,056, filed on October 7, 2011 It serves as the basis for the argument, and uses it in its entirety.

International application PCT/US2012/041749 is based on US Provisional Application No. 61/495,873, filed on June 10, 2011 under the title "INFRARED CAMERA PACKAGING SYSTEMS AND METHODS," use the whole

International application PCT/US2012/041749 is based on US Provisional Application No. 61/495,879 filed on June 10, 2011 under the name of "INFRARED CAMERA SYSTEM ARCHITECTURES" as the basis for claiming priority, and is incorporated herein by reference in its entirety. .

International application PCT/US2012/041749 is based on US Provisional Application No. 61/495,888 filed on June 10, 2011 under the name of "INFRARED CAMERA CALIBRATION TECHNIQUES" as the basis for claiming priority, and is incorporated herein by reference in its entirety. .

International application PCT/US2012/041749 is a CIP application of US Application No. 14/101,258 filed on December 9, 2013 under the name of "INFRARED CAMERA SYSTEM ARCHITECTURES", the entirety of which is incorporated herein by reference.

U.S. Application 14/101,258 is a continuation of International Application PCT/US2012/041739, filed on June 8, 2012, entitled "INFRARED CAMERA SYSTEM ARCHITECTURES," which is incorporated herein by reference in its entirety.

International application PCT/US2012/041739 is based on U.S. Provisional Application No. 61/495,873, filed on June 10, 2011 under the title "INFRARED CAMERA PACKAGING SYSTEMS AND METHODS," use the whole

International application PCT/US2012/041739 is based on U.S. Provisional Application No. 61/495,873, filed on June 10, 2011 under the title "INFRARED CAMERA PACKAGING SYSTEMS AND METHODS," use the whole

International application PCT/US2012/041739 is based on US Provisional Application No. 61/495,888 filed on June 10, 2011 under the title of "INFRARED CAMERA CALIBRATION TECHNIQUES" as the basis for claiming priority, and is incorporated herein by reference in its entirety. .

International application PCT/US2014/59200 is a CIP application of U.S. Application No. 14/138,058, filed December 21, 2013, entitled "COMPACT MULTI-SPECTRUM IMAGING WITH FUSION," in its entirety. use the

U.S. Application No. 14/138,058 claims priority to U.S. Provisional Application No. 61/748,018, filed December 31, 2012, entitled "COMPACT MULTI-SPECTRUM IMAGING WITH FUSION," use the whole

The present application is a CIP application in U.S. Application No. 14/299,987, filed June 9, 2014, entitled “INFRARED CAMERA SYSTEMS AND METHODS FOR DUAL SENSOR APPLICATIONS,” in its entirety. use the

U.S. Application 14/299,987 is a CIP Application of U.S. Application Serial No. 12/477,828, filed June 3, 2009, entitled “INFRARED CAMERA SYSTEMS AND METHODS FOR DUAL SENSOR APPLICATIONS,” use the whole

International application PCT/US2014/59200 is a CIP application of US Application No. 14/138,040 filed on December 21, 2013 under the title "TIME SPACED INFRARED IMAGE ENHANCEMENT", the entirety of which is incorporated herein by reference.

U.S. Application No. 14/138,040 claims priority to U.S. Provisional Application No. 61/792,582, filed March 15, 2013, entitled “TIME SPACED INFRARED IMAGE ENHANCEMENT,” and is incorporated herein by reference in its entirety. do.

U.S. Application No. 14/138,040 claims priority to U.S. Provisional Application No. 61/746,069, filed December 26, 2012, entitled "TIME SPACED INFRARED IMAGE ENHANCEMENT," and is incorporated herein by reference in its entirety. do.

International Application PCT/US2014/59200 is a CIP application of U.S. Application No. 14/138,052, filed December 21, 2013, entitled "INFRARED IMAGING ENHANCEMENT WITH FUSION," which is incorporated herein by reference in its entirety. .

U.S. Application No. 14/138,052 claims priority to U.S. Provisional Application No. 61/793,952, filed March 15, 2013, entitled "INFRARED IMAGING ENHANCEMENT WITH FUSION," wish to

U.S. Application No. 14/138,052 also claims priority to U.S. Provisional Application No. 61/746,074, filed December 26, 2012, entitled "INFRARED IMAGING ENHANCEMENT WITH FUSION," use the

The present application is a CIP application of US Application No. 14/246,006 filed on April 4, 2014 under the name "SMART SURVEILLANCE CAMERA SYSTEMS AND METHODS", the entirety of which is incorporated herein by reference.

U.S. Application 14/246,006 is a CIP application of U.S. Application No. 13/437,645, filed April 2, 2012, entitled “INFRARED RESOLUTION AND CONTRAST ENHANCEMENT WITH FUSION,” in its entirety. wish to

U.S. Application 13/437,645 is a CIP application of U.S. Application No. 13/105,765, filed May 11, 2011, entitled “INFRARED RESOLUTION AND CONTRAST ENHANCEMENT WITH FUSION,” in its entirety. wish to

U.S. Application 13/437,645 claims priority to U.S. Application No. 61/473,207, filed April 8, 2011, entitled "INFRARED RESOLUTION AND CONTRAST ENHANCEMENT WITH FUSION," , use the whole.

U.S. Application 13/437,645 claims priority to U.S. Application Serial No. 12/766,739, filed April 23, 2010, entitled "INFRARED RESOLUTION AND CONTRAST ENHANCEMENT WITH FUSION," , and use it in its entirety.

U.S. Application 13/105,765 is a continuation of International Application No. PCT/EP2011/056432, filed April 21, 2011, entitled "INFRARED RESOLUTION AND CONTRAST ENHANCEMENT WITH FUSION," use the whole

US Application 13/105,765 is a CIP application of US Application 12/766,739, which is incorporated herein by reference in its entirety.

International application PCT/EP2011/056432 is a CIP application of US application 12/766,739, and uses the whole.

International application PCT/EP2011/056432 uses US Provisional Application 61/473,207 as a basis for claiming priority, and uses the entirety of US Provisional Application No. 61/473,207.

The present application is a CIP application of U.S. Application No. 14/029,716, filed September 17, 2013, entitled "ROW AND COLUMN NOISE REDUCTION IN THERMAL IMAGES," which is incorporated herein by reference in its entirety.

U.S. Application No. 14/029,716 is based on U.S. Provisional Application No. 61/745,489, filed December 21, 2012, entitled “ROW AND COLUMN NOISE REDUCTION IN THERMAL IMAGES,” as the basis for claiming priority, and use the whole

U.S. Application 14/029,716 claims priority to U.S. Provisional Application No. 61/745,504, filed December 21, 2012, entitled "PIXEL-WISE NOISE REDUCTION IN THERMAL IMAGES," , and use it in its entirety.

U.S. Application No. 14/029,716 is a CIP application of U.S. Application No. 13/622,178, filed September 18, 2012, entitled "SYSTEMS AND METHODS FOR PROCESSING INFRARED IMAGES," which is incorporated herein by reference in its entirety. wish to

U.S. Application 13/622,178 is a CIP application of U.S. Application No. 13/529,772, filed on June 21, 2012, entitled "SYSTEMS AND METHODS FOR PROCESSING INFRARED IMAGES," and is incorporated herein by reference in its entirety. wish to

U.S. Application 13/529,772 is a continuation of U.S. Application Serial No. 12/396,340, filed March 2, 2009, entitled "SYSTEMS AND METHODS FOR PROCESSING INFRARED IMAGES," and is incorporated herein by reference in its entirety. wish to

One or more embodiments of the present invention relate to infrared imaging apparatuses, and more particularly to infrared imaging apparatuses for portable devices, and to systems and methods for multi-spectral imaging using infrared imaging apparatuses.

Various types of portable electronic devices, such as smart phones, cell phones, tablet devices, portable media players, portable game devices, digital cameras, and laptop computers, are widely used. These devices include a visible light image sensor or camera that allows the user to take still pictures or video clips.

One reason for increasing the popularity of such embedded cameras is the ubiquitous nature of mobile phones and other portable electronic devices. That is, users carry mobile phones and other portable electronic devices with them, and such a built-in camera is always at hand when the user needs it.

Another reason for increasing popularity is that, using mobile phones and other portable electronic devices, increased processing power, storage capacity, and/or display capabilities exist to allow for sufficiently fast capturing, processing, and quality image storage. .

However, the image sensors used in these portable electronic devices are usually CCD-based or CMOS-based sensors and have limited visible light image capturing. Thus, these sensors can only detect a very limited range of visible light or wavelengths that are close to visible light (eg, the near-infrared spectrum when an object is illuminated with light in near-infrared). Consequently, there is a need for a technique for providing infrared imaging capabilities in a portable electronic device form factor.

Various techniques are disclosed to provide an attachment device configured to be removably attached to a mobile phone or other portable electronic device to provide infrared imaging functionality. For example, the attachment device may be configured to at least partially receive a user device and be disposed within the housing, the housing having a partial enclosure (eg, a tub or cutout) at a rear surface thereof, the infrared image data and visible light image data. a multi-wavelength image sensor configured to capture data, and a processing module communicatively coupled to the multi-wavelength sensor assembly and configured to transmit infrared image data and/or visible light image data to the user device.

The attachment device is configured to cooperate with one or more components of an attached device, such as a smartphone, to capture and process image data. For example, an additional visible light camera may be used in a smart phone attached with an attachment device to capture additional visible light images that may be used in conjunction with a visible light image captured using a visible light image sensor within the attachment device; Use the parallax of the object between the two visible light image sensors to measure the distance to the object in the scene.

The measured distance can be used to align and combine the visible light image from the visible light imaging module and the infrared image from the infrared image sensor. As another example, a smart phone attached to the attachment device may be operable to illuminate some or all of the scene to be imaged by an imaging module within the attachment device for use in the combined infrared and visible light images.

A timer is used to determine when the thermal imaging module in the attachment device is used to determine the calibrated temperature of the imaged object.

The scope of the invention is defined by the claims. A more complete understanding of the practice of the present invention may be provided from the following detailed description of one or more embodiments. A description of the drawings is first briefly provided.

1 is a front perspective view of an attachment device according to an embodiment of the present invention;
2 is a view showing a slider module of an attachment device according to an embodiment of the present invention;
2 is a rear perspective view of an attachment device according to an embodiment of the present invention;
4 illustrates an attachment apparatus and an attached device showing how the attachment apparatus and non-thermal image data from the attached device integrate non-thermal and thermal image data from the attachment apparatus in accordance with an embodiment of the present invention; drawing.
5 is a flow diagram of various operations for using non-thermal image data from an attachment device and an attached device in passing through non-thermal and thermal image data from the attachment device in accordance with an embodiment of the present invention.
6 illustrates various operations for calibrating non-thermal image data from an attachment device and an attached device for later use in integrating non-thermal and thermal image data from an attachment device in accordance with an embodiment of the present invention. flow chart.
7 is a flow diagram of various operations for using time after calibration to determine whether a calibrated image-based temperature can be determined in accordance with an embodiment of the present invention.
8 is a flowchart illustrating various operations for improving imaging of a scene according to an embodiment of the present invention.
9 is a flowchart of various operations for improving imaging of a scene based on a user input according to an embodiment of the present invention;
Embodiments of the present invention and their advantages will be better understood from the following detailed description. In various drawings, like parts are denoted by like reference numerals.

1 and 2 , various views are shown in which an attachment device 1250 having an infrared imaging module 7000 and a non-thermal camera module 7002 is shown. An infrared image sensor, such as the infrared imaging module 7000, can capture images of thermal energy radiation emitted from any object having a temperature above absolute zero degrees Celsius, and thus to detect an abnormality in a person's body temperature (eg, a disease). ), infrared inspection provides useful information, such as detecting invisible gases, inspecting leaky structures and damaged insulation, and allowing viewing in low or no light intensity conditions. It can be used to generate an infrared image (eg, a thermogram) that can be useful in a situation.

Attachment device 1250 may be configured to receive a suitable electronic device, such as user device 1200 . In the embodiment of FIG. 1 , a rear perspective view of an attachment device is configured to receive an Apple device (smartphone) 1200 (eg, an iPhone™ device, an iPad™ device, or an iPod Touch™ device). show However, such an implementation is for illustrative purposes only. This attachment device 1250 may be a device from Samsung Electronics (eg, a Galaxy Tab™ device, another Galaxy™ device, or another device from Samsung Electronics) or a smartphone, tablet or portable device from another manufacturer, as needed. It may have a shape suitable for accommodating an electronic device.

As shown in FIG. 1 , the attachment device 1250 includes a camera window 1240 through which a device camera 101 (eg, a non-thermal camera module, such as a visible light camera module) may capture an image. A device light source 103 (eg, a camera flash or flash light) may illuminate some or all of the scene, and other sensors 105 of the device 1200 may receive or emit light. have. The attachment device 1250 includes a number of imaging components, such as an infrared imaging module 7000 and a non-thermal camera module 7002 , such as one or more internal electronic components such as a battery 1208 or a processor. other internal components, or communication components (as an example).

If desired, attachment device 1250 may also include a mechanical shutter, such as a user-operable shutter. The user-operable shutter may be moved by the user of the attachment device such that by sliding the button 7004 it may selectively block or not block the imaging components 7000 , 7002 with an internal shutter member attached to the button 7004 . have.

2 is a perspective view of a shutter member 250 having openings 252 , 254 and a slider assembly 248 having a button 7004 . Button 7004 may be used to push shutter member 250 along the direction indicated by arrow 256 to selectively move opening 252 , 254 in front of imaging module 7000 , 7002 of FIG. 1 . have. When the opening 252, 254 is in front of the imaging module 7000, 7002, the imaging module 7000, 7002 may receive light from the scene through the opening 252, 254 for an image capture operation. . When the button 7004 is moved so that a portion of the shutter member 250 blocks the imaging module 7000, 7002, it is blocked from reaching the ray imaging module 7000, 7002 from the scene. In an embodiment, button 7004 is configured to power or de-energize attachment device 1250 and moves shutter member 250 to block or not block imaging component 7000 , 7002 .

In an embodiment, a shutter member is used to protect the imaging component 7000 when not in use. Shutter 250 may also provide a calibration process for infrared imaging module 7000 (eg, U.S. Application 14/099,818, filed December 6, 2013, entitled 'Radiation Calibration Process'), as will be appreciated by those skilled in the art. It may also be used as a temperature reference as part of a non-uniformity correction (NUC) process or other calibration process, as described in . The attachment device 1250 includes a front portion 7007 and a rear portion 7009 . Front portion 7007 is formed from a housing that contains functional components of an attachment device, such as a battery, connector, imaging component, processor, memory, communication component, and/or other components of the attachment device described herein. The back portion 7009 may be a housing portion having a shape that defines a recess configured to allow a user device 1200 to be removably attached thereto.

3 shows how a user device 1200 from Apple Corporation having a display 201 by inserting the device into a recess in the housing for an attachment arrangement formed from at least one side wall and a rear wall at least partially surrounding the device. It is a front perspective view of the attachment device of FIG. 1, showing whether it is detachably attached.

The attachment device 1250 includes a device connector 8012 that, when attached, sends and receives various signals and/or electrical power to and from the user device 1200 . The device connector is positioned in a position properly aligned with a corresponding device connector receptacle or socket of the user device 1200 so that when the attachment apparatus 1250 is attached to the user device 1200 , the device connector engages the corresponding device connector receptacle or socket of the user device 1200 . to allow the device to be tethered to a connector receptacle or socket. For example, once the user device 1200 is equipped with a connector receptacle on the bottom side surface, the device connector can be positioned in a suitable location on the bottom side wall of the attachment device 1250 . The device connector may also include a mechanical fastener (eg, a lock/latch connector plug) used to support and align the user device 1200 .

Device connector 8012 may be implemented according to connector specifications associated with the type of user device 1200 . For example, a device connector can be a connector (Apple® dock connector or Lightning™ connector for iPod™ and iPhone™) or a standard connector (e.g., Universal Serial Bus (USB) connector in several versions, Portable Digital Media Interface (PDMI) , or other standard connector provided by the user device).

In one embodiment, the device connector is provided to be replaceable so that the attachment apparatus 1250 can accommodate different types of user devices that accept different device connectors. For example, various types of device connector plugs may be provided and configured to be attached to the base connector attachment apparatus 1250 , and a connector plug compatible with the user device 1200 may attach the attachment apparatus 1250 to the user device 1200 . Make sure it can be attached to the base connector before attaching it to the In another embodiment, the device connector may be provided in a fixed manner.

The attachment device 150 may also communicate with the user device 1200 via a wireless connection. In this regard, the attachment device 1250 may include a wireless communication module configured to enable wireless communication between the user device 1200 and the attachment device 1250 . In various embodiments, the wireless communication module may support the IEEE 802.11 WiFi standard, the Bluetooth™ standard, the ZigBee™ standard, or other suitable short-range wireless communication standard. Accordingly, the attachment apparatus 1250 can be used with the user device 1200 without relying on a device connector if a connection through a device connector is not available or desirable.

The infrared imaging module 700 has a small form factor and is implemented according to wafer level packaging technology or other packaging technology. The infrared imaging module 7000 includes a lens barrel, a housing, an infrared ray assembly, a circuit board, a base, and a processing module.

The infrared sensor assembly includes a plurality of infrared sensors (eg, infrared detectors) embodied in an array or other form on a substrate and covered by a cap. For example, in one embodiment, the infrared sensor assembly may be implemented as a focal plane array (FPA). Such focal plane arrays may be implemented as, for example, vacuum package assemblies.

In one embodiment, the infrared sensor assembly is implemented as a wafer level package (eg, singulated from a set of vacuum package assemblies provided on a wafer). In one embodiment, the infrared sensor assembly is implemented to operate using a power source of 2.4 volts, 2.5 volts, or similar voltage.

Infrared sensors within the infrared imaging module 7000 may include infrared radiation (e.g., infrared radiation) from the target scene, including medium-wave infrared wavelength bands (MWIR), long-wave infrared wavelength bands (LWIR), and/or other thermal imaging bands as required in a particular implementation. , infrared energy). The infrared sensor is embodied as an infrared thermal imager or other type of thermal imaging infrared sensor arranged in a desired array pattern to provide a plurality of pixels.

User device 1200 communicates with attachment device 1250 to receive infrared images and/or non-thermal images captured by infrared sensor assembly 7000 , such as visible light images, from non-thermal imaging module 7002 . It is a portable electronic device configured to do so.

Non-thermal image data, such as infrared image data captured by infrared imaging module 7000 and/or visible image data captured by non-thermal imaging module 7002, can be transferred to attachment device 1250 and / or provided to the processing module of the device 1200 .

The processing module is configured to perform appropriate processing of the captured infrared image data, and transmit the unprocessed and/or processed infrared image data to the user device 1200 . For example, when attachment apparatus 1250 is attached to user device 1200 , the processing module may wirelessly transmit raw and/or processed infrared image data via a wired device connector or via an appropriate wireless component to the user device. (1200).

Thus, for example, the user device 1200 displays a user-viewable infrared image (eg, a thermogram) to the user on the display 201 and allows the user to display infrared image data, non-thermal image data, multi- appropriate to receive infrared image data (eg, thermal image data) and/or non-thermal image data from attachment device 1250 to allow storage of wavelength image data, and/or user-viewable infrared images. is composed

That is, the user device 1200 may use appropriate software instructions (eg, smart phone "application"). Attachment device 1250 and user device 1200 may be configured to perform other infrared imaging functions, such as storing and analyzing thermographic data (eg, temperature information) contained within infrared image data.

The attachment device 1250 may also include a battery 1208 (see FIG. 1 ). The battery 1208 is configured to be used as a power source for the internal components of the attachment device 1250 , such that it does not deplete the battery of the user device 1200 when attached. Also, the battery 1208 of the attachment device 1250 may be configured to provide power to the user device 1200 via, for example, a device connector.

Accordingly, the battery 1208 can provide backup power from which the user device 1200 is charged. Conversely, the various components of the attachment device 1250 may be disconnected from the battery of the user device 1200 (e.g., may be configured to use power (via the device connector).

In some embodiments, the non-thermal camera module 101 of the device 1200 may be used with the non-thermal camera module 7002 of the attachment apparatus 1250 . When merging infrared (eg, thermal) and non-thermal (eg, visible) video images, the two images may be mapped pixel-by-pixel to each other. Differences between the two cameras (eg, distortion, parallax, orientation angle, etc.) may be compensated for. Imaging modules 7000 and 7002 may be mounted close to each other to reduce parallax differences between images captured using the imaging module. In particular, the non-thermal camera 101 in the device 1200 is used in conjunction with the non-thermal camera module 7002 to provide correction for any remaining parallax differences, for very close objects in the image, You can determine the distance to an object in the scene. The determined distance can then be used to adjust the alignment of infrared (eg, thermal) and non-thermal (eg, visible) video images even at variable scene distances.

As shown in FIG. 4 , each of the non-thermal camera module 7002 and the non-thermal camera module 101 sends a non-thermal (eg, visible light) image of the scene to processing circuitry, such as a distance measurement engine 301 . can provide The distance measurement engine 301 compares the measured shift of the position of the scene object in the images provided by the non-thermal camera module 7002 and the non-thermal camera module 101 with the non-thermal camera module 7002 . - The known distance D between the thermal camera modules 101 can be used to determine the distance to the scene object.

The measurement distance, the non-thermal image captured by the non-thermal imaging module 7002 , and the thermal image (eg, infrared image) from the thermal imaging module 7000 are processed by a merge engine 303 . may be provided in the circuit. The merging engine 303 may use the measurement distance to correct for any parallax differences between the thermal image and the non-thermal image, so that the thermal image and the non-thermal image are combined and presented to the display 201 for display to the user. can be The distance measurement engine 301 and the merging engine 303 may represent algorithms performed by logic devices.

5 uses a non-thermal imaging module in the attachment device and a non-thermal imaging module in the attachment device to provide parallax correction of images captured by the thermal imaging module in the attachment device and the non-thermal imaging module in the attached device. It is a flowchart of example operations for doing so.

At block 400 , a first non-thermal image may be captured using a non-thermal image sensor in the attachment device, and, optionally, a thermal image may be captured using a thermal image sensor in the attachment device. .

At block 402 , a second non-thermal image may be captured using a non-thermal image sensor in the camera device.

At block 404 , (eg, determining a parallax-induced shift of the object in the first and second non-thermal images to compare the known relative positions of the non-thermal image sensor within the device and the non-thermal image sensor within the attachment device) , by triangulating the distance to the object using the determined spit) the first and second non-thermal images can be used to determine the distance to the scene object. The known relative position can be determined based on the known position of the device within the attachment apparatus and the known position of the non-thermal image sensor within the respective device, and/or using both non-thermal image sensors to image the object at a known distance. can be determined, based on a corrective action performed by capturing, and by determining the relative position of the non-thermal image sensor using an image of the object and a known distance.

In some embodiments, the capture of the non-thermal image may be controlled to improve the accuracy of determining the parallax-induced shift between the first and second non-thermal images, which of course improves the accuracy of the determined distance and parallax correction. will do For example, if first and second non-thermal images are captured when the user and/or object is in motion, the accuracy of the parallax-induced shift determination will depend on the shaking and/or shift of the object in the image caused by the motion. understanding may be affected. Such motion-induced shifts may occur, for example, if the capture timings by the first and second non-thermal image sensors are not properly synchronized.

Accordingly, in one embodiment, operation of FIG. 5 includes detecting movement of a device and/or attachment device (eg, by an accelerometer or other type of motion detector provided in the device and/or attachment device) and processing the captured image as understood by those skilled in the art) may involve the detection of movement of a target object in the scene. In this embodiment, the non-thermal image may be captured when the motion being captured is below a desired threshold and/or when the captured image is synchronized to obtain a non-thermal image that is less affected by the motion.

In another embodiment, the actuation of FIG. 5 may actuate a light source (eg, light source 103 of user device 1200 ) to flash (eg, illuminate for a short period of time) all or part of a scene, while activating first and second 2 may involve capturing a plurality of frames of non-thermal images by a non-thermal image sensor. Frames captured by the first non-thermal image sensor may be processed to detect and select a frame with an image of the flashed scene. Likewise, a frame with an image of a flashed scene (eg, a frame at the beginning, middle, or end of a flash event) can be detected and selected. For example, in this way, when a scene is illuminated, selected frames can be substantially synchronized to that instant (or some moment in time), thus reducing the effect of motion-induced shifting (that ) (eg, sufficient synchronization of captured images can be realized).

At block 406 , the thermal image and the first non-thermal image are to be combined using the determined distance to the object (eg, by performing parallax correction between the thermal image and the first non-thermal image using the determined distance). can

Any distortion and misalignment errors between the non-thermal camera module in the attachment device and the non-thermal camera module in the device to improve parallax correction determined using the non-thermal camera module in the attachment device and the non-thermal camera module in the device. can be calibrated. For example, an image of an object, such as a hand, is captured in front of a thermally and visually uniform background using a non-thermal camera module in the attachment device, a thermal imaging module in the attachment device, and a non-thermal camera module in the device. can be Processing circuitry (eg, a smartphone app running on the device processor) can be used to match the edges of the hand in all three images, and to match the alignment between the two non-thermal camera modules with the non-thermal camera module in the attachment device. It can be correlated to the factory-factory calibration alignment between the thermal imaging modules within the attachment device.

6 is a flowchart of operations for calibrating distortion and/or alignment between a non-thermal camera module in an attachment device and a non-thermal camera module in a device.

At block 500 , the image is imaged of an object (eg, a hand) at a common time using each of a thermal image sensor in the attachment device, a non-thermal image sensor in the attachment device, and a non-thermal image sensor in the attached device. can be captured.

At block 502 , an edge of an object in each captured image may be detected.

At block 504 , alignment and distortion correction between the non-thermal image sensor in the attachment apparatus and the non-thermal image sensor in the attached device may be determined based on the location in the image of the edge being detected.

At block 506 , alignment and distortion correction is performed (eg, to the attachment device or to the device) for use in distance measurements for parallax correction between images captured using a thermal image sensor and a non-thermal image sensor in the attachment device. ) can be saved.

Although the various embodiments illustrated above with reference to FIGS. 5 and 6 have been described with respect to the use of non-thermal image sensors in user devices (eg, phone cameras) and thermal image sensors and non-thermal image sensors in attachment devices, The principles and teachings of the present invention may be applied to any other suitable combination of image sensors in a user device and/or attachment device. For example, a thermal image sensor may be provided in addition to or alternatively to the user device, using a non-thermal image sensor of the user device and a non-thermal image sensor of an attachment apparatus for the user device, Parallax correction may be provided between the image sensor and a non-thermal image sensor of the user device or attachment device. In another example, in the case of a user device that may include two or more non-thermal image sensors (eg, for stereoscopic imaging or other uses), using the non-thermal image sensor of the user device, the device (and/or or provide parallax correction to the image sensor of the attachment device, if present.

In some embodiments, the thermal imaging module 7000 is used in some embodiments (eg, by capturing one or more calibrated thermal images, and by determining the temperature of the object from the intensity and/or spectrum of the object in the thermal image, at the level of the skilled artisan). can be used to determine the image-based calibration temperature of an object. The accuracy of this type of image-based temperature measurement can be improved by ensuring that the thermal imaging module has been calibrated recently when the image-based temperature measurement is made.

7 is a flow chart of exemplary operations to ensure that the thermal imaging module has been recently calibrated when an image-based temperature measurement is taken.

At block 600 , a system, such as a system including a thermal image sensor and an attachment device having an attached device, is configured to: (eg, by closing a shutter, and capturing one or more images of the shutter using the thermal image sensor) ) a closed shutter can be used to perform calibration of thermal image sensors, such as thermal image sensors in attachments.

At block 602 , the system may monitor (eg, by a processor in the attachment apparatus or a processor in the attached device) time since the last calibration of the thermal image sensor.

At block 604 , the system may receive an image-based temperature determination request from a user.

At block 606 , the system may determine whether the elapsed time since calibration is less than the maximum allowable time using the monitored time. The maximum allowable time may be, for example, less than 20 seconds since the last calibration, less than 10 seconds since the last calibration, less than 1 minute since the last calibration, or less than 30 seconds since the last calibration. In response to determining that the elapsed time since the last calibration is less than the maximum allowable time, the system may proceed to block 608 .

At block 608 , one or more thermal images and/or infrared spectra of the object may be captured.

At block 610 , the system may determine the temperature of the object from the thermal image and/or from the infrared spectrum.

If it is determined at block 606 that the elapsed time since the last calibration is greater than the maximum allowable time, then the system may proceed to block 612 .

At block 612 , the system may instruct the user to perform a new calibration of the thermal imaging module using the closed shutter to ensure that subsequent temperature measurements are accurate.

In embodiments, a light source in a portable electronic device attached to an attachment device having a thermal imaging module may be used in conjunction with a thermal imaging module and a non-thermal imaging module to enhance imaging of a scene. For example, light source 103 (see FIG. 1 ) of device 1200 is used to illuminate a portion of a scene with a spectrum sensed by one or more imaging modules 7000 , 7002 , 101 . Lights 103 may be flashed or operated in a flash light mode to illuminate all or part of the scene while image capture operates using imaging modules 7000 and 7002 . Light source 103 may be turned on or flashed in response to user input, or may be automatically turned on or flashed based on a light level determined using imaging module 7002 , device camera 101 , or other light sensor.

8 depicts a flow diagram of various operations for enhancing the imaging of a scene and active lighting of a scene using thermal images in accordance with one embodiment.

At 800 , thermal image data may be captured using a thermal image sensor in the attachment device and non-thermal image data may be captured using a non-thermal image sensor in the attachment device. If desired, additional non-thermal image data may be captured using a camera in the attachment device.

At 802 , a light source of the attachment device may be operated while the attachment device is used to capture thermal image data and non-thermal image data. The light source may be activated (eg, flashed or held) during an image capture operation based on, for example, a user input or an automatically determined light level. Illuminating the scene using a light source may enhance the non-thermal image captured by the non-thermal image sensor in the attachment device.

In step 804, the captured thermal image data and the captured non-thermal image data from the attachment device are configured to form an enhanced output image comprising all or a portion of the thermal image data and the actively illuminated non-thermal image data. can be combined. In embodiments, thermal and non-thermal images may be processed to generate a combined image using high contrast processing.

In the context of high contrast processing, high spatial frequency content may be obtained from one or more thermal and non-thermal images (eg, by performing high pass filtering, different imaging and/or other techniques). The combined image may include a radiometric component of a blend component and a thermal image including infrared (eg, thermal) characteristics of the blended scene with high spatial frequency content according to the blending parameter.

In an embodiment, the high spatial frequency content from the non-thermal image may be blended with the thermal image by superimposing the high spatial frequency content with the thermal image, wherein the high spatial frequency content corresponds to a column in which the high spatial frequency content is present. Replace the image part. For example, high spatial frequency content may include edges of depicted objects in an image of a scene, but may also include high spatial frequency content that may only be encoded into one or more components of the combined image behind the scenes.

For example, the radiometric component of the thermal image may be a chrominance component of the thermal image, and the high spatial frequency content may be derived from the luminance and/or chrominance of the non-thermal image. In such an embodiment, the combined image has a radiometric component (eg, a chrominance component of a thermal image) encoded into the chrominance component of the combined image and high spatial encoded directly into the luminance component of the combined image. frequency content (eg, as blended image data with no thermal image contribution).

By doing so, a radiometric calibration of the radiometric component of the thermal image can be obtained. In a similar embodiment, the blended image data may include high spatial frequency content added into the luminance component of the thermal image and the resulting blended data encoded into the luminance component of the resulting combined image. Non-thermal images are generated from non-thermal imager types, such as visible light imagers, low visible light high imagers, CCD imaging devices, EMCCD imaging devices, CMOS imaging devices, CMOS imaging devices, NIR. imaging devices, SWIR imaging devices, or other types of non-thermal imagers (eg, including passive or active illumination as would be appreciated by those skilled in the art)

For example, the techniques disclosed in the following application may be used in various embodiments:

US Application 12/477,828, filed June 3, 2009; US Application 12/766,739, filed April 23, 201; US Application 13/105,765, filed May 11, 2011; US Application 13/437,645, filed April 2, 2012; U.S. Application 61/473,207, filed April 8, 2011; US application 61/746,069, filed December 26, 2012; US application 61/746,074, filed December 26, 2012; US application 61/748,018, filed December 31, 2012; US application 61/792,582, filed March 15, 2013; US application 61/793,952, filed March 15, 2013; and International Application PCT/EP2011/056432, filed on April 21, 2011.

In an embodiment, one of the attachment apparatus 1250 or device 1200 receives a user input indicating a portion to be imaged by a first imaging module (eg, infrared imaging module 7000 ), and a second imaging module (eg, infrared imaging module 7000 ) , adjusts the light source 103 to illuminate at least a portion of interest in the spectrum sensed by the visible spectrum imaging module 7002 ( 101 ), receiving an illuminated captured image of the portion of interest from a second imaging module; , and to generate a combined image comprising lighting characteristics of the scene derived from the captured captured image. In embodiments, thermal images may be used to detect “hot” spots in an image, such as an image of a circuit breaker box.

9 is a flowchart of various operations to improve imaging of longevity based on a user input according to an embodiment of the present invention. For example, one or more processing portions 5800 may be performed by attachment device 1250 , device 1200 , and/or imaging modules 7000 , 7002 , respectively, using components as described. It should be understood that any step, sub-step, sub-process, or block of process 5800 may be performed in an order or arrangement other than the implementation described in FIG. 9 .

In embodiments, any portion of process 5800 may be performed in a loop, such as a video scene, to continuously operate with a series of infrared and/or visible spectral images. In other embodiments, processing 5800 includes intermediate processing (eg, receiving infrared and/or visible spectral images, registering the images with each other, generating an illuminated or combined image, or other of processing 5800 ). may be implemented in a partial feedback loop, including during or after performing processing) for display, and/or including receiving user input, such as user input directed to an intermediate processing step. Further, in embodiments, process 5800 may include interception of one or more steps, sub-steps, sub-processes, or any other process described herein.

At block 5810 , the attachment device 1250 generates a visible spectrum image of the scene. For example, the imaging module 7002 may be configured to generate one or more visible spectral images of a scene. In embodiments, block 5810 may include one or more operations described in connection with the processing of FIGS. 5-8 . If desired, a device camera 101 , such as camera 101 , may also capture the visible spectrum image.

At block 5812 , optionally concurrently with block 5810 , the attachment device 1250 generates an infrared image of the scene. For example, the imaging module 7000 may be configured to generate infrared images of one or more of the scenes. In some embodiments, block 5812 may include one or more operations described in connection with the processing of FIGS. 5-8 .

At block 5820 , the attachment device 1250 generates a data output signal corresponding to the generated image. For example, the imaging module 7000 , 7002 and/or the processor may be adapted to generate a data output signal corresponding to the image generated at block 5810 , 5812 . In some embodiments, the output signal may adhere to a specific interface standard, such as MIPI®.

At block 5830 , the attachment device 1250 and/or device 1200 stores data according to a common data format. For example, data may be stored in a desired data file according to a common data format.

At block 5840 , attachment device 1250 and/or device 1200 register images with each other. For example, attachment apparatus 1250 and/or device 1200 performs one or more of interpolation, scaling, cropping, rotational transformation, morphing, and/or filtering operations on one or more images for matchspecial content within the image. It can be applied to register any one of the generated images as the other of the generated images.

At block 5850 , attachment device 1250 and/or device 1200 receives user input indicating a portion of interest in a scene. For example, attachment apparatus 1250 and/or device 1200 may be adapted to receive user input provided by one or more other components, a touch screen display, or other apparatus representing a portion of interest in an already imaged scene. User input may be used to designate a pixel or group of pixels corresponding to a portion of interest. In some embodiments, the user input may be combined with the selection of the registration operation performed within block 5840 to determine corresponding pixels in the various captured images.

At block 5852 , a light source, such as light source 103 of device 1200 , illuminates a portion of interest. For example, either the attachment device 1250 and/or the device 1200 may be adapted to tune the light source 103 to illuminate a portion or all of a particular scene of interest. In some embodiments, portions of the scene and/or specific spectra may be selected by adjusting MEMS lenses and/or other systems coupled or coupled to the illumination module.

At block 5854 , the attachment apparatus 1250 and/or device 1200 generates an illuminated image of the portion of interest. For example, any one of the illuminated spectral sensitive imaging modules 7000 , 7002 or 101 within block 5852 can be captured while the light source 103 illuminates a specified portion of interest within specified block 5850 . It can be applied to generate an image.

At block 5860 , the attachment apparatus 1250 and/or device 1200 generates a combined image of the scene from the visible light image, the infrared image, and/or the illuminated image. In one embodiment, the combined image may include a visible light image having embedded data corresponding to infrared image data for each pixel of the visible light data. When such a combined image is displayed, the user can select a pixel or group of pixels with the user interface and text corresponding to the infrared image can be displayed along with the visible light image, for example in a text box or legend. . In some embodiments, any of the imaging modules 7000 , 7002 , 101 may be applied to generate a combined image using one or more of the processes described herein, as described with reference to FIGS. 5-8 . included processing.

At block 5870 , device 1200 displays one or more of the generated images. For example, device 1200 may be adapted to use a display (eg, display 201 in FIG. 2 ) to display one or more images generated in process 5800 . In some embodiments, block 5870 includes one or more operations described in connection with the processing of FIGS. 4-6 . The above-described embodiments do not limit the present invention. It should be understood that various modifications and variations are possible in accordance with the spirit of the present invention. Accordingly, the scope of the present invention is limited only by the following claims.

Although various embodiments have been described with respect to an attachment device, one or more embodiments of the present invention are applicable to a device alone or in combination with an attachment device. For example, a thermal image sensor may be implemented directly in the device (ie, device 1200 ), and optionally additional non-thermal image sensors may be implemented within the device. Consequently, the principles described herein may be applied based on sensors implemented within a device.

While the present invention has been described in detail with reference to a limited number of embodiments, it is to be understood that the invention is not limited to such disclosed practice. The present invention may be modified using variations, modifications, substitutions or equivalent arrangements not described herein but are consistent with the spirit of the present invention. Thus, while various embodiments of the present invention have been described, it should be understood that the features of the present invention include only some of the described embodiments. Accordingly, the present invention is limited only by the appended claims and not by the detailed description.

Claims (22)

an attachment device 1250 having a thermal imaging module configured to capture a thermal image of a scene and a non-thermal imaging module 7002 configured to capture a first non-thermal image of a scene; and
a processor in the attachment device or a user device (1200) attached to the attachment device;
the processor receives: a thermal image from a thermal imaging module 7000, a first non-thermal image from a non-thermal imaging module 7002, and a second non-thermal image from a camera 101 of the user device;
determine a distance to the object in the scene based on the object captured in the first non-thermal image and the object captured in the second non-thermal image, based on the determined distance the thermal imaging module 7000 and the non-thermal image and determine a parallax correction for the imaging module (7002). delete The method of claim 1 , wherein the processor determines a parallax-induced shift of the object in the first and second non-thermal images;
and using the determined parallax induced shift and the known distance between the non-thermal imaging module of the attachment device and the camera of the attachment device to determine the distance by triangulating the distance to the object. 4. The system of claim 3, wherein the processor comprises a processor within the housing of the attachment device. 4. The system of claim 3, further comprising a user device attached to the attachment apparatus. 6. The system of claim 5, wherein the processor includes a processor of a user device and a device connector receptacle configured to receive a device connector of an attachment apparatus for attaching an attachment apparatus to the user device. 7. The system of claim 6, wherein the user device comprises a mobile phone. The system of claim 1 , wherein the processor is configured to combine the thermal image from the thermal imaging module with a first non-thermal image using the determined parallax correction. capture a first non-thermal image for a scene using the camera 101 in the portable electronic device;
capture a second non-thermal image of the scene using the non-thermal imaging module 7002 in the attachment device 1250 attached to the portable electronic device;
determine a distance to an object in the scene using the first and second non-thermal images;
capture a thermal image of the scene using the thermal imaging module 7000 in the attachment device;
perform parallax correction between the thermal image and the second non-thermal image based on the determination;
and subsequent to performing the parallax correction, combining the thermal image and the second non-thermal image to form an enhanced output image. 10. The method of claim 9, further comprising providing power from a battery in the attachment device to the portable electronic device. 10. The method of claim 9, further comprising: operating a light source of the portable electronic device to remove at least a portion of a scene; and
and synchronizing the capturing of the first and second non-thermal images based on the operation. 10. The method of claim 9, further comprising: monitoring movement of the portable electronic device and attachment apparatus; and capturing first and second non-thermal images when the movement is below a requested threshold value. The method of claim 9 , further comprising: capturing additional images of additional objects using a camera, a non-thermal imaging module, and a thermal imaging module;
detect an edge of the additional object in each of the additional images; and
and determining alignment and distortion correction between the camera and the non-thermal imaging module for use in performing parallax correction. perform a calibration operation for the thermal imaging module 7000 in the attachment device 1250 attached to the user device 1200;
performing the calibration operation includes closing a shutter of the attachment device to block the thermal imaging module, and capturing, using the thermal imaging module, one or more images;
receive an image-based temperature determination request from a user;
determine whether a time after performing the calibration operation is less than or equal to a maximum allowed time; and
and capturing a thermal image using a thermal imaging module (7000) if it is determined that the time after performing the calibration operation is less than or equal to the maximum allowed time. delete 15. A method according to claim 14, characterized in that if it is determined that performing the calibration operation is longer than or equal to a maximum time allowed, providing a command to the user to close the shutter (250) to perform a new calibration of the thermal imaging module. Attachment is:
housing;
An infrared sensor assembly within a housing comprising: an infrared sensor assembly configured to capture infrared image data of a scene;
A non-thermal camera module in a housing, comprising: a non-thermal camera module configured to capture non-thermal image data of a scene;
A user device removably attached to a housing of an attachment apparatus, comprising: a non-thermal camera module; and the user device 1200 including a light source;
a processing module in the attachment device or user device, the processing module: cooperating with an infrared sensor assembly to receive infrared image data from the infrared sensor assembly, and non-thermal image data from a non-thermal camera module; cooperate with a non-thermal camera module in the attachment device to receive the and
and combine the infrared image data and the non-thermal image data to form an enhanced output image. 18. The method of claim 17, wherein the processing module is configured to illuminate at least a portion of the scene in cooperation with the light source, wherein the infrared image data and the non-thermal image data are captured by the infrared sensor assembly and the non-thermal camera module in the attachment device, respectively. A system characterized by being. 19. The method of claim 18, wherein the processing module is configured to combine infrared image data and non-thermal image data while the light source is operated to form an enhanced output image, and wherein the infrared sensor assembly comprises a plurality of infrared sensors. A system characterized by 18. The method of claim 17, wherein the processing module comprises:
receive additional non-thermal image data from a non-thermal camera module in the user device; and combine the infrared image data and the non-thermal image data based on a calibration determined using the non-thermal image data and the additional non-thermal image data. The apparatus of claim 1 , wherein the attachment device comprises: a shutter configured to selectively block or unblock the thermal imaging module and the non-thermal imaging module (7002); and
and a slidable slider assembly (248) for moving the shutter to selectively block or unblock the thermal imaging and non-thermal imaging modules. 18. The method of claim 17, wherein the non-thermal camera module of the user device (1200) is configured to capture a second non-thermal image of the scene;
The processing module is:
determine a distance to the object in the scene based on the captured object in the first non-thermal image and the captured object in the second non-thermal image; and determine a parallax correction for the infrared sensor assembly of the attachment device and the non-thermal imaging module (7002) based on the determined distance.

Images