KR20220115866A - Image non-uniformity mitigation systems and methods - Google Patents

Abstract

Provided is a technology for facilitating the mitigation of image non-uniformity for an imaging system and method. As one embodiment of the present invention, the method includes cutting the inside of the image based on defects in the image for obtaining a cropped image. The method further includes cutting a supplemental flat field correction (SFFC) map based on the detects in the image for obtaining a cropped SFFC map. The method includes determining a scaling value of a scaling section at least based on a cost function. The method further includes scaling the SFFC map based on the scaling value for obtaining a scaled SFFC map. In addition, related apparatuses and systems are provided.

Description

IMAGE NON-UNIFORMITY MITIGATION SYSTEMS AND METHODS

One or more embodiments relate generally to imaging, and more particularly to, for example, image non-uniformity mitigation systems and methods.

The imaging system may include an array of detectors, each detector functioning as a pixel generating a portion of a two-dimensional image. There are various image detectors, such as visible light image detectors, infrared image detectors, or other types of image detectors that may be provided in an image detector array for capturing images. As an example, a plurality of sensors may be provided in the image detector array to detect electromagnetic (EM) radiation at a desired wavelength. In some cases, such as infrared imaging, reading of image data captured by a detector may be performed in a time-multiplexed manner by a readout integrated circuit (ROIC). The read image data may be passed to other circuitry for processing, storage, and/or display, for example. In some cases, the combination of a detector array and an ROIC may be referred to as a focal plane array (FPA). Advances in FPA and process technology for image processing have made imaging systems more capable and more sophisticated.

SUMMARY The present disclosure provides a non-uniformity mitigation system and method.

In one or more embodiments, a method includes cropping an image based on defects in the image to obtain a cropped image. The method further includes cropping the SFFC map based on the defects in the image to obtain a cropped Supplemental Flat Field Correction (SFFC) map. The method further includes determining a scaling value of a scaling term based at least on the cost function. The method further includes scaling the SFFC map based on the scaling value to obtain a scaled SFFC map.

In one or more embodiments, a system includes a memory configured to store an SFFC map. The system further includes processing circuitry configured to crop the image based on the defects in the image to obtain a cropped image and also crop the SFFC map based on the defects in the image to obtain a cropped SFFC map. The processing circuitry is further configured to determine a scaling value of the scaling term based at least on the cost function. The processing circuitry is further configured to scale the SFFC map based on the scaling value to obtain a scaled SFFC map.

The scope of the present disclosure is defined by the claims, which are incorporated in this section by reference. A more complete understanding of the embodiments of the present disclosure as well as the realization of additional advantages will be provided to those skilled in the art upon consideration of the following detailed description of one or more embodiments. Reference will be made to the accompanying drawings, which will be briefly described first.

1 depicts a block diagram of an exemplary imaging system in accordance with one or more embodiments of the present disclosure.
2 illustrates a block diagram of an example image sensor assembly in accordance with one or more embodiments of the present disclosure.
3 illustrates an example image sensor assembly in accordance with one or more embodiments of the present disclosure.
4 and 5 illustrate example systems for facilitating image non-uniformity mitigation in accordance with one or more embodiments of the present disclosure.
6 illustrates an example system for iteratively adjusting a scalar value to minimize a cost function in accordance with one or more embodiments of the present disclosure.
7 depicts a flow diagram of an example process for facilitating image non-uniformity mitigation in accordance with one or more embodiments of the present disclosure.
Fig. 8a shows a defective image.
8B depicts an image obtained by performing non-uniformity mitigation on the image of FIG. 8A in accordance with one or more embodiments of the present disclosure.
Embodiments of the present disclosure and their advantages are best understood by reference to the following detailed description. It should be noted that the sizes of the various components and the distances between these components are not drawn to scale in the drawings. It should be understood that like reference numbers are used to identify like elements shown in one or more of the drawings.

The detailed description set forth below is for describing various configurations of the target technology, and does not represent the only configuration in which the target technology can be implemented. The accompanying drawings are incorporated herein and constitute a part of the detailed description. The detailed description includes specific details for the purpose of providing a thorough understanding of the subject technology. However, it will be apparent and apparent to one of ordinary skill in the art that the subject technology is not limited to the specific details set forth herein and that it may be practiced using one or more embodiments. In one or more instances, structures and components are shown in block diagram form in order to avoid obscuring the concepts of the subject technology. One or more embodiments of the present disclosure are illustrated by and/or described in connection with one or more drawings and are set forth in the claims.

Various techniques are provided to facilitate image non-uniformity mitigation. In some embodiments, the imaging system includes a detector array and a readout circuit. A detector array includes detectors (eg, also referred to as detector pixels, detector elements, or simply pixels). Each detector pixel detects incident EM radiation and generates image data (eg, infrared image data) representative of the detected EM radiation of the scene. In some embodiments, the detector array is used to detect infrared (eg, thermal infrared). For a pixel in an infrared image (e.g. a thermal infrared image), each output value of the pixel is expressed as a temperature, a digital count value, a percentage of the total temperature range, or in general any value that can be mapped to a temperature. provided and/or corresponding thereto. For example, a digital numeric value of 13,000 output by a pixel may represent a temperature of 160°C.

In some embodiments, scene non-uniformities within an image, such as those associated with defects in the optical path to the detector array, are mitigated (eg, reduced or eliminated) by applying a scaled SFFC map (eg, also referred to as an SFFC image) to that image. )do. The SFFC map contains SFFC values. In some cases, the defect may include a defect or debris of an imaging system, such as a wafer-level packaged sensor. Such defects may be in the cover, window, or other component of the imaging system within the field of view of the FPA. As an example, in some cases, the lid may be close to the FPA, such proximity that any defect in the lid will have a good focus. As another example, defects on the inner surface of a window tend to be closer to the FPA and better focused, and are generally smaller but more intense, whereas defects on the outer surface of the window are generally larger but more diffuse. These defects are typically non-uniformities whose diameter spans several pixels of the FPA and can be detected in the SFFC map. These defects associated with FPA are typically present consistently in images captured by FPA as their position remains constant. In some cases, the SFFC map may be flattened. For some imaging and/or imaging systems, flattening the SFFC map can help isolate defects. A flattened SFFC map may be obtained from the SFFC map by downsampling/upsampling the SFFC map and subtracting the SFFC map to provide a flattened SFFC map. In other cases, the SFFC map is not planarized (eg, planarization does not help isolate defects).

A scaled SFFC map can be obtained by applying a scaling term (P) to the SFFC map. In some embodiments, determining the value of the scaling term P utilizes a cost function. In this case, the cost function is a function of P and can be expressed as FVAL(P) . The value may be a scalar value (eg, also referred to as a scaling value or a scaling factor). The cost function may provide a representation (eg, a mathematical representation) of a quality associated with the image. In one aspect, a minimization process (e.g., iterative minimization) to minimize the cost function (e.g., to access a higher quality image) to allow convergence of the values of the scaling term (P) to apply to the SFFC map. process) can be performed. In an aspect, P may refer to a scaling term (P) or a value of a scaling term (P). The scaled SFFC map may then be applied to the image to obtain a corrected image (eg, non-uniformity corrected image).

In some aspects, the scaling term (P) may be determined based on a portion of the image. Although determined based on a portion of the image in some aspects, the scaling term (P) may be applied to the entire image to correct for non-uniformity of the overall image. Such application of the locally determined scaling term to the entire image may reduce computational complexity and/or computational time. In an aspect, said portion of an image may include a defect captured in the image. In some cases, this defect may be considered the worst defect represented in the image. This value of the scaling term (P) is applicable to mitigate the worst defects as well as other defects in the image. Defects in an image (eg, including worst-case defects) may be determined/identified by a user of the imaging system (eg, via visual inspection of said image by the human eye) and/or by a machine. . In some cases, the user and/or machine may determine the worst defect based on the size of the defect and/or the intensity of the defect in the image.

In some embodiments, the image processing pipeline may be divided into a main processing path and a background processing path. Operations in the main processing pipeline may include processing performed on images or videos as they are recorded and stored in the data store. The main processing path may include capturing an image, applying a scaling term (P) to the SFFC map to obtain a scaled SFFC map, and applying the scaled SFFC map to the image. In some cases, the scaling term (P) may be attenuated in the main processing path. In some cases, when capturing the input image, the gain used to apply the SFFC at that point in time may be captured. The main processing paths include automatic gain control, other types of noise suppression, gain and offset adjustment, spatial filtering, temporal filtering, radiometry conversions (eg, for infrared images), and/or display, storage and/or display, storage and/or or other operations to generate appropriate image data for further processing. The background processing path performs a minimization search and related operations to find the value of the scaling term (P) for the image (e.g. cropping an image and SFFC map for use in cost function calculation and calculating a gradient). can be done The operation of the background processing path may be performed outside the main processing path to avoid interruption or delay of image or video recording. In some embodiments, the scaling term may be determined per frame (eg, in real time) rather than as a background process.

Although various embodiments for non-uniformity mitigation are described in the context of infrared imaging (eg, thermal infrared imaging), non-uniformity mitigation may also be applied to image data of other wavelength bands. Various embodiments of the methods and systems disclosed herein include a visible light imaging system, an infrared imaging system, an imaging system with visible and infrared imaging capabilities, a short-wave infrared (SWIR) imaging system, and a Light Detection and Ranging (LIDAR) imaging. system, a radar detection and ranging (RADAR) imaging system, a millimeter wavelength (MMW) imaging system, an ultrasound imaging system, an X-ray imaging system, a mobile digital camera, a video surveillance system, a video processing system, or one or more of the EM spectrum. It may be included in or implemented as a variety of devices and systems, such as other systems or devices that may be required to obtain image data from a portion.

Referring now to the drawings, FIG. 1 illustrates an exemplary imaging system 100 (eg, infrared camera, tablet computer, laptop, personal digital assistant (PDA), mobile device, desktop) in accordance with one or more embodiments of the present disclosure. a block diagram of a computer, or other electronic device). However, not all illustrated components are required, and one or more embodiments may include additional components not shown in the drawings. Changes in the arrangement and type of elements may be made without departing from the spirit or scope of the claims set forth herein. Additional components, different components and/or fewer components may be provided.

Imaging system 100 may be used to capture and process images in accordance with an embodiment of the present disclosure. Imaging system 100 may be any type of imaging system that detects one or more ranges (eg, wavelength bands) of EM radiation and provides representative data (eg, one or more still image frames or video image frames). can indicate Imaging system 100 may include a housing that at least partially encloses components of imaging system 100 , for example, to facilitate miniaturization and protection of imaging system 100 . For example, the solid box labeled '175' in FIG. 1 may represent the housing of the imaging system 100 . The housing may include more, fewer, and/or different components of the imaging system 100 than those shown in the solid box of FIG. 1 . In one embodiment, imaging system 100 may include a portable device and may be integrated, for example, in a vehicle or non-mobile facility where storage and/or display of images is required. The vehicle may be a land-based vehicle (eg, a car, a truck), a sea-based vehicle, an air vehicle (eg, an unmanned aerial vehicle (UAV)), a space vehicle, or in general an integrated imaging system 100 (eg, For example, it can be any type of vehicle that can be installed inside, mounted, etc.). As another example, imaging system 100 may be coupled to various types of fixed locations (eg, home security mounts, campsite mounts or outdoor mounts, or other locations) via one or more types of mounts. .

According to one implementation, imaging system 100 includes a processing component 105 , a memory component 110 , an image capture component 115 , an image interface 120 , a control component 125 , a display component ( 130 ), a sensing component 135 , and/or a network interface 140 . According to various embodiments, processing component 105 is a processor, microprocessor, central processing unit (CPU), graphics processing unit (GPU), single-core processor, multi-core processor, microcontroller, programmable logic unit (PLD). ) (e.g., field programmable gate arrays (FPGAs)), application specific integrated circuits (ASICs), digital signal processing (DSP) devices, or by hardware wiring, software instruction execution, or a combination of both, in an embodiment of the present disclosure. and one or more of other logic devices that may be configured to perform the various operations discussed herein. The processing component 105 is configured to interface and communicate with various other components of the imaging system 100 (eg, 110, 115, 120, 125, 130, 135, 140, etc.) to perform these operations. can be For example, processing component 105 may process captured image data received from imaging capture component 115 , store the image data in memory component 110 , and/or memory component ( 110) may be configured to retrieve stored image data. In one aspect, the processing component 105 is configured to perform various system control operations (eg, to control communication and operation of various components of the imaging system 100 ) and other image processing operations (eg, data conversion). , video analysis, etc.).

In some cases, processing component 105 may perform nonuniformity correction (NUC) (eg, flat field correction (FFC) or other correction technique), spatial and/or temporal, for pixel values. It may perform operations such as filtering, and/or radiometric transformation. For example, an FFC calibration process (eg, also referred to as an FFC event) typically involves variations in the pixel-to-pixel output of the image of the detector circuit 165 (eg, individual microbolometers of the image detector circuit 165 ). It may refer to a correction technique performed in digital imaging to remove artifacts from the frame caused by distortion of the optical path) and/or by distortion of the optical path. In one case, imaging system 100 may include internal structures (eg, shutters, lids, covers, paddles) for selectively blocking image detector circuitry 165 . The structure may be used to provide/present a uniform scene to the detectors of the image detector circuit 165 . The detectors are effectively isolated from the scene. In an aspect, the FFC event may include capturing and averaging a number of frames while the shutter of the imaging system 100 is in a closed position to cover the image detector circuitry 165 , and thus the image detector circuitry 165 . captures image data of the structure and is blocked for the scene. In other cases, the FFC event may include the use of an external FFC source, where corrections are applied to the captured image data so that the radiometric data of the images is accurate when viewing the external FFC source. In general, the internal FFC source and/or the external FFC source are at a precisely known temperature. The captured frames of the internal source (eg, internal structures) and/or external sources are read by the read circuitry 170 (eg, by processing circuitry that receives the frames output by the read circuitry 170 ). It is accumulated and used to update the FFC correction term to be applied to the output frames and is generally not provided as an output of the imaging system 100 . In some cases, a calibration event may be performed periodically and/or according to a user command.

In some aspects, the FFC values may be used to correct for non-uniformities associated with various optical paths of imaging system 100 . An SFFC value can also be determined and applied to the image data to further correct for this non-uniformity. Examples of systems and methods for determining FFC values and/or SFFC values and related operations are provided in US Patent Application Publication Nos. 2020/0005440 and 2020/0154063, which are incorporated herein by reference in their entirety.

In one embodiment, memory component 110 includes one or more memory devices configured to store data and information, including infrared image data and information. Memory component 110 includes random access memory (RAM), dynamic RAM (DRAM), static RAM (SRAM), non-volatile random access memory (NVRAM), read-only memory (ROM), programmable read-only memory (PROM) , one or more volatile and non-volatile memory devices, such as erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), flash memory, hard disk drives, and/or other types of memory. It may include various types of memory devices. As discussed above, processing component 105 may be configured to execute software instructions stored in memory component 110 to perform method and process steps and/or operations. In one or more embodiments, these instructions, when executed by the processing component 105 , may cause the imaging system 100 to perform operations to generate a scaling term and mitigate non-uniformities. Processing component 105 and/or image interface 120 may be configured to store images or digital image data captured by image capture component 115 in memory component 110 . Processing component 105 may be configured to store FFC values, SFFC values, processed (eg, non-uniformity corrected) still and/or video images, and/or other data to memory component 110 . .

In some embodiments, a separate machine-readable medium 145 (eg, a memory such as a hard drive, compact disk, digital video disk, or flash memory) may be provided in a variety of ways, such as methods and operations related to image data processing, and It may store software instructions and/or configuration data that can be executed or accessed by a computer (eg, a logical device or processor-based system) to perform an operation. In an aspect, the machine-readable medium 145 may be portable and/or located separately from the imaging system 100 , where stored software instructions and/or data image the machine-readable medium 145 . The imaging system 100 is provided to the imaging system 100 by coupling to the system and/or by downloading the imaging system 100 from the machine-readable medium 145 (eg, via a wired link and/or a wireless link). Various modules may be incorporated into software and/or hardware as part of the processing component 105 , where code (eg, software or configuration data) for the modules may be, for example, in the memory component 110 . is saved

Imaging system 100 may represent an imaging device, such as a video and/or still camera, for capturing and processing images and/or video of scene 160 . In this regard, the image capture component 115 of the imaging system 100 may be configured to capture images (eg, still and/or video images) of the scene 160 with a particular spectrum or modality. have. Image capture component 115 includes image detector circuitry 165 (eg, thermal infrared detector circuit) and readout circuitry 170 (eg, ROIC). For example, image capture component 115 detects IR radiation in the near infrared, mid-infrared, and/or far-infrared spectrum from scene 160 and generates an IR image (eg, IR image data or signal) representative of the radiation. and an IR imaging sensor (eg, an IR imaging sensor array) configured to provide. For example, the image detector circuit 165 may capture (eg, detect, sense) IR radiation having a wavelength in the range of about 700 nm to about 2 mm or a portion thereof. For example, in some aspects, the image detector circuit 165 is configured with short wave IR (SWIR) radiation, mid-wave IR (MWIR) radiation (eg, EM radiation having a wavelength between 2 μm and 5 μm). ), and/or long wave IR (LWIR) radiation (eg, EM radiation having a wavelength between 7 μm and 14 μm), or at any desired IR wavelength (eg, generally in the range of 0.7 μm to 14 μm). may be sensitive (eg, better detectable). In another aspect, the image detector circuit 165 may capture radiation from one or more other wavelength bands of the EM spectrum, such as visible light, ultraviolet light, and the like.

Image detector circuitry 165 may capture image data (eg, infrared image data) associated with scene 160 . To capture an image, image detector circuitry 165 may detect image data (eg, in the form of EM radiation) of scene 160 and generate pixel values of the image based on scene 160 . . An image may be referred to as a frame or an image frame. In some cases, the image detector circuit 165 detects radiation of a specific wavelength band, converts the detected radiation into an electrical signal (eg, voltage, current, etc.), and calculates a pixel value based on the electrical signal. It may include an array of detectors capable of generating (eg, also referred to as an array of pixels). Each detector in the array may capture a respective portion of the image data and generate a pixel value based on each portion captured by the detector. The pixel values generated by the detector may be referred to as the output of the detector. As a non-limiting example, each detector can convert an avalanche photodiode, infrared photodetector, quantum well infrared photodetector, microbolometer, or EM radiation (e.g., of a specific wavelength) into pixel values. It may be a photodetector, such as other detectors with The array of detectors may be arranged in rows and columns. In one embodiment, the image detector circuit 165 receives an energy flux (eg, a thermal infrared energy flux) from the object(s) in the scene 160 and transmits the energy flux to the object(s) in the scene 160 . can be converted to a data value representing the temperature of Imaging system 100 may be radiometrically calibrated to ensure an accurate conversion from the amount of energy received by image detector circuitry 165 to data values generated by image detector circuitry 165 . In some cases, the uncertainty factor may be based on parameters related to the flux-temperature conversion.

The image can be or can be considered a data structure or a data structure that includes pixels and is a representation of image data associated with a scene 160 , where each pixel is emitted or reflected from a portion of the scene and produces the pixel value. has a pixel value representing the EM radiation received by Based on context, a pixel may refer to a detector of the image detector circuitry 165 that generates a pixel (eg, pixel location, pixel coordinate) or associated pixel value of an image formed from the generated pixel values. In an embodiment, the image may be a thermal infrared image based on thermal infrared image data (eg, also referred to as a thermal image). Each pixel value of the thermal infrared image represents the temperature of that part of the scene 160 .

In one aspect, the pixel values generated by the image detector circuitry 165 may be represented as digital numeric values generated based on electrical signals obtained by converting the detected radiation. For example, if the image detector circuit 165 includes or is coupled to analog-to-digital (ADC) circuitry, the ADC circuitry may generate a digital numeric value based on the electrical signal. For an ADC circuit that can represent an electrical signal using 14 bits, the digital numeric value can range from 0 to 16,383. In this case, the pixel value of the detector may be a digital numeric value output from the ADC circuit. In other cases (eg, in the absence of ADC circuitry), the pixel value may be essentially the value of an electrical signal or analog having a value representing it. For example, for infrared imaging, a higher amount of IR radiation incident upon and detected by image detector circuit 165 (eg, IR image detector circuit) is associated with a higher digital numeric value and higher temperature.

For infrared imaging, characteristics associated with image detector circuitry 165 (and associated ADC circuitry, if any) include, by way of non-limiting example, dynamic range, the lowest temperature that can be stably represented, and the highest temperature that can be stably represented. , and sensitivity. The dynamic range may be or represent a range (eg, a difference) between the lowest and highest temperatures that may be captured by the image detector circuitry 165 and represented in an infrared image. In this regard, areas of the scene 160 that are below the lowest temperature may be buried in the noise floor of the imaging system 100 and may appear faded and/or noisy in the IR image. Regions of the scene 160 above the highest temperature cause saturation in the infrared image, where the saturated regions are in the same way (e.g., using the same color values or the same grayscale values) as regions of the highest temperature. is expressed For example, when the image detector circuit 165 uses an ADC circuit to generate a digital numeric value, any temperature above the highest temperature is the highest value that the ADC circuit can represent (eg, for a 14-bit ADC circuit). 16,383), and any temperature below the lowest temperature can be mapped to the lowest value (eg, 0) that the ADC circuit can represent. That is, the infrared image does not distinguish the region above the maximum temperature from the region with the highest temperature, and does not distinguish the region below the lowest temperature from the region with the lowest temperature.

The readout circuitry 170 serves as an interface between the image detector circuitry 165 for detecting image data and a processing component 105 for processing the detected image data as read out by the readout circuitry 170 . data communication from the readout circuitry 170 to the processing component 105 is facilitated by the image interface 120 . The image capture frame rate may refer to the rate (eg, images per second) at which images are sequentially detected by the image detector circuitry 165 and provided to the processing component 105 by the readout circuitry 170 . . The readout circuitry 170 may read out the pixel values generated by the image detector circuitry 165 according to an integration time (eg, also referred to as an integration period).

In various embodiments, the combination of image detector circuitry 165 and readout circuitry 170 may be, include, or provide an FPA together. In some aspects, image detector circuit 165 may be a thermal image detector circuit comprising an array of microbolometers, and the combination of image detector circuit 165 and read circuit 170 may be referred to as a microbolometer FPA. . In some cases, the microbolometer array may be arranged in rows and columns. The microbolometer may detect IR radiation and generate pixel values based on the detected IR radiation. For example, in some cases, the microbolometer may be a thermal IR detector that detects IR radiation in the form of thermal energy and generates a pixel value based on an amount of detected thermal energy. The microbolometer can absorb incident IR radiation and produce a corresponding temperature change in the microbolometer. This temperature change is related to the resistance change of the microbolometer. When each microbolometer acts as a pixel, a two-dimensional image or pictorial representation of the incident IR radiation can be created by translating the resistance change of each microbolometer into a time-multiplexed electrical signal. The translation may be performed by ROIC. The microbolometer FPA may include an IR detection material, such as amorphous silicon (a-Si), vanadium oxide (VOx), combinations thereof, and/or other detection material(s). In one aspect, for a microbolometer FPA, the integration time may be or represent a time interval at which the microbolometer is biased. A longer integration time in this case may be associated with a higher gain of the IR signal, but no more IR radiation is collected. IR radiation can be collected in the form of thermal energy by a microbolometer.

In some cases, image capture component 115 may include one or more filters configured to pass some wavelengths of radiation but substantially block other wavelengths of radiation. For example, image capture component 115 may include one or more filters (eg, MWIR filters, thermal IR filters, and narrow filters configured to pass IR radiation of some wavelengths while substantially blocking IR radiation of other wavelengths). band filter). In this example, such a filter may be used to tailor the image capture component 115 for increased sensitivity to a desired band of IR wavelengths. In an aspect, an IR imaging device may be referred to as a thermal imaging device when it is customized to capture a thermal IR image. Other imaging devices, including IR imaging devices that are tailored to capture infrared IR images outside of the thermal range, may be referred to as non-thermal imaging devices.

In one specific, non-limiting example, image capture component 115 is an IR having an FPA of detectors responsive to IR radiation including near infrared (NIR), SWIR, MWIR, LWIR and/or very long wave IR (VLWIR). It may include an imaging sensor. In some other embodiments, alternatively or additionally, the image capture component 115 may be a Complementary Metal Oxide Semiconductor (CMOS) sensor or Charge -Coupled Device) may include a sensor.

Other imaging sensors that may be implemented in image capture component 115 include photonic mixer device (PMD) imaging sensors or other time of flight (ToF) imaging sensors, LIDAR imaging devices, RADAR imaging devices, millimeter imaging devices. , including positron emission tomography (PET) scanners, single photon emission computed tomography (SPECT) scanners, ultrasound imaging devices, or other imaging devices operating in a specific modality and/or spectrum. do. In the case of some of these imaging sensors configured to capture images of a specific modality and/or spectrum (eg, the infrared spectrum, etc.), they may be, for example, a typical CMOS-based or CCD-based imaging sensor or other imaging sensor, imaging scanner , or when compared to various types of imaging devices, it tends to produce images with low frequency shading.

An image provided by the image capture component 115 , or digital image data corresponding to the image, may be associated with a respective image dimension (also referred to as a pixel dimension). An image dimension or pixel dimension generally refers to the number of pixels in an image, which can be expressed, for example, by multiplying the width and height in the case of a two-dimensional image, or as appropriate to the relevant dimensions or shape of the image. Thus, an image having a native resolution may be resized to a smaller size (eg, having a smaller pixel dimension), eg, to reduce the cost of processing and analyzing the image. A filter (eg, a non-uniformity estimate) may be generated based on the analysis of the scaled image. The filter can then be scaled to the image's native resolution and dimensions before being applied to the image.

Image interface 120, in some embodiments, is configured to receive images (eg, digital image data) generated by or stored thereon by an external device (eg, remote device 150 and/or other devices). It may include suitable input ports, connectors, switches, and/or circuitry configured to interface with the external device for this purpose. In one aspect, image interface 120 may include a serial interface and telemetry line for providing metadata related to image data. The received image or image data may be provided to a processing component 105 . In this regard, the received image or image data may be converted into a signal or data suitable for processing by the processing component 105 . For example, in one embodiment, image interface 120 may be configured to receive analog video data and convert it to appropriate digital data to be provided to processing component 105 .

Image interface 120 may include various standard video ports, which may be connected to a video player, video camera, or other device capable of generating standard video signals, and may process the received video signals into processing component 105 . ) can be converted into digital video/image data suitable for processing by In some embodiments, image interface 120 may also be configured to interface with and receive images (eg, image data) from image capture component 115 . In other embodiments, the image capture component 115 may interface directly with the processing component 105 .

The control component 125, in one embodiment, is a user input and/or interface device, such as a rotatable knob (eg, a potentiometer), a push button, a slide bar, a keyboard, and/or a user input control signal. other devices configured to create The processing component 105 may be configured to sense a control input signal from a user via the control component 125 and to respond to any sensed control input signal received therefrom. The processing component 105 may be configured to interpret such a control input signal as a value, as is commonly understood by one of ordinary skill in the art. In one embodiment, the control component 125 may include a control unit (eg, a wired or wireless handheld control unit) having a push button configured to interface with a user and receive user input control values. In one implementation, the push buttons of the control unit are configured to enable the imaging system 100, such as autofocus, menu activation and selection, field of view, brightness, contrast, noise filtering, image enhancement, and/or various other functions of the imaging system or camera. It can also be used to control various functions of

Display component 130 includes, in one embodiment, an image display device (eg, a liquid crystal display (LCD)) or various other types of generally known video displays or monitors. The processing component 105 may be configured to display image data and information on the display component 130 . Processing component 105 may be configured to retrieve image data and information from memory component 110 and display any retrieved image data and information to display component 130 . Display component 130 may include circuitry in a display that may be utilized by processing component 105 to display image data and information. Display component 130 may be configured to receive image data and information directly from image capture component 115 , processing component 105 , and/or image interface 120 , or image data and information It may also be transmitted from the memory component 110 via the processing component 105 .

The sensing component 135, in one embodiment, includes one or more sensors of various types depending on the application or implementation requirements, as would be understood by one of ordinary skill in the art. The sensors of the sensing component 135 provide data and/or information to at least the processing component 105 . In an aspect, the processing component 105 may be configured to communicate with the sensing component 135 . In various implementations, sensing component 135 may provide information regarding environmental conditions, such as external temperature, lighting conditions (eg, day, night, dusk, and/or dawn), humidity levels, and specific weather conditions (eg, sunny). , rain and/or snow), distance (eg, laser range finder or ToF camera), and/or whether you have entered or exited a tunnel or other type of enclosure. The sensing component 135 is typically for monitoring various conditions (eg, environmental conditions) that may affect the image data (eg, image appearance) provided by the image capture component 115 . It may refer to a conventional sensor as generally known to those skilled in the art.

In some implementations, sensing component 135 (eg, one or more sensors) may include devices that relay information to processing component 105 via wired and/or wireless communication. For example, the sensing component 135 may be configured to perform a local broadcast (eg, via radio frequency (RF) transmission), via a mobile or cellular network, and/or via an information beacon of an infrastructure (eg, , traffic or highway information beacon infrastructure), or various other wired and/or wireless technologies, etc. In some embodiments, the processing component 105 may include the sensing component 135 . ) may be used to modify the configuration of the image capture component 115 (eg, to adjust the light sensitivity level, or the orientation of the image capture component 115 and Adjusting the angle, adjusting the aperture, etc.).

In some embodiments, the various components of imaging system 100 may be distributed and communicate with each other via network 155 . In this regard, imaging system 100 may include network interface 140 configured to facilitate wired and/or wireless communication between various components of imaging system 100 over network 155 . In such an embodiment, components may also be replicated for a particular application of imaging system 100 if desired. That is, components configured for the same or similar operation may be distributed over a network. In addition, all or some of any one of the various components may, if desired, be configured with a remote device 150 ( For example, it may be implemented using suitable components of a conventional digital video recorder (DVR), a computer configured for image processing, and/or other devices). Thus, for example, all or a portion of the processing component 105 , all or a portion of the memory component 110 , and/or some or all of the display component 130 are implemented on the remote device 150 or can be duplicated. In some embodiments, imaging system 100 may not include an imaging sensor (eg, image capture component 115 ), but instead of processing component 105 and/or imaging system 100 . Receive image or image data from an imaging sensor located separately or remotely from other components. It should be understood that many other combinations of distributed implementations of imaging system 100 are possible without departing from the scope and spirit of the present disclosure.

Further, in various embodiments, the various components of imaging system 100 may or may not be combined and/or implemented, as desired or depending on an application or requirement. In one example, processing component 105 includes memory component 110 , image capture component 115 , image interface 120 , display component 130 , sensing component 135 , and/or a network It may be coupled with the interface 140 . In another example, processing component 105 is configured such that certain functions of processing component 105 are implemented by circuitry (eg, processor, microprocessor, logic device, microcontroller, etc.) within image capture component 115 . may be coupled with the image capture component 115 to be performed.

2 illustrates a block diagram of an example image sensor assembly 200 in accordance with one or more embodiments of the present disclosure. However, not all illustrated components are required, and one or more embodiments may include additional components not shown in the drawings. Changes in the arrangement and type of elements may be made without departing from the spirit or scope of the claims set forth herein. Additional components, different components, and/or fewer components may be provided. In one embodiment, the image sensor assembly 200 may be, for example, an FPA implemented as the image capture component 115 of FIG. 1 .

The image sensor assembly 200 includes a unit cell array 205 , column multiplexers 210 and 215 , column amplifiers 220 and 225 , a row multiplexer 230 , a control bias and timing circuit 235 , and digital-to-analog. a converter (DAC) 240 , and a data output buffer 245 . In some aspects, operations of and/or pertaining to unit cell array 205 and other components may be performed according to a system clock and/or synchronization signal (eg, a line synchronization (LSYNC) signal). The unit cell array 205 includes an array of unit cells. In an aspect, each unit cell may include a detector (eg, a pixel) and interface circuitry. The interface circuit of each unit cell may provide an output signal such as an output voltage or an output current in response to a detection signal (eg, detection current, detection voltage) provided by the detector of the unit cell. The output signal may represent the magnitude of the EM radiation received by the detector and may be referred to as image pixel data or simply image data. The column multiplexer 215 , the column amplifier 220 , the row multiplexer 230 , and the data output buffer 245 receive the output signal from the unit cell array 205 as a data output signal on the data output line 250 . can be used to provide The output signal of the data on the data output line 250 may include components downstream of the image sensor assembly 200 , such as processing circuitry (eg, processing component 105 in FIG. 1 ), memory (eg, For example, memory component 110 of FIG. 1 ), a display device (eg, memory component 110 of FIG. 1 ), and/or other components may be provided to process, store, and/or process the output signal; Alternatively, display may be performed. The data output signal may be an image composed of pixel values for the image sensor assembly 200 . In this regard, the column multiplexer 215 , the column amplifier 220 , the row multiplexer 230 , and the data output buffer 245 may collectively provide the ROIC (or a portion thereof) of the image sensor assembly 200 . can In an aspect, the interface circuitry may be considered part of the ROIC, or an interface between the detectors and the ROIC. In some embodiments, the components of the image sensor assembly 200 may be implemented such that the unit cell array 205 and the ROIC may be part of a single die. In one embodiment, the components of the image sensor assembly 200 are implemented such that the unit cell array 205 is hybridized (eg, bonded, jointed, mated) to the ROIC. can be An example of such blending is described with respect to FIG. 3 .

Column amplifier 225 may generally represent any column processing circuit suitable for a given application (analog and/or digital), and is not limited to amplifier circuitry for analog signals. In this regard, thermal amplifier 225 may be referred to more generally in this respect as a thermal processor. Signals received by column amplifier 225, such as analog signals on an analog bus and/or digital signals on a digital bus, may be processed according to the analog or digital characteristics of the signal. As an example, the column amplifier 225 may include circuitry for processing digital signals. As another example, the column amplifier 225 may be a path (eg, without processing) that the digital signal from the unit cell array 205 traverses to reach the column multiplexer 215 . As another example, column amplifier 225 may include an ADC that converts an analog signal to a digital signal (eg, to obtain a digital numeric value). This digital signal may be provided to the column multiplexer 215 .

Each unit cell is provided with a bias signal (eg, bias signal) for biasing the detector of the unit cell to compensate for different response characteristics of the unit cell due to, for example, changes in temperature, manufacturing variations, and/or other factors. voltage, bias current). For example, the control bias and timing circuit 235 may generate a bias signal and provide it to a unit cell. By providing an appropriate bias signal to each unit cell, the unit cell array 205 can be effectively calibrated to provide accurate image data in response to light (eg, IR light) incident on the unit cell's detector. In an aspect, the control bias and timing circuit 235 may be, include, or be part of a logic circuit.

The control bias and timing circuit 235 may generate a control signal for addressing the unit cell array 205 to allow access and reading of image data from the addressed portion of the unit cell array 205 . The unit cell array 205 may be addressed to access and read image data from the unit cell array 205 on a row-by-row basis, although in other implementations the unit cell array 205 may be addressed column-by-column or otherwise. can

The control bias and timing circuit 235 may generate a bias value and a timing control voltage. In some cases, DAC 240 converts a bias value received as or as part of a data input signal on data input signal line 255 to a bias signal (eg, an analog signal on analog signal line(s) 260 ). , and this bias signal is provided to each unit cell through the operation of the column multiplexer 210 , the column amplifier 220 , and the row multiplexer 230 . For example, DAC 240 may drive a digital control signal (eg, provided as a bit) to an appropriate analog signal level for the unit cells. In some techniques, a digital control signal of 0 or 1 may be driven to an appropriate logic low voltage level or an appropriate logic high voltage level, respectively. In another aspect, the control bias and timing circuit 235 may generate a bias signal (eg, an analog signal) and provide the bias signal to a unit cell without using the DAC 240 . In this regard, some implementations do not include DAC 240 , data input signal line 255 , and/or analog signal line(s) 260 . In one embodiment, the control bias and timing circuit 235 may be, include, be part of, or otherwise coupled to the image capture component 115 and/or the processing component 105 of FIG. 1 . .

In one embodiment, the image sensor assembly 200 may be implemented as part of an imaging system (eg, 100 ). In addition to the various components of the image sensor assembly 200, the imaging system may also be suitable for one or more processors, memories, logic, displays, interfaces, optics (eg, lenses, mirrors, beamsplitters), and/or various implementations. Other components may be included. In an aspect, the data output signal on data output line 250 may be provided to a processor (not shown) for further processing. For example, the data output signal may be an image formed by pixel values from unit cells of the image sensor assembly 200 . Processors may perform operations such as non-uniformity correction (eg, FFC or other correction techniques), spatial and/or temporal filtering, and/or other operations. Images (eg, processed images) are stored in memory (eg, external to or internal to the imaging system) and/or displayed on a display device (eg, external to and/or integrated into the imaging system) can be The various components of FIG. 2 may be implemented on a single chip or multiple chips. Also, although the various components are illustrated as a set of individual blocks, the various blocks may be merged together or the various blocks shown in FIG. 2 may be separated into separate blocks.

In FIG. 2 , the unit cell array 205 is shown as 8x8 (eg, 8 rows and 8 columns) of unit cells. However, the unit cell array 205 may have other array sizes. As a non-limiting example, unit cell array 205 may have 512x512 (eg, 512 rows and 512 columns of unit cells), 1024x1024, 2048x2048, 4096x4096, 8192x8192, and/or other array sizes. In some cases, the array size may have a row size (eg, number of detectors in a row) different from the column size. Examples of frame rates may include 30 Hz, 60 Hz, and 120 Hz. In one aspect, each unit cell of the unit cell array 205 may represent a pixel.

In some embodiments, the components of image sensor assembly 200 may be implemented such that a detector array (eg, an array of infrared detectors such as a microbolometer) and readout circuitry may be part of a single die. In another embodiment, the components of the image sensor assembly 200 may be implemented such that the detector array is compatible with (eg, bonded to) a readout circuit. For example, FIG. 3 illustrates an example image sensor assembly 300 in accordance with one or more embodiments of the present disclosure. However, not all illustrated components are required, and one or more embodiments may include additional components not shown in the drawings. Changes in the arrangement and type of elements may be made without departing from the spirit or scope of the claims set forth herein. Additional components, different components, and/or fewer components may be provided. In one embodiment, the image sensor assembly 300 may be, include, or be part of the image sensor assembly 200 .

The image sensor assembly 300 includes a device wafer 305 , a read circuit 310 , and contacts 315 for bonding (eg, mechanically and electrically bonding) the device wafer 305 to the read circuit 310 . ) is included. The device wafer 305 may include a detector (eg, a unit cell array 205 ). For example, the detector may be an InGaAs type detector. Contact 315 may bond detector and readout circuit 310 of device wafer 305 . Contact 315 may include a conductive contact of a detector of the device wafer 305 , a conductive contact of a read circuit 310 , and/or a metal junction between the conductive contact of the detector and a conductive contact of the read circuit 310 . have. In one embodiment, the device wafer 305 may be bump-bonded to the read circuit 310 using a junction-bump (eg, indium bump). A junction-bump may be formed on the device wafer 305 and/or the read circuit 310 to allow a connection between the device wafer 305 and the read circuit 310 . In one aspect, blending the device wafer 305 to the read circuitry 310 bonds the device wafer 305 (eg, a detector of the device wafer 305) to the readout circuitry 310 to bond the device wafer 305 to the readout circuitry 310 . ) and the read circuit 310 mechanically and electrically.

4 illustrates an example system 400 for facilitating image non-uniformity mitigation in accordance with one or more embodiments of the present disclosure. However, not all illustrated components are required, and one or more embodiments may include additional components not shown in the drawings. Changes in the arrangement and type of elements may be made without departing from the spirit or scope of the claims set forth herein. Additional components, other components, and/or fewer components may be provided.

In system 400 , a primary processing path includes performing non-uniformity correction on image data 405 to obtain corrected image data 410 (eg, non-uniformity corrected image data). The image data 405 and the corrected image data 410 may each include an image. In some cases, image data 405 may include an image to which a gain and/or offset has been applied, and thus image data 405 may be referred to as post-gain/offset image data.

The image 415 of the image data 405 may be provided to a background processing path. The image 415 may be referred to as a snapshot of the image data 405 . In some cases, a tapping circuit may trigger a snapshot and/or a tap to obtain the snapshot (eg, periodically, aperiodically, and/or in response to user input). ). A cropping circuit 420 receives the image 415 and crops the image 415 based on the defect data 425 to obtain a cropped image 430 . Defect data 425 may include data indicative of location, size, and/or other data related to defects in image 415 .

In an aspect, a defect may be determined/identified by a user (eg, via visual inspection by a human eye) and/or a machine of the imaging system 100 . In one embodiment, the defect data 425 may belong to the defect identified as the worst defect in the image 415 . For example, the user and/or machine may determine the worst defect based on the size and/or intensity of the defect in the image. In some aspects, cropped image 430 is larger than and includes defects (eg, worst-case defects) in image 415 . In some cases, such as when the cropping circuit 420 generates a crop image of a fixed size and/or a maximum size that is smaller than the size of the defect, the crop image may include a portion of the defect.

A portion of the SFFC map 435 corresponding to the cropped image 430 (eg, in the size, location, and shape of the cropped image 430 relative to the image 415 ) is cropped to provide the SFFC map 440 . . In some cases, for example, an image tap may be used for SFFC map 435 and/or cropped SFFC map 440 for testing/troubleshooting purposes. As a non-limiting example, the cropped image 430 and cropped SFFC map 440 may be 32 pixels by 32 pixels, 64 pixels by 64 pixels, 32 pixels by 64 pixels, or other sizes. The cropped image 430 and cropped SFFC map 440 may be of different sizes and/or shapes (eg, non-rectangular maps/images).

The minimization circuit 445 receives the cropped image 430 and the cropped SFFC map 440 and iteratively adjusts the scalar value P to minimize the cost function FVAL(P). In one embodiment, the cost function may be provided by:

는 크롭 이미지(430)이고 NU(예를 들어, 불균일 추정)는 크롭 SFFC 맵(440)이다.

는 크롭 이미지로 지칭될 수 있고(예를 들어, 불균일 보정된 이미지, 깨끗해진 이미지, 보정된/깨끗해진 크롭 이미지 또는 보정된/깨끗해진 이미지 부분이라고도 함), 이는 크롭 이미지(430)로부터 크롭 SFFC ao맵(440)(스케일링 항(P)에 의해 스케일링 후)만큼 감산하여 형성된다. 일부 경우들에서,

와 P x NU는 0부터 1까지의 범위(0과 1을 포함)로 정규화될 수 있다. here

is a cropped image 430 and NU (eg, non-uniformity estimation) is a cropped SFFC map 440 .

may be referred to as a cropped image (eg, non-uniformity corrected image, cleared image, corrected/cleaned cropped image, or corrected/cleaned image portion), which is a cropped SFFC from cropped image 430 . It is formed by subtracting the ao map 440 (after scaling by the scaling term P). In some cases,

and P x NU can be normalized to a range from 0 to 1 (including 0 and 1).

In an aspect, the minimization circuit 445 can determine a value for the scaling term P that minimizes the cost function based on a Nelder-Mead scalar minimization technique. In an aspect, other non-linear direct-search minimization approaches may be used, such as coordinate descent approaches, particle swarm optimization approaches, and the like. Other embodiments may use other minimization/optimization processes to determine the value of the scaling term P that minimizes the cost function. The minimization/optimization process utilized to determine the scalar value P may be selected based on computational complexity, computation time, efficiency, and/or other considerations.

이고, 여기서

, 및

는 크롭 이미지(430) 및 크롭 SFFC 맵(440)과 크기가 동일한 어레이이다. 어레이(x)의 구배는

에 의해 제공될 수 있다. 일 양태에서, 어레이(x)의 각 라인에 대한, 함수

및

는 어레이(x)의 라인(예를 들어, 행 또는 열)을 취하여 그것을 어레이(x)의 다른 라인에서 빼거나 그 역으로 한다. 일 예에서,

는 수직 방향으로 일어날 수 있으며 여기서 어레이(x)의 모든 행은 두 행 위로부터 감산되고(예를 들어, 3행의 픽셀 빼기 1행의 픽셀, 4행의 픽셀 빼기 2행의 픽셀, 5행의 픽셀 빼기 3행의 픽셀 등), 및/또는

는 수평 방향에서 일어날 수 있으며 여기서 어레이(x)의 모든 열은 두 열 위로부터 감산된다(예를 들어, 3열 픽셀 빼기 1열 픽셀, 4열 픽셀 빼기 2열 픽셀, 5열 픽셀 빼기 3열 픽셀 등). 상기 감산(빼기)는 반드시 2만큼 이격되어야 하는 것은 아니다(예를 들면, 라인 3 빼기 라인 1). 예를 들면, 서로 감산되는 라인들은 3 이상 이격될 수도 있다. 어레이(x)와 관련된 구배를 결정하는 다른 방식이 사용될 수도 있다. In the cost function FVAL(P) presented above,

and where

, and

is an array having the same size as the cropped image 430 and the cropped SFFC map 440 . The gradient of array (x) is

can be provided by In one aspect, for each line of array (x), a function

and

takes a line (eg, a row or column) of array (x) and subtracts it from another line of array (x) and vice versa. In one example,

can occur in the vertical direction where every row of array (x) is subtracted from above two rows (e.g., row 3 pixels minus row 1 pixels, row 4 pixels minus row 2 pixels, row 5 pixels) pixels minus 3 rows of pixels, etc.), and/or

can occur in the horizontal direction, where every column of array (x) is subtracted from above two columns (eg, column 3 pixels minus column 1 pixels, column 4 pixels minus column 2 pixels, column 5 pixels minus column 3 pixels) etc). The subtraction (subtraction) is not necessarily spaced apart by two (eg, line 3 minus line 1). For example, lines subtracted from each other may be spaced apart by three or more. Other ways of determining the gradient associated with array (x) may be used.

The value of the scaling term (P) can be iteratively adjusted and the corresponding cost function FVAL(P) computed to obtain (e.g., converge toward) a value of P that minimizes the cost function FVAL(P). can Under this iterative approach, the crop SFFC map 440 is fixed while the value of the scaling term P is iteratively updated. Each update involves the adjustment of P. In some cases, P may continue to be adjusted up to a fixed number of iterations. In some cases, the update of P may be repeated as many times as necessary (eg, until a condition is satisfied). For example, after several adjustments to P, the update may be repeated until the cost function does not decrease or decreases insufficiently. In some cases, the value of the scaling term (P) in one or more of the iterations may be used, for example, for application to image data and/or for reading out for testing/troubleshooting purposes.

In some cases, the value for the scaling term P determined to minimize the cost function is performed within one frame time (eg, such that the value can be applied to image 415 ) or over multiple frames. there may be In some cases, if the value for the scaling term (P) determined to minimize the cost function takes several frames to determine, the previously determined value for the scaling term may continue to be used until the minimization process is complete. . The computation time associated with the minimization process may be based at least on the size of the cropped image 430 , the cropped SFFC map 440 , and any termination conditions associated with the minimization process. In an embodiment, the scaling term may be determined frame-by-frame (eg, in real-time) or periodically (eg, in the background).

The attenuation circuit 450 receives from the minimization circuit 445 the value of the scaling term P that minimizes the cost function FVAL(P) (eg, according to a desired termination condition). The attenuation circuit 450 also receives the previous value P _prev of the scaling term. For example, the previous value P _prev is based on a scaling term generated by the minimization circuit 445 based on a snapshot from the image data 405 received by the background processing path prior to receiving the image 415 . may be a value for As another example, the previous value (P _prev ) may be the value obtained for the scaling term during a previous iteration of adjusting the scaling term (e.g., before the minimization process to determine P that minimizes the cost function is complete), This previous value is therefore based on image 415 . The damping circuit 450 generates a damped scaling term P _damp based on at least P and P _prev . In one aspect, the attenuation of the scaling term (P) in the main processing path prevents an abrupt shift of the image due to the adjustment of the value of the scaling term (P). The attenuation circuit 450 may generate the attenuated scaling term P by applying an attenuation factor (eg, a temporal attenuation factor) to the scaling term P.

In some cases, the attenuation term may be implemented using alpha blending, infinite impulse response (IIR) filtering, and/or other attenuation techniques. The decay term may be based on a linear combination (eg, a weighted average) of a value of the scaling term P at the current time and one or more values of the scaling term P determined at a time prior to the current time. The damping term may be used to minimize relatively large and/or fast fluctuations associated with the applied scaling term (P). Different imaging systems and/or components thereof may be associated with different attenuation terms.

The combiner circuit 455 receives the attenuated scaling term (P _damp ) and applies the attenuated scaling term (P _damp ) to the SFFC map 435 to obtain a scaled SFFC map. The combiner circuit 455 may multiply the SFFC map 435 by the damped scaling term (P _damp ). The combiner circuit 460 receives the image data 405 in the main processing path and applies the scaled SFFC map to the image of the image data 405 . In one case, the image to which the damped scaling term P is applied is one captured subsequent to image 415 and/or image 415 used to generate the scaling term P (for which P _damp is determined). It may be more than one image. The combiner circuit 460 may subtract the scaled SFFC map from this image to obtain a corrected image (eg, a non-uniformity corrected image) that forms part of the corrected image data 410 .

In some cases, the corrected image data 410 may be further processed along the main processing path. In some cases, the further processing may include a spatial filtering operation and/or a temporal filtering operation. In an aspect, spatial filtering may apply spatial filter parameters to the corrected image data 410 . The temporal filtering may receive the pixel values output from the spatial filtering and perform temporal filtering on the pixel values in the temporal escape based on the temporal filter parameter and data related to the previous image(s). In some cases, temporal filtering may precede spatial filtering. In some cases, spatial and temporal filtering may occur together rather than two discrete sequential filtering operations. Other and/or alternative operations may be performed to obtain image data 405 and/or corrected image data 410 and/or to further process corrected image data 410 .

In some embodiments, the processing component 105 is configured to determine a scaling term P in background processing, frame by frame (eg, in real time), periodically (eg, every 100 images received; The image may be selected every few minutes, every hour, etc.), aperiodically (eg, based on image quality and/or in response to a user request for calibration). For example, the user and/or machine may instruct or request that a new scaling term (P) be determined if there is a defect in the image to which the previous scaling term has been applied. Once the scaling term (P) is determined, a scaled non-uniformity estimate can be applied to the received image as part of the main processing path. In some cases/applications, determining P may be part of the main processing path and may be performed in each frame.

In some embodiments, a snapshot may be taken after non-estimation correction is performed. 5 illustrates an example system 500 for facilitating image non-uniformity mitigation in accordance with one or more embodiments of the present disclosure. However, not all illustrated components are required, and one or more embodiments may include additional components not shown in the drawings. Changes in the arrangement and type of elements may be made without departing from the spirit or scope of the claims set forth herein. Additional components, other components, and/or fewer components may be provided. The description of system 400 of FIG. 4 applies generally to system 500 of FIG. 5, examples of differences are provided herein along with other descriptions.

In system 500 , a primary processing path includes performing non-uniformity correction on image data 505 to obtain corrected image data 510 (eg, non-uniformity corrected image data). The image data 505 and the corrected image data 510 may each include an image. In some cases, image data 505 may include an image to which gains and/or offsets have been applied.

In the main processing path, combiner circuit 515 receives image data 505 and applies the scaled SFFC map to image data 505 to obtain corrected image data 510 . The combiner circuit 515 subtracts the scaled SFFC map from the images of the image data 505 to obtain an image of the corrected image data 510 . The combiner circuit 520 applies the previous attenuated scaling term (P _{prev -d} ) to the SFFC map 525 to obtain a scaled SFFC map. The previous attenuated scaling term (P _{prev -d} ) may be a previous value (P _prev ) of the attenuated scaling term by the attenuation circuit 565 . The previous value of the scaling term (P _prev ) is computed by a minimization circuit 555 to minimize a cost function for a previous snapshot taken from image data 505 (eg, a snapshot from a previous time instance from image 530 ). may be a value determined by

The image 530 of the corrected image data 510 may be provided to the background processing path (eg, as a snapshot of the corrected image data 510 ). The cropping circuit 535 receives the image 530 , and crops the image 530 based on the defect data 540 to obtain a cropped image 545 . Defect data 540 may include data indicative of location, size, and/or other data relating to defects in image 530 . A portion of the SFFC map 525 corresponding to the cropped image 545 (eg, in size, location and shape) is cropped to provide a cropped SFFC map 550 . The cropped image 545 and cropped SFFC map 550 may be of different sizes and/or shapes (eg, non-rectangular maps/images).

Minimization circuitry 555 receives crop image 545 and non-uniformity SFFC map 550 and iteratively adjusts scalar value P to minimize cost function FVAL(P). In one embodiment, the cost function may be provided by:

는 크롭 이미지(545)이고 NU(예를 들어, 불균일 추정)는 크롭 SFFC 맵(550)이다. here

is a cropped image 545 and NU (eg, non-uniformity estimation) is a cropped SFFC map 550 .

The combiner circuit 560 receives the scaling term P from the minimization circuit 555 and combines it with the previous attenuated value of the scalar term P _{prev -d} to obtain a new value of the scalar term P _new . The combiner circuit 560 adds the previously attenuated value P _{prev -d} to the value P from the minimization circuit 555 to obtain a new value P _new of the scaling term. The attenuation circuit 565 receives the values of the scaling terms (P _new , P _{prev -d} ) and obtains an attenuated new value (P _{new -d} ) (not explicitly indicated in FIG. 5 ). The attenuated new value P _{new -d} may be provided to the combiner circuit 520 to scale the SFFC map 525 for correcting the image of the image data 505 . The attenuated new value P _{new -d} may be provided to the attenuation circuit 565 and the combiner circuit 560 . The attenuated new value P _{new -d} may be applied to an image subsequent to image 530 . In the system 500 of FIG. 4 , the value P from the minimization circuit 445 is attenuated and used to scale the SFFC map 435 , whereas in the system 500 of FIG. 5 the value from the minimization circuit 555 is used. (P) is not directly attenuated and used to scale the SFFC map 525 .

6 illustrates an example system 600 for iteratively adjusting a scalar value P to minimize a cost function FVAL(P) in accordance with one or more embodiments of the present disclosure. Although system 600 is primarily described herein with reference to imaging system 100 of FIG. 1 and the exemplary cost functions described in connection with FIGS. 4 and 5 , system 600 may be used with other systems and/or other cost functions. can be performed in relation to

으로 표시)와 스케일링된 크롭 SFFC 맵이 결합기 회로(620)에 제공된다. 크롭 SFFC 맵(605), 스케일링 된 크롭 SFFC 맵, 및 크롭 이미지(615)는 동일한 크기이다. 비제한적인 예로서, 크롭 SFFC 맵(605), 스케일링된 크롭 SFFC 맵, 및 크롭 이미지(615)는 32픽셀 × 32픽셀, 64픽셀 × 64픽셀, 32픽셀 × 64픽셀, 또는 다른 크기일 수 있다. 크롭 SFFC 맵(605), 스케일링된 크롭 SFFC 맵, 및 크롭 이미지(615)는 다른 크기 및/또는 모양(예를 들어, 직사각형이 아닌 맵/이미지)일 수 있다. 결합기 회로(620)는 크롭 SFFC 맵(605)으로부터 스케일링된 크롭 SFFC 맵을 감산하여

를 얻는다.A crop SFFC map 605 (denoted NU) and a scaling term P are provided to a combiner circuit 610 . The combiner circuit 610 multiplies the cropped SFFC map 605 by the scaling term P to obtain a scaled cropped SFFC map (P x NU). Cropped image(615)((

) and the scaled cropped SFFC map are provided to the combiner circuit 620 . The cropped SFFC map 605 , the scaled cropped SFFC map, and the cropped image 615 are the same size. As a non-limiting example, the cropped SFFC map 605, the scaled cropped SFFC map, and the cropped image 615 may be 32 pixels by 32 pixels, 64 pixels by 64 pixels, 32 pixels by 64 pixels, or other sizes. . The cropped SFFC map 605 , the scaled cropped SFFC map, and the cropped image 615 may be of different sizes and/or shapes (eg, non-rectangular maps/images). The combiner circuit 620 subtracts the scaled cropped SFFC map from the cropped SFFC map 605 to

get

는 보정된 이미지로 지칭될 수 있고(예를 들어, 보정된 크롭 이미지 또는 보정된 이미지 부분으로도 지칭됨) 크롭 이미지(615)로부터 스케일링된 크롭 SFFC 맵을 감산하여 형성된다.

may be referred to as a corrected image (eg, also referred to as a corrected cropped image or corrected image portion) and is formed by subtracting the scaled cropped SFFC map from the cropped image 615 .

를 이용하여 비용 함수의 값을 계산한다. 일 실시형태에서, 비용 함수는 다음에 의해 제공될 수 있다: The cost function calculation circuit 625 is

is used to calculate the value of the cost function. In one embodiment, the cost function may be provided by:

The scalar value adjustment circuit 630 receives the value FVAL(P) from the cost function calculation circuit 625 and adjusts the scaling term P. This adjustment is to provide a value of P that reduces the cost function FVAL(P). The adjusted scalar value is provided to the combiner circuit 610 by the scalar value adjustment circuit 630 . System 600 may then be performed again to iteratively adjust P to converge toward a value of P that minimizes the cost function FVAL(P). In this iterative approach, the non-uniformity estimation part (NU) is fixed while the scalar value P is iteratively updated.

7 depicts a flow diagram of an example process 700 for facilitating image non-uniformity mitigation in accordance with one or more embodiments of the present disclosure. Although process 700 is described herein for illustrative purposes primarily with reference to systems 100 , 400 , 500 of FIGS. 1 , 4 and 5 , process 700 is performed in connection with other systems for image non-uniformity mitigation. it might be One or more operations of FIG. 7 may be combined, omitted, and/or performed in a different order as desired.

At block 705 , image detector circuitry 165 captures image data (eg, thermal image data) in response to radiation (eg, infrared light) received by image detector circuitry 165 . In this regard, to reach image detector circuitry 165 , the radiation may propagate through the optical path of imaging system 100 to be received by image detector circuitry 165 . In some cases, image detector circuitry 165 and/or circuitry coupled to image detector circuitry 165 converts radiation into electrical signals (eg, voltages, currents, etc.) and generates pixel values based on the electrical signals. create In one aspect, the pixel values generated by the image detector circuitry 165 and/or related circuitry may be represented as digital numeric values generated based on electrical signals obtained by converting the detected infrared rays. For example, if the image detector circuit 165 includes or is coupled to an ADC circuit, the ADC circuit may generate a digital numeric value based on the electrical signal. In the case of an ADC circuit that can represent an electrical signal using 14 bits, the range of digital numeric values can be between 0 and 16,383.

At block 710 , the processing component 105 determines an image based on the image data. In an aspect, the processing component 105 may apply a gain and/or an offset to the image data to obtain an image. In some cases, processing component 105 may receive image data via image interface 120 . As an example, image data may be received from the image capture component 115 . As another example, image data may be received or retrieved from an image capture component external to imaging system 100 and/or memory of imaging system 100 or other system that stores captured image data.

At block 715 , the processing component 105 crops the image based on the identified defects in the image to obtain a cropped image. In an aspect, a defect may be determined/identified by a user and/or machine of the imaging system 100 (eg, via visual inspection by a human eye). For example, a user may provide an input to the processing component 105 to indicate a defect location in an image (eg, a pixel in an image that surrounds the image). In some cases, a defect may be the worst defect in the image. In some cases, if the image is free of defects, any location within the image may be provided to the processing component 105 to compute a scaling term of zero. For example, the user may perform the cropping and then select any location within the image to determine the scaling term (P). The end result of the minimization process leads to a zero scaling term because the cropped image has no defects. If the scaling term P is zero, the non-uniformity correction is set to zero and the image is not adversely affected by performing process 700 . At block 720, the processing component 105 crops the non-uniformity map (eg, SFFC map) according to the crop image to obtain a crop non-uniformity map. The non-uniformity map can be cut out at positions corresponding to the positions of the defects in the image. The crop non-uniformity map is associated with the same size, shape and position as the crop image.

At block 725 , the processing component 105 determines a value of the scaling term P based on the cost function. In an aspect, the processing component 105 may perform a minimization to determine a value of the scaling term P that minimizes the cost function. The value of the cost function may be based on the variation between the pixel values of the corrected image after the scaling term (P) and the crop non-uniformity estimate are applied to the crop image. In an aspect, the value of the cost function may be associated with the quality of the image after non-uniformity correction. In one embodiment, the cost function FVAL(P) may be provided as:

At block 730, the processing component 105 scales the non-uniformity map based on the scaling term P to obtain a scaled non-uniformity map. In an aspect, the non-uniformity map may be multiplied by a scaling term (P). In an aspect as in this system 400 , the scaling term P is attenuated to obtain an attenuated scaling term P _damp and this attenuated scaling term P _damp is applied directly to the non-uniformity map. For example, the non-uniformity map is multiplied (eg, scaled) by the scaling term P to obtain a scaled non-uniformity map. In an aspect as in system 500 , the scaling term P may be adjusted based on the previous scaling term P _prev to obtain a new scaling term P _new . A new scaling term (P _new ) may be applied to the non-uniformity map, or a new scaling term (P _new ) may be applied to the non-uniformity map after being attenuated.

At block 735, the processing component 105 applies the scaled non-uniformity map to one or more images to obtain one or more non-uniformity corrected images. In some cases, the scaled non-uniformity map is applied to the captured image data subsequent to the captured image data at block 705 . In some cases, the scaled non-uniformity map is applied to the image determined at block 710 . In one aspect, the non-uniformity corrected image(s) are further processed (eg, temporal and/or spatial filtering, subsequent processing in the main processing path, such as a radiation conversion operation for an infrared image, etc.), stored, and/or or may be displayed. In some embodiments, the scaling term may be determined frame by frame (eg, in real time) in the main processing path rather than as a background process. In some cases, low-frequency shading reduction may be performed (eg, to mitigate large-scale shading correction from field irradiance) after or before non-uniformity correction based on the scaling term P.

Although systems 400, 500, and 700 are primarily described in situations where image data is captured and non-uniformity correction is performed (e.g., a scaling term (P) is generated for a snapshot when image data is captured in near real-time) , the system 400, 500, 700 may generate a scaling term P for previously captured/generated (eg, by any imaging system with appropriate functionality) and/or stored image data and/or images. and/or perform non-uniformity correction. The stored image data and/or images (eg, stored snapshots and/or video clips) and related data may be accessed/retrieved for processing at a later point in time. The related data may include metadata provided in a header appended to the image data and/or otherwise stored with an appropriate indication of the image data and/or data association with the image. For example, the metadata may include operating conditions such as the operating temperature of the imaging system 100 when the image was captured, the date and time the image data was captured, and/or other data.

8A shows an image with defects (surrounded by dotted ellipses 805, 810, and 815, also referred to as defects 805, 810, and 815). 8B illustrates an image obtained by performing non-uniformity mitigation on the image of FIG. 8A in accordance with one or more embodiments of the present disclosure. In one case, non-uniformity mitigation, in accordance with one or more embodiments, crops the image at the location of the defect 805 (eg, the defect 805 is considered the worst defect), and slices that portion of the SFFC map. based on truncation, using a cost function to determine a scaling term, and so on. The scaling term determined based on the defects 805 provides a non-uniformity estimate applicable to the image of FIG. 8A to mitigate the defects 810 and 815 as well as the defects 805 . In some cases, low frequency shading reduction may be performed to remove shading from the corners of the images of FIGS. 8A and 8B .

Where applicable, the various embodiments provided by this disclosure may be implemented using hardware, software, or a combination of hardware and software. Also, where applicable, various hardware components and/or software components described herein may be combined into complex components comprising software, hardware, and/or both without departing from the spirit of the present disclosure. Where applicable, various hardware components and/or software components described herein may be separated into subcomponents comprising software, hardware, or both without departing from the spirit of the present disclosure. Also, where applicable, it is contemplated that software components may be implemented as hardware components and vice versa.

Software according to the present invention, such as non-transitory instructions, program code, and/or data, may be stored on one or more non-transitory machine-readable media. It is also contemplated that the software identified herein may be implemented using one or more general purpose or special purpose computers and/or computer systems, networks, and/or the like. Where applicable, the order of the various steps described herein may be varied, combined into multiple steps, and/or separated into substeps to provide the functionality described herein.

The foregoing description is not intended to limit the disclosure to the precise form disclosed or to the particular field of use. The embodiments described above illustrate, but do not limit, the present invention. It is contemplated that various alternative embodiments and/or modifications to the invention, whether expressly described or implied herein, are possible in light of the present disclosure. Accordingly, the scope of the present invention is defined only by the following claims.

Claims (21)

cropping the first image based on a defect in the first image to obtain a first cropped image;
cropping a supplemental flat field correction (SFFC) map based on the defects in the first image to obtain a first cropped SFFC map;
determining a first scaling value of a scaling term based at least on a cost function, the first cropped image, and the first cropped SFFC map; and
and scaling the SFFC map based on the first scaling value to obtain a first scaled SFFC map. According to claim 1,
and applying the scaled SFFC map to one or more images. 3. The method of claim 2,
wherein the one or more images include the first image. According to claim 1,
capturing image data by the focal plane array; and
processing the image data to obtain the first image, the processing comprising applying at least one of a gain and an offset to the image data;
Further comprising a, image non-uniformity mitigation method. 5. The method of claim 4,
wherein the focal plane array comprises a plurality of infrared detectors. According to claim 1,
and applying an attenuation factor to the first scaling value to obtain an attenuated scaling value, wherein the scaling is based on the attenuated scaling value. According to claim 1,
and determining a first scaling value comprises iteratively adjusting a value of the scaling term to obtain the first scaling value that minimizes the cost function. According to claim 1,
and the cost function is based on the first cropped image, the value of the scaling term, and the first cropped SFFC map. 9. The method of claim 8,
and the cost function is based on a gradient of a difference between the first cropped SFFC map and the first cropped image scaled by the value of the scaling term. According to claim 1,
cropping the second image based on defects in the second image to obtain a second cropped image;
cropping the SFFC map based on defects in the second image to obtain a second cropped SFFC map;
determining a second scaling value of the scaling term based on at least the cost function, the second cropped image, and the second cropped SFFC map; and
scaling the SFFC map based at least on the second scaling value to obtain a second scaled SFFC map;
Further comprising, the image non-uniformity mitigation method. 11. The method of claim 10,
and the second scaled SFFC map is further based on the first scaling value. According to claim 1,
wherein the first image comprises a thermal image. a memory configured to store a Supplemental Flat Field Correction (SFFC) map; and
crop the first image based on defects in the first image to obtain a first cropped image; crop the SFFC map based on defects in the first image to obtain a first cropped SFFC map; determine a first scaling value of a scaling term based on at least a cost function, the first cropped image, and the first cropped SFFC map; and a processing circuit configured to scale the SFFC map based on the first scaling value to obtain a first scaled SFFC map;
Including, image non-uniformity mitigation system. 14. The method of claim 13,
and the processing circuitry is further configured to apply the first scaled SFFC map to one or more images. 14. The method of claim 13,
the processing circuitry is further configured to apply an attenuation factor to the first scaling value to obtain an attenuated scaling value, and the processing circuitry is configured to scale the SFFC map based on the attenuated scaling value. mitigation system. 14. The method of claim 13,
and a focal plane array configured to capture thermal image data, wherein the processing circuitry is further configured to apply at least one of a gain and an offset to the thermal image data to obtain the first image. 17. The method of claim 16,
wherein the focal plane array comprises a plurality of microbolometers. 14. The method of claim 13,
and the processing circuitry is configured to determine the first scaling value by iteratively adjusting a value of the scaling term to obtain the first scaling value that minimizes the cost function. 14. The method of claim 13,
and the cost function is based on the first cropped image, the value of the scaling term, and the first cropped SFFC map. 14. The method of claim 13,
The processing circuit is
crop a second image based on defects in the second image to obtain a second cropped image;
crop the SFFC map based on defects in the second image to obtain a second cropped SFFC map;
determine a second scaling value of the scaling term based on at least the cost function, the second cropped image, and the second cropped SFFC map; and
and scale the SFFC map based at least on the second scaling value to obtain a second scaled SFFC map. 21. The method of claim 20,
and the second scaled SFFC map is further based on the first scaling value.

Images