TW202211160A - Dual sensor imaging system and calibration method thereof - Google Patents

Dual sensor imaging system and calibration method thereof Download PDF

Info

Publication number
TW202211160A
TW202211160A TW109146764A TW109146764A TW202211160A TW 202211160 A TW202211160 A TW 202211160A TW 109146764 A TW109146764 A TW 109146764A TW 109146764 A TW109146764 A TW 109146764A TW 202211160 A TW202211160 A TW 202211160A
Authority
TW
Taiwan
Prior art keywords
infrared
image
color
sensor
test image
Prior art date
Application number
TW109146764A
Other languages
Chinese (zh)
Other versions
TWI764484B (en
Inventor
彭詩淵
鄭書峻
黃旭鍊
李運錦
賴國銘
Original Assignee
聚晶半導體股份有限公司
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by 聚晶半導體股份有限公司 filed Critical 聚晶半導體股份有限公司
Priority to US17/191,701 priority Critical patent/US11418719B2/en
Publication of TW202211160A publication Critical patent/TW202211160A/en
Application granted granted Critical
Publication of TWI764484B publication Critical patent/TWI764484B/en

Links

Images

Classifications

    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N23/00—Cameras or camera modules comprising electronic image sensors; Control thereof
    • H04N23/45—Cameras or camera modules comprising electronic image sensors; Control thereof for generating image signals from two or more image sensors being of different type or operating in different modes, e.g. with a CMOS sensor for moving images in combination with a charge-coupled device [CCD] for still images
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T7/00—Image analysis
    • G06T7/50—Depth or shape recovery
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N23/00—Cameras or camera modules comprising electronic image sensors; Control thereof
    • H04N23/60—Control of cameras or camera modules
    • H04N23/64—Computer-aided capture of images, e.g. transfer from script file into camera, check of taken image quality, advice or proposal for image composition or decision on when to take image
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N23/00—Cameras or camera modules comprising electronic image sensors; Control thereof
    • H04N23/70—Circuitry for compensating brightness variation in the scene
    • H04N23/743—Bracketing, i.e. taking a series of images with varying exposure conditions

Landscapes

  • Engineering & Computer Science (AREA)
  • Multimedia (AREA)
  • Signal Processing (AREA)
  • Human Computer Interaction (AREA)
  • Computer Vision & Pattern Recognition (AREA)
  • Physics & Mathematics (AREA)
  • General Physics & Mathematics (AREA)
  • Theoretical Computer Science (AREA)
  • Color Television Image Signal Generators (AREA)
  • Studio Devices (AREA)
  • Image Processing (AREA)
  • Cameras In General (AREA)
  • Transforming Light Signals Into Electric Signals (AREA)
  • Traffic Control Systems (AREA)
  • Measurement Of Optical Distance (AREA)
  • Radar Systems Or Details Thereof (AREA)
  • Air Bags (AREA)
  • Burglar Alarm Systems (AREA)

Abstract

A dual sensor imaging system and a calibration method thereof are provided. The dual sensor imaging system includes at least one color sensor, at least one infrared ray (IR) sensor, a storage device and a processor. The processor is configured to load and execute a computer program stored in the storage device to: control the color sensor and the IR sensor to respectively capture multiple color images and multiple IR images of an imaging scene using multiple capturing conditions; calculate multiple color image parameters of the color images captured under each capturing condition and multiple IR image parameters of the IR images captured under each capturing condition, and use the same to calculate a difference between a luminance of the color images and a luminance of the IR images; and determines an exposure setting adapted for the color sensor and the IR sensor according to the calculated difference.

Description

雙感測器攝像系統及其校準方法Dual-sensor camera system and its calibration method

本發明是有關於一種攝像系統及方法，且特別是有關於一種雙感測器攝像系統及其校準方法。The present invention relates to a camera system and method, and more particularly, to a dual-sensor camera system and a calibration method thereof.

相機的曝光條件（包括光圈、快門、感光度）會影響所拍攝影像的品質，因此許多相機在拍攝影像的過程中會自動調整曝光條件，以獲得清晰且明亮的影像。然而，在低光源或是背光等高反差的場景中，相機調整曝光條件的結果可能會產生雜訊過高或是部分區域過曝的結果，無法兼顧所有區域的影像品質。The camera's exposure conditions (including aperture, shutter, and sensitivity) affect the quality of the images captured, so many cameras automatically adjust exposure conditions during image capture to obtain clear and bright images. However, in scenes with high contrast such as low light source or backlight, the result of camera adjustment of exposure conditions may result in excessive noise or overexposure in some areas, and the image quality in all areas cannot be balanced.

對此，目前技術有採用一種新的影像感測器架構，其是利用紅外線（IR）感測器高光敏感度的特性，在影像感測器的色彩像素中穿插配置IR像素，以輔助亮度偵測。舉例來說，圖1是習知使用影像感測器擷取影像的示意圖。請參照圖1，習知的影像感測器10中除了配置有紅（R）、綠（G）、藍（B）等顏色像素外，還穿插配置有紅外線（I）像素。藉此，影像感測器10能夠將R、G、B顏色像素所擷取的色彩資訊12與I像素所擷取的亮度資訊14結合，而獲得色彩及亮度適中的影像16。In this regard, the current technology adopts a new image sensor architecture, which utilizes the characteristics of high light sensitivity of infrared (IR) sensors to intersperse and configure IR pixels among the color pixels of the image sensor to assist brightness detection. Measurement. For example, FIG. 1 is a schematic diagram of conventionally using an image sensor to capture images. Referring to FIG. 1 , in addition to red (R), green (G), blue (B) and other color pixels, the conventional image sensor 10 is also interspersed with infrared (I) pixels. Thereby, the image sensor 10 can combine the color information 12 captured by the R, G, and B color pixels with the luminance information 14 captured by the I pixel to obtain an image 16 with moderate color and brightness.

然而，在上述單一影像感測器的架構下，影像感測器中每個像素的曝光條件相同，因此只能選擇較適用於顏色像素或紅外線像素的曝光條件來擷取影像，結果仍無法有效地利用兩種像素的特性來改善所擷取影像的影像品質。However, in the above-mentioned single image sensor structure, the exposure conditions of each pixel in the image sensor are the same, so only the exposure conditions that are more suitable for color pixels or infrared pixels can be selected to capture images, and the result is still ineffective. The characteristics of the two pixels are used to improve the image quality of the captured image.

本發明提供一種雙感測器攝像系統及其校準方法，利用獨立配置的色彩及紅外線感測器分別擷取不同拍攝條件下的多張影像，據以進行影像對位及亮度匹配，並應用於後續擷取的影像，藉此可提高所擷取影像的影像品質。The present invention provides a dual-sensor camera system and a calibration method thereof. The independently configured color and infrared sensors are used to capture multiple images under different shooting conditions, so as to perform image alignment and brightness matching, and are applied to The image captured subsequently can improve the image quality of the captured image.

本發明的雙感測器攝像系統包括至少一個色彩感測器、至少一個紅外線感測器、儲存裝置以及耦接所述色彩感測器、紅外光感測器及儲存裝置的處理器。所述處理器經配置以載入並執行儲存在儲存裝置中的電腦程式以：控制色彩感測器及紅外線感測器採用多個拍攝條件分別擷取一攝像場景的多張色彩影像及多張紅外線影像；計算在各拍攝條件下擷取的色彩影像的多個色彩影像參數以及在各拍攝條件下擷取的紅外線影像的多個紅外線影像參數，用以計算色彩影像的亮度及紅外線影像的亮度之間的差異；以及根據所計算的差異，決定適於色彩感測器及紅外線感測器的曝光設定。The dual-sensor camera system of the present invention includes at least one color sensor, at least one infrared sensor, a storage device, and a processor coupled to the color sensor, the infrared light sensor, and the storage device. The processor is configured to load and execute a computer program stored in the storage device to: control the color sensor and the infrared sensor to capture a plurality of color images and a plurality of images of a camera scene using a plurality of shooting conditions, respectively Infrared image; calculate a plurality of color image parameters of the color image captured under each shooting condition and a plurality of infrared image parameters of the infrared image captured under each shooting condition, to calculate the brightness of the color image and the brightness of the infrared image difference between them; and determining exposure settings suitable for the color sensor and the infrared sensor according to the calculated difference.

本發明的雙感測器攝像系統的校準方法，適用於包括至少一個色彩感測器、至少一個紅外線感測器及處理器的雙感測器攝像系統。所述方法包括下列步驟：控制色彩感測器及紅外線感測器採用多個拍攝條件分別擷取一攝像場景的多張色彩影像及多張紅外線影像；計算在各拍攝條件下擷取的色彩影像的多個色彩影像參數以及在各拍攝條件下擷取的紅外線影像的多個紅外線影像參數，用以計算色彩影像的亮度及紅外線影像的亮度之間的差異；以及根據所計算的差異，決定適於色彩感測器及紅外線感測器的曝光設定。The calibration method for a dual-sensor camera system of the present invention is suitable for a dual-sensor camera system comprising at least one color sensor, at least one infrared sensor and a processor. The method includes the following steps: controlling a color sensor and an infrared sensor to capture a plurality of color images and a plurality of infrared images of a shooting scene respectively using a plurality of shooting conditions; calculating the color images captured under each shooting condition A plurality of color image parameters and a plurality of infrared image parameters of the infrared image captured under each shooting condition are used to calculate the difference between the brightness of the color image and the brightness of the infrared image; and according to the calculated difference, determine the appropriate Exposure settings for color sensor and infrared sensor.

基於上述，本發明的雙感測器攝像系統及其校準方法，在獨立配置的色彩感測器及紅外線感測器上採用不同拍攝條件擷取多張影像，並根據這些影像中對應像素的位置關係和這些影像之間的亮度差異，決定適於色彩感測器及紅外線感測器的曝光及對位設定，用以對後續擷取的影像進行影像對位及亮度匹配，而可提高所攝影像的影像品質。Based on the above, in the dual-sensor camera system and the calibration method thereof of the present invention, a plurality of images are captured using different shooting conditions on the independently configured color sensor and the infrared sensor, and the positions of the corresponding pixels in the images are captured according to the positions of the corresponding pixels in the images. The relationship and the brightness difference between these images determine the exposure and alignment settings suitable for the color sensor and the infrared sensor, which are used to perform image alignment and brightness matching on the subsequently captured images, which can improve the photographic image quality.

圖2是依照本發明一實施例所繪示的使用影像感測器擷取影像的示意圖。請參照圖2，本發明實施例的影像感測器20採用獨立配置色彩感測器22與紅外線（IR）感測器24的雙感測器架構，利用色彩感測器22與紅外線感測器24各自的特性，採用適於當前攝像場景的多個曝光條件分別擷取多張影像，並從中選擇曝光條件適當的色彩影像22a與紅外線影像24a，透過影像融合的方式，使用紅外線影像24a來補足色彩影像22a中缺乏的紋理細節，從而獲得色彩及紋理細節均佳的場景影像26。FIG. 2 is a schematic diagram of capturing an image using an image sensor according to an embodiment of the present invention. Referring to FIG. 2 , the image sensor 20 of the embodiment of the present invention adopts a dual-sensor structure in which a color sensor 22 and an infrared (IR) sensor 24 are independently configured, and the color sensor 22 and the infrared sensor are used. 24 have their respective characteristics, use multiple exposure conditions suitable for the current shooting scene to capture multiple images respectively, and select the color image 22a and infrared image 24a with appropriate exposure conditions from them, and use the infrared image 24a to complement the image fusion method. The lack of texture details in the color image 22a results in a scene image 26 with good color and texture details.

圖3是依照本發明一實施例所繪示的雙感測器攝像系統的方塊圖。請參照圖3，本實施例的雙感測器攝像系統30可配置於手機、平板電腦、筆記型電腦、導航裝置、行車紀錄器、數位相機、數位攝影機等電子裝置中，用以提供攝像功能。雙感測器攝像系統30包括至少一個色彩感測器32、至少一個紅外線感測器34、儲存裝置36及處理器38，其功能分述如下：FIG. 3 is a block diagram of a dual-sensor camera system according to an embodiment of the present invention. Please refer to FIG. 3 , the dual-sensor camera system 30 of this embodiment can be configured in electronic devices such as mobile phones, tablet computers, notebook computers, navigation devices, driving recorders, digital cameras, digital cameras, etc., to provide a camera function . The dual-sensor camera system 30 includes at least one color sensor 32, at least one infrared sensor 34, a storage device 36 and a processor 38, and its functions are described as follows:

色彩感測器32例如包括電荷耦合元件（Charge Coupled Device，CCD）、互補性氧化金屬半導體（Complementary Metal-Oxide Semiconductor，CMOS）元件或其他種類的感光元件，而可感測光線強度以產生攝像場景的影像。色彩感測器32例如是紅綠藍（RGB）影像感測器，其中包括紅（R）、綠（G）、藍（B）顏色像素，用以擷取攝像場景中的紅光、綠光、藍光等色彩資訊，並將這些色彩資訊合成以生成攝像場景的色彩影像。The color sensor 32 includes, for example, a Charge Coupled Device (CCD), a Complementary Metal-Oxide Semiconductor (CMOS) device, or other types of photosensitive devices, and can sense light intensity to generate a camera scene image. The color sensor 32 is, for example, a red-green-blue (RGB) image sensor, which includes red (R), green (G), and blue (B) color pixels for capturing red light and green light in the camera scene , blue light and other color information, and combine these color information to generate a color image of the camera scene.

紅外線感測器34例如包括CCD、CMOS元件或其他種類的感光元件，其經由調整感光元件的波長感測範圍，而能夠感測紅外光。紅外線感測器34例如是以上述感光元件作為像素來擷取攝像場景中的紅外光資訊，並將這些紅外光資訊合成以生成攝像場景的紅外線影像。The infrared sensor 34 includes, for example, a CCD, a CMOS element or other types of photosensitive elements, which can sense infrared light by adjusting the wavelength sensing range of the photosensitive element. The infrared sensor 34, for example, uses the above-mentioned photosensitive elements as pixels to capture infrared light information in the imaging scene, and synthesizes the infrared light information to generate an infrared image of the imaging scene.

儲存裝置36例如是任意型式的固定式或可移動式隨機存取記憶體（Random Access Memory，RAM）、唯讀記憶體（Read-Only Memory，ROM）、快閃記憶體（Flash memory）、硬碟或類似元件或上述元件的組合，而用以儲存可由處理器38執行的電腦程式。在一些實施例中，儲存裝置36例如還可儲存由色彩感測器32所擷取的色彩影像及紅外線感測器34所擷取的紅外線影像。The storage device 36 is, for example, any type of fixed or removable random access memory (Random Access Memory, RAM), read-only memory (Read-Only Memory, ROM), flash memory (Flash memory), hard drive A disk or similar element, or a combination of the foregoing, for storing computer programs executable by the processor 38 . In some embodiments, the storage device 36 may also store, for example, the color image captured by the color sensor 32 and the infrared image captured by the infrared sensor 34 .

處理器38例如是中央處理單元（Central Processing Unit，CPU），或是其他可程式化之一般用途或特殊用途的微處理器（Microprocessor）、微控制器（Microcontroller）、數位訊號處理器（Digital Signal Processor，DSP）、可程式化控制器、特殊應用積體電路（Application Specific Integrated Circuits，ASIC）、可程式化邏輯裝置（Programmable Logic Device，PLD）或其他類似裝置或這些裝置的組合，本發明不在此限制。在本實施例中，處理器38可從儲存裝置36載入電腦程式，以執行本發明實施例的雙感測器攝像系統的校準方法。The processor 38 is, for example, a central processing unit (CPU), or other programmable general-purpose or special-purpose microprocessors (Microprocessors), microcontrollers (Microcontrollers), and digital signal processors (Digital Signal Processors). Processor, DSP), programmable controller, application specific integrated circuit (Application Specific Integrated Circuits, ASIC), programmable logic device (Programmable Logic Device, PLD) or other similar devices or a combination of these devices, the present invention does not this limit. In this embodiment, the processor 38 can load a computer program from the storage device 36 to execute the calibration method of the dual-sensor camera system according to the embodiment of the present invention.

基於色彩感測器32與紅外線感測器34的特性（例如解析度、波長範圍、視野（field of view，FOV））不同，本發明實施例提供一種校準方法，可在生產階段（production stage）對裝配在雙感測器攝像系統30上的色彩感測器32與紅外線感測器34進行校準，以平衡色彩感測器32與紅外線感測器34在不同拍攝條件下的差異，此校準結果將儲存於儲存裝置36，而可在後續執行階段（run stage）中用以作為對擷取影像進行調整的依據。Based on the different characteristics (eg, resolution, wavelength range, field of view (FOV)) of the color sensor 32 and the infrared sensor 34 , embodiments of the present invention provide a calibration method that can be used in the production stage. The color sensor 32 and the infrared sensor 34 mounted on the dual-sensor camera system 30 are calibrated to balance the difference between the color sensor 32 and the infrared sensor 34 under different shooting conditions. The calibration result It will be stored in the storage device 36 and can be used as a basis for adjusting the captured image in subsequent run stages.

圖4是依照本發明一實施例所繪示的雙感測器攝像系統的校準方法的流程圖。請同時參照圖3及圖4，本實施例的方法適用於上述的雙感測器攝像系統30，適於在生產階段對雙感測器攝像系統30的色彩感測器32與紅外線感測器34進行校準，以下即搭配雙感測器攝像系統30的各項元件說明本實施例的校準方法的詳細步驟。FIG. 4 is a flowchart of a calibration method of a dual-sensor camera system according to an embodiment of the present invention. Please refer to FIG. 3 and FIG. 4 at the same time. The method of this embodiment is applicable to the above-mentioned dual-sensor camera system 30, and is suitable for the color sensor 32 and the infrared sensor of the dual-sensor camera system 30 in the production stage. 34 to perform calibration, and the following describes the detailed steps of the calibration method of this embodiment in conjunction with various elements of the dual-sensor camera system 30 .

在步驟S402中，將至少一個色彩感測器32及至少一個紅外線感測器34裝配於雙感測器攝像系統30。其中，色彩感測器32及紅外線感測器34例如是由機械人裝配於影像感測器中，例如圖2所示的裝配在影像感測器20中的色彩感測器22與紅外線感測器24。In step S402 , at least one color sensor 32 and at least one infrared sensor 34 are assembled in the dual-sensor camera system 30 . The color sensor 32 and the infrared sensor 34 are assembled in the image sensor by, for example, a robot, such as the color sensor 22 and the infrared sensor assembled in the image sensor 20 shown in FIG. 2 . device 24.

在步驟S404中，由處理器38執行色彩感測器32及紅外線感測器34之間對位的校準。其中，處理器38例如會執行暴力法（bruteforce）、光流法（optical flow）、單應性變換法（homography）或局部翹曲法（local warping）等影像對位演算法，以對色彩感測器32所擷取的色彩影像與紅外線感測器34所擷取的紅外線影像進行對位，後文將描述其詳細的實施方式。In step S404 , the processor 38 performs alignment calibration between the color sensor 32 and the infrared sensor 34 . The processor 38 may execute image alignment algorithms such as bruteforce, optical flow, homography, or local warping, etc., so as to detect the color perception. The color image captured by the detector 32 is aligned with the infrared image captured by the infrared sensor 34, and the detailed implementation will be described later.

在步驟S406中，由處理器38執行色彩感測器32及紅外線感測器34在不同拍攝條件下的亮度匹配的校準。其中，處理器38例如會計算在不同拍攝條件下擷取的色彩影像及紅外線影像之間的差異，據以決定適於色彩感測器32及紅外線感測器34的曝光設定，後文將描述其詳細的實施方式。In step S406, the processor 38 performs the calibration of the brightness matching of the color sensor 32 and the infrared sensor 34 under different shooting conditions. The processor 38, for example, calculates the difference between the color image and the infrared image captured under different shooting conditions, so as to determine the exposure settings suitable for the color sensor 32 and the infrared sensor 34, which will be described later. its detailed implementation.

對於上述對位的校準，圖5是依照本發明一實施例所繪示的雙感測器攝像系統的對位校準方法的流程圖。請同時參照圖3及圖5，本實施例的方法適用於上述的雙感測器攝像系統30，以下即搭配雙感測器攝像系統30的各項元件說明本實施例的對位校準方法的詳細步驟。For the above-mentioned alignment calibration, FIG. 5 is a flowchart of a method for alignment and calibration of a dual-sensor camera system according to an embodiment of the present invention. Please refer to FIG. 3 and FIG. 5 at the same time, the method of this embodiment is applicable to the above-mentioned dual-sensor camera system 30 , and the following describes the alignment and calibration method of this embodiment in combination with various elements of the dual-sensor camera system 30 . detailed steps.

在步驟S502中，由處理器38控制色彩感測器32及紅外線感測器34分別拍攝具有特殊圖案的測試圖（test chart），以獲得一色彩測試影像及一紅外線測試影像。其中，所述特殊圖案例如是黑白棋盤圖案，或是其他可明顯區別出特徵的圖案，在此不設限。In step S502 , the processor 38 controls the color sensor 32 and the infrared sensor 34 to capture a test chart with a special pattern, respectively, to obtain a color test image and an infrared test image. Wherein, the special pattern is, for example, a black and white checkerboard pattern, or other patterns that can clearly distinguish features, which are not limited herein.

在步驟S504中，由處理器38偵測色彩測試影像及紅外線測試影像中的特殊圖案的多個特徵點。In step S504, the processor 38 detects a plurality of feature points of the special pattern in the color test image and the infrared test image.

在一些實施例中，處理器38例如會將每張色彩測試影像及紅外線測試影像切割為多個區塊，並執行一特徵偵測演算法以偵測各個區塊內的至少一個特徵點。其中，處理器38例如是根據自身的計算能力決定從各個區塊偵測出特徵點的數目，而所述的特徵偵測演算法例如是哈里斯邊角偵測法（Harris corner detection）。在一些實施例中，處理器38例如會選擇每個區塊中的邊緣像素，或是每個區塊中具有高區域偏離（local deviation）的像素，作為所偵測的特徵點，在此不設限。In some embodiments, the processor 38, for example, divides each color test image and the infrared test image into a plurality of blocks, and executes a feature detection algorithm to detect at least one feature point in each block. The processor 38 determines, for example, the number of feature points detected from each block according to its own computing capability, and the feature detection algorithm is, for example, Harris corner detection. In some embodiments, the processor 38 selects, for example, edge pixels in each block, or pixels with high local deviation in each block, as the detected feature points. set limits.

在步驟S506中，由處理器38執行影像對位演算法以根據色彩測試影像及紅外線測試影像中相對應的特徵點之間的位置關係，計算色彩測試影像及紅外線測試影像之間的匹配關係，用以對後續擷取的色彩影像及紅外線影像進行對位。其中，在執行影像對位時，處理器38例如會取得色彩影像中的所有特徵點以及紅外線影像中的所有特徵點，從而針對這些特徵點執行影像對位演算法。In step S506, the processor 38 executes an image alignment algorithm to calculate the matching relationship between the color test image and the infrared test image according to the positional relationship between the corresponding feature points in the color test image and the infrared test image, It is used to align the subsequently captured color image and infrared image. Wherein, when performing image alignment, the processor 38, for example, obtains all feature points in the color image and all feature points in the infrared image, so as to execute an image alignment algorithm for these feature points.

在一些實施例中，當所攝像場景為平面場景時，處理器38會針對從色彩測試影像中偵測出的特徵點中的一個指定特徵點，在紅外線測試影像中的對應位置移動一個包括多個像素的補丁（patch），來搜尋紅外線測試影像中對應於此指定特徵點的對應特徵點。其中，處理器38例如是以紅外線測試影像中對應於此指定特徵點的像素為中心，在其周圍移動補丁，並將位於補丁內的像素與色彩測試影像中位於指定特徵點周圍的像素進行比較，直到補丁內的像素與指定特徵點周圍的像素匹配（例如所有像素的像素值的差值總和小於預定門檻值）為止。最終，處理器38即可將達成匹配時補丁所在位置的中心點像素判定為對應於指定特徵點的對應特徵點。處理器38將重複執行上述匹配動作，直到獲得所有特徵點的對應關係。In some embodiments, when the captured scene is a flat scene, the processor 38 will move a corresponding position in the infrared test image by one including multiple points for a specified feature point among the feature points detected from the color test image. A patch of pixels is used to search for the corresponding feature point in the infrared test image corresponding to the specified feature point. The processor 38, for example, takes the pixel corresponding to the specified feature point in the infrared test image as the center, moves the patch around it, and compares the pixels located in the patch with the pixels located around the specified feature point in the color test image , until the pixels in the patch match the pixels around the specified feature point (eg, the sum of the differences of the pixel values of all pixels is less than a predetermined threshold). Finally, the processor 38 can determine the center point pixel at the position of the patch when the matching is achieved as the corresponding feature point corresponding to the specified feature point. The processor 38 will repeatedly perform the above matching action until the correspondence of all feature points is obtained.

之後，處理器38例如會執行隨機抽樣一致（RANdom SAmple Consensus，RANSAC）演算法以建立單應性變換矩陣（homography transformation matrix），如下：

Figure 02_image001
Afterwards, the processor 38 performs, for example, a random sampling consensus (RANdom SAmple Consensus, RANSAC) algorithm to create a homography transformation matrix, as follows:
Figure 02_image001

其中，(x ,y )代表色彩測試影像中的指定特徵點的位置，(x’ ,y’ )則代表紅外線測試影像中的對應特徵點的位置，a~h代表變數。處理器38例如會將色彩測試影像中的各個特徵點與紅外線測試影像中的對應特徵點的位置帶入上述的單應性變換矩陣求解，從而以所求得的解作為色彩測試影像及紅外線測試影像之間的匹配關係。Among them, ( x , y ) represents the position of the specified feature point in the color test image, ( x' , y' ) represents the position of the corresponding feature point in the infrared test image, and a~h represent variables. For example, the processor 38 brings the positions of each feature point in the color test image and the corresponding feature point in the infrared test image into the above-mentioned homography transformation matrix for solution, so that the obtained solution is used as the color test image and the infrared test image. matching relationship between images.

在一些實施例中，當所攝像場景為具有多個深度的場景時，由於色彩感測器32及紅外線感測器34之間具有視差（parallax），其所擷取的影像會有像差（aberration），因此需要針對不同深度的影像平面計算其匹配關係。此時，處理器38會利用色彩測試影像及紅外線測試影像計算攝像場景的多個深度，據以將攝像場景區分為不同深度的多個深度場景（例如區分為近景和遠景）。其中，處理器38例如會針對各個深度場景，建立一個二次方程式（quadratic equation），如下：

Figure 02_image003
In some embodiments, when the captured scene is a scene with multiple depths, due to the parallax between the color sensor 32 and the infrared sensor 34, the captured image will have aberration ( aberration), so the matching relationship needs to be calculated for the image planes of different depths. At this time, the processor 38 uses the color test image and the infrared test image to calculate multiple depths of the shooting scene, so as to distinguish the shooting scene into multiple depth scenes with different depths (eg, close-range and far-range). The processor 38 may, for example, establish a quadratic equation for each depth scene, as follows:
Figure 02_image003

其中，(x ,y )代表色彩測試影像中的指定特徵點的位置，(x’ ,y’ )則代表紅外線測試影像中的對應特徵點的位置，a~f代表變數。處理器38例如會將色彩測試影像中的各個特徵點與紅外線測試影像中的對應特徵點的位置帶入上述的二次方程式求解，從而以所求得的解作為色彩測試影像及紅外線測試影像之間的匹配關係。Among them, ( x , y ) represents the position of the specified feature point in the color test image, ( x' , y' ) represents the position of the corresponding feature point in the infrared test image, and a~f represent variables. For example, the processor 38 brings the positions of each feature point in the color test image and the corresponding feature point in the infrared test image into the quadratic equation to solve, so that the obtained solution is used as the difference between the color test image and the infrared test image. matching relationship between.

另一方面，對於上述亮度匹配的校準，圖6是依照本發明一實施例所繪示的雙感測器攝像系統的亮度匹配校準方法的流程圖。請同時參照圖3及圖6，本實施例的方法適用於上述的雙感測器攝像系統30，以下即搭配雙感測器攝像系統30的各項元件說明本實施例的亮度匹配校準方法的詳細步驟。On the other hand, for the calibration of the above brightness matching, FIG. 6 is a flowchart of a brightness matching calibration method of a dual-sensor camera system according to an embodiment of the present invention. Please refer to FIG. 3 and FIG. 6 at the same time, the method of this embodiment is applicable to the above-mentioned dual-sensor camera system 30 , and the following describes the brightness matching calibration method of this embodiment in combination with various elements of the dual-sensor camera system 30 . detailed steps.

在步驟S602中，由處理器38控制色彩感測器32及紅外線感測器34採用多個拍攝條件分別擷取一攝像場景的多張色彩影像及多張紅外線影像。所述的拍攝條件例如包括環境光（ambient light）的波長範圍、亮度，與攝像場景中的主體、背景的距離其中之一或其組合，在此不設限。In step S602, the color sensor 32 and the infrared sensor 34 are controlled by the processor 38 to capture a plurality of color images and a plurality of infrared images of a shooting scene using a plurality of shooting conditions, respectively. The shooting conditions include, for example, the wavelength range and brightness of ambient light, one of the distances from the subject and the background in the shooting scene, or a combination thereof, which is not limited herein.

在步驟S604中，由處理器38計算在各拍攝條件下擷取的色彩影像的多個色彩影像參數以及在各拍攝條件下擷取的紅外線影像的多個紅外線影像參數，並用以計算色彩影像的亮度及紅外線影像的亮度之間的差異。In step S604, the processor 38 calculates a plurality of color image parameters of the color image captured under each shooting condition and a plurality of infrared image parameters of the infrared image captured under each shooting condition, and used to calculate the color image parameters. The difference between brightness and the brightness of an infrared image.

在一些實施例中，對於在不同拍攝條件下拍攝的影像，處理器38例如會計算每張影像的3A（包括自動對焦（Auto Focus，AF）、自動曝光（Auto Exposure，AE）、自動白平衡（Auto White Balance，AWB））統計值，並用以計算所述影像亮度之間的差異（例如為差值或比值）。In some embodiments, for images captured under different shooting conditions, the processor 38 may, for example, calculate 3A (including Auto Focus (AF), Auto Exposure (AE), Auto White Balance) of each image (Auto White Balance, AWB)) statistics, and used to calculate the difference (eg difference or ratio) between the image brightness.

在一些實施例中，處理器38例如會將各張色彩影像及各張紅外線影像切割為多個區塊（block），並計算各個區塊內所有像素的像素值平均，從而計算相對應區塊的像素值平均的差異，用以作為色彩影像的亮度及紅外線影像的亮度之間的差異。In some embodiments, the processor 38, for example, divides each color image and each infrared image into a plurality of blocks, and calculates the average of the pixel values of all pixels in each block, so as to calculate the corresponding block The average difference of the pixel values of , which is used as the difference between the brightness of the color image and the brightness of the infrared image.

在一些實施例中，處理器38例如會計算各張色彩影像及各張紅外線影像的影像直方圖（histogram），從而計算色彩影像及紅外線影像的影像直方圖的差異，用以作為色彩影像的亮度及紅外線影像的亮度之間的差異。In some embodiments, the processor 38, for example, calculates an image histogram of each color image and each infrared image, so as to calculate the difference between the image histograms of the color image and the infrared image, which is used as the brightness of the color image and the difference between the brightness of the infrared image.

回到圖6的流程，在步驟S606中，由處理器38根據所計算的差異，決定適於色彩感測器32及紅外線感測器34的曝光設定。Returning to the flowchart of FIG. 6 , in step S606 , the processor 38 determines the exposure settings suitable for the color sensor 32 and the infrared sensor 34 according to the calculated difference.

在一些實施例中，為了達到畫面同步，處理器38例如會控制色彩感測器32及紅外線感測器34採用相同的曝光時間分別擷取攝像場景的色彩影像及紅外線影像，並計算色彩影像及紅外線影像之間的亮度差異，從而計算用以調整色彩影像的亮度及/或紅外線影像的亮度的增益（gain）。即，處理器38會計算可補償色彩影像及紅外線影像的亮度差異的增益，其可以是針對色彩影像的增益、針對於紅外線影像的增益，或是針對上述兩者的增益，在此不設限。In some embodiments, in order to achieve frame synchronization, the processor 38 controls, for example, the color sensor 32 and the infrared sensor 34 to capture the color image and the infrared image of the camera scene with the same exposure time, and calculate the color image and the infrared image respectively. The difference in brightness between the infrared images to calculate the gain used to adjust the brightness of the color image and/or the brightness of the infrared image. That is, the processor 38 calculates a gain that can compensate for the difference in brightness between the color image and the infrared image, which can be the gain for the color image, the gain for the infrared image, or the gain for both, which is not limited here. .

舉例來說，若採用相同曝光時間擷取的色彩影像較亮，則可計算用以調整紅外線影像的亮度的增益，使得紅外線影像的亮度乘上該增益後，與色彩影像的亮度相當。所計算的增益例如會連同其對應的拍攝條件儲存於儲存裝置36中，從而在後續的執行階段中，每當擷取色彩影像和紅外線影像時，處理器38即可藉由識別拍攝條件，從儲存裝置36中取得應對於該拍攝條件的增益，並將所擷取的色彩影像或紅外線影像的像素值乘上所取得的增益，從而使得色彩影像的亮度能夠與紅外線影像的亮度匹配。For example, if the color image captured with the same exposure time is brighter, a gain for adjusting the brightness of the infrared image can be calculated so that the brightness of the infrared image is equal to the brightness of the color image after multiplying the gain by the gain. For example, the calculated gain will be stored in the storage device 36 together with its corresponding shooting conditions, so that in the subsequent execution stage, whenever the color image and the infrared image are captured, the processor 38 can identify the shooting conditions, from the The gain corresponding to the shooting condition is obtained in the storage device 36, and the pixel value of the captured color image or infrared image is multiplied by the obtained gain, so that the brightness of the color image can match the brightness of the infrared image.

在一些實施例中，雙感測器攝像系統30中可額外配置一個紅外線投射器（IR projector），從而搭配紅外線感測器32來輔助處理器38計算雙感測器攝像系統30與攝像場景中的主體和背景之間的距離。In some embodiments, the dual-sensor camera system 30 may be additionally configured with an IR projector, so as to cooperate with the infrared sensor 32 to assist the processor 38 in calculating the distance between the dual-sensor camera system 30 and the camera scene. the distance between the subject and the background.

詳言之，圖7是依照本發明一實施例所繪示的雙感測器攝像系統的校準方法的流程圖。請同時參照圖3及圖7，本實施例的方法適用於上述的雙感測器攝像系統30，以下即搭配雙感測器攝像系統30的各項元件說明本實施例的校準方法的詳細步驟。In detail, FIG. 7 is a flowchart of a calibration method of a dual-sensor camera system according to an embodiment of the present invention. Please refer to FIG. 3 and FIG. 7 at the same time. The method of this embodiment is applicable to the above-mentioned dual-sensor camera system 30 . The following describes the detailed steps of the calibration method of this embodiment in combination with various elements of the dual-sensor camera system 30 . .

在步驟S702中，由處理器38控制紅外線投射器投射具有特殊圖案的不可見光至攝像場景。In step S702, the processor 38 controls the infrared projector to project invisible light with a special pattern to the imaging scene.

在步驟S704中，由處理器38控制紅外線感測器32中的兩個紅外線感測器分別擷取具有特殊圖案的攝像場景的多張紅外線影像。In step S704, the processor 38 controls the two infrared sensors in the infrared sensor 32 to capture a plurality of infrared images of the shooting scene with a special pattern, respectively.

在步驟S706中，由處理器38根據所擷取的紅外線影像中的特殊圖案以及兩個紅外線感測器之間的視差（parallax），分別計算雙感測器攝像系統30與攝像場景中的主體和背景之間的距離。其中，由於由紅外線投射器投射到攝像場景的特殊圖案不易受到環境的影響，因此藉由上述方法可取得較精確的拍攝主體及/或背景的距離，並用以識別拍攝條件來作為後續進行影像補償的依據。In step S706, the processor 38 calculates the subject in the dual-sensor camera system 30 and the camera scene respectively according to the special pattern in the captured infrared image and the parallax between the two infrared sensors distance from the background. Among them, since the special pattern projected by the infrared projector to the shooting scene is not easily affected by the environment, the above method can obtain a more accurate distance between the subject and/or the background, and use it to identify the shooting conditions for subsequent image compensation. basis.

綜上所述，本發明的雙感測器攝像系統及其校準方法針對配置於雙感測器攝像系統上的色彩感測器與紅外線感測器，採用不同的拍攝條件分別擷取多張影像，從而對色彩感測器與紅外線感測器進行影像對位及亮度匹配的校準，並將校準結果用以作為對後續擷取影像進行調整的依據。藉此，可提高雙感測器攝像系統所擷取影像的影像品質。To sum up, the dual-sensor camera system and the calibration method thereof of the present invention capture a plurality of images respectively using different shooting conditions for the color sensor and the infrared sensor configured on the dual-sensor camera system , so as to calibrate the image alignment and brightness matching of the color sensor and the infrared sensor, and use the calibration result as a basis for adjusting the subsequent captured images. Thereby, the image quality of the image captured by the dual-sensor camera system can be improved.

10、20:影像感測器 12:色彩資訊 14:亮度資訊 16:影像 22:色彩感測器 22a:色彩影像 24:紅外線感測器 24a:紅外線影像 26:場景影像 30:雙感測器攝像系統 32:色彩感測器 34:紅外線感測器 36:儲存裝置 38:處理器 R、G、B、I:像素 S402~S406、S502~S506、S602~S606、S702~S706:步驟10, 20: Image sensor 12: Color Information 14: Brightness information 16: Video 22: Color sensor 22a: Color Image 24: Infrared sensor 24a: Infrared Imaging 26: Scene image 30: Dual-sensor camera system 32: Color Sensor 34: Infrared sensor 36: Storage Device 38: Processor R, G, B, I: pixels S402~S406, S502~S506, S602~S606, S702~S706: Steps

圖1是習知使用影像感測器擷取影像的示意圖。 圖2是依照本發明一實施例所繪示的使用影像感測器擷取影像的示意圖。 圖3是依照本發明一實施例所繪示的雙感測器攝像系統的方塊圖。 圖4是依照本發明一實施例所繪示的雙感測器攝像系統的校準方法的流程圖。 圖5是依照本發明一實施例所繪示的雙感測器攝像系統的對位校準方法的流程圖。 圖6是依照本發明一實施例所繪示的雙感測器攝像系統的亮度匹配校準方法的流程圖。 圖7是依照本發明一實施例所繪示的雙感測器攝像系統的校準方法的流程圖。FIG. 1 is a schematic diagram of conventionally using an image sensor to capture images. FIG. 2 is a schematic diagram of capturing an image using an image sensor according to an embodiment of the present invention. FIG. 3 is a block diagram of a dual-sensor camera system according to an embodiment of the present invention. FIG. 4 is a flowchart of a calibration method of a dual-sensor camera system according to an embodiment of the present invention. FIG. 5 is a flowchart of a method for alignment and calibration of a dual-sensor camera system according to an embodiment of the present invention. FIG. 6 is a flowchart of a brightness matching calibration method of a dual-sensor camera system according to an embodiment of the present invention. FIG. 7 is a flowchart of a calibration method of a dual-sensor camera system according to an embodiment of the present invention.

30:雙感測器攝像系統30: Dual-sensor camera system

32:色彩感測器32: Color Sensor

34:紅外線感測器34: Infrared sensor

36:儲存裝置36: Storage Device

38:處理器38: Processor

Claims (20)

一種雙感測器攝像系統，包括： 至少一色彩感測器； 至少一紅外線感測器； 儲存裝置，儲存電腦程式；以及 處理器，耦接所述至少一色彩感測器、所述至少一紅外光感測器及所述儲存裝置，經配置以載入並執行所述電腦程式以： 控制所述至少一色彩感測器及所述至少一紅外線感測器採用多個拍攝條件分別擷取一攝像場景的多張色彩影像及多張紅外線影像； 計算在各所述拍攝條件下擷取的所述色彩影像的多個色彩影像參數以及在各所述拍攝條件下擷取的所述紅外線影像的多個紅外線影像參數，用以計算所述色彩影像的亮度及所述紅外線影像的亮度之間的差異；以及 根據所計算的差異，決定適於所述至少一色彩感測器及所述至少一紅外線感測器的曝光設定。A dual-sensor camera system comprising: at least one color sensor; at least one infrared sensor; storage devices to store computer programs; and A processor, coupled to the at least one color sensor, the at least one infrared light sensor, and the storage device, is configured to load and execute the computer program to: controlling the at least one color sensor and the at least one infrared sensor to capture a plurality of color images and a plurality of infrared images of a camera scene respectively using a plurality of shooting conditions; calculating a plurality of color image parameters of the color image captured under each of the shooting conditions and a plurality of infrared image parameters of the infrared image captured under each of the shooting conditions, so as to calculate the color image the difference between the brightness of the infrared image and the brightness of the infrared image; and According to the calculated differences, exposure settings suitable for the at least one color sensor and the at least one infrared sensor are determined. 如請求項1所述的雙感測器攝像系統，其中所述處理器包括： 將各所述色彩影像及各所述紅外線影像切割為多個區塊（block），並計算各所述區塊內所有像素的像素值平均；以及 計算相對應的所述區塊的所述像素值平均的差異作為所述色彩影像的亮度及所述紅外線影像的亮度之間的差異。The dual-sensor camera system of claim 1, wherein the processor comprises: dividing each of the color images and each of the infrared images into a plurality of blocks, and calculating an average of pixel values of all pixels in each of the blocks; and The average difference of the pixel values of the corresponding blocks is calculated as the difference between the brightness of the color image and the brightness of the infrared image. 如請求項1所述的雙感測器攝像系統，其中所述處理器包括： 計算各所述色彩影像及各所述紅外線影像的影像直方圖（histogram）；以及 計算所述色彩影像及所述紅外線影像的所述影像直方圖的差異作為所述色彩影像的亮度及所述紅外線影像的亮度之間的差異。The dual-sensor camera system of claim 1, wherein the processor comprises: calculating an image histogram for each of the color images and each of the infrared images; and The difference between the image histograms of the color image and the infrared image is calculated as the difference between the brightness of the color image and the brightness of the infrared image. 如請求項1所述的雙感測器攝像系統，其中所述處理器包括： 控制所述至少一色彩感測器及所述至少一紅外線感測器採用相同的曝光時間分別擷取所述攝像場景的所述色彩影像及所述紅外線影像；以及 根據所計算的所述色彩影像的亮度及所述紅外線影像的亮度之間的差異，計算用以調整所述色彩影像的亮度或所述紅外線影像的亮度的增益（gain）。The dual-sensor camera system of claim 1, wherein the processor comprises: controlling the at least one color sensor and the at least one infrared sensor to capture the color image and the infrared image of the camera scene with the same exposure time, respectively; and According to the difference between the calculated brightness of the color image and the brightness of the infrared image, a gain for adjusting the brightness of the color image or the brightness of the infrared image is calculated. 如請求項1所述的雙感測器攝像系統，其中所述處理器更包括： 控制所述至少一色彩感測器及所述至少一紅外線感測器分別拍攝具有特殊圖案的測試圖（test chart），以獲得一色彩測試影像及一紅外線測試影像； 偵測所述色彩測試影像及所述紅外線測試影像中的所述特殊圖案的多個特徵點；以及 執行影像對位演算法以根據所述色彩測試影像及所述紅外線測試影像中相對應的所述特徵點之間的位置關係，計算所述色彩測試影像及所述紅外線測試影像之間的匹配關係，用以對後續擷取的所述色彩影像及所述紅外線影像進行對位。The dual-sensor camera system of claim 1, wherein the processor further comprises: controlling the at least one color sensor and the at least one infrared sensor to capture a test chart with a special pattern, respectively, to obtain a color test image and an infrared test image; detecting a plurality of feature points of the special pattern in the color test image and the infrared test image; and Execute an image alignment algorithm to calculate the matching relationship between the color test image and the infrared test image according to the positional relationship between the corresponding feature points in the color test image and the infrared test image , which is used to align the color image and the infrared image captured subsequently. 如請求項5所述的雙感測器攝像系統，其中所述處理器包括： 將所述色彩測試影像及所述紅外線測試影像切割為多個區塊，並執行一特徵偵測演算法以偵測各所述區塊內的至少一個所述特徵點，其中所述特徵偵測演算法包括哈里斯邊角偵測法（Harris corner detection）。The dual-sensor camera system of claim 5, wherein the processor comprises: Divide the color test image and the infrared test image into a plurality of blocks, and execute a feature detection algorithm to detect at least one of the feature points in each of the blocks, wherein the feature detection Algorithms include Harris corner detection. 如請求項5所述的雙感測器攝像系統，其中所述處理器包括： 針對從所述色彩測試影像中偵測出的所述特徵點中的一指定特徵點，以所述紅外線測試影像中對應於所述指定特徵點的像素為中心移動包括多個像素的補丁（patch）來搜尋所述紅外線測試影像中對應於所述指定特徵點的對應特徵點；以及 執行隨機抽樣一致（RANdom SAmple Consensus，RANSAC）演算法以建立單應性變換矩陣（homography transformation matrix），並將所述色彩測試影像中的所述特徵點與所述紅外線測試影像中的所述對應特徵點的位置帶入所述單應性變換矩陣求解，而以所求得的解作為所述色彩測試影像及所述紅外線測試影像之間的所述匹配關係。The dual-sensor camera system of claim 5, wherein the processor comprises: For a specified feature point among the feature points detected from the color test image, a patch including a plurality of pixels is moved around the pixel corresponding to the specified feature point in the infrared test image ) to search for corresponding feature points in the infrared test image corresponding to the specified feature points; and Execute a random sampling consensus (RANdom SAmple Consensus, RANSAC) algorithm to establish a homography transformation matrix, and associate the feature points in the color test image with the corresponding ones in the infrared test image The position of the feature point is brought into the homography transformation matrix to solve, and the obtained solution is used as the matching relationship between the color test image and the infrared test image. 如請求項5所述的雙感測器攝像系統，其中所述處理器包括： 利用所述色彩測試影像及所述紅外線測試影像計算攝像場景的多個深度，據以將所述攝像場景區分為多個深度場景；以及 針對各所述深度場景建立二次方程式（quadratic equation），並將位於各所述深度場景內的所述色彩測試影像中的所述特徵點與所述紅外線測試影像中的所述對應特徵點的位置帶入對應的所述二次方程式求解，而以所求得的解作為所述色彩測試影像及所述紅外線測試影像之間的所述匹配關係。The dual-sensor camera system of claim 5, wherein the processor comprises: Using the color test image and the infrared test image to calculate a plurality of depths of the imaging scene, so as to distinguish the imaging scene into a plurality of depth scenes; and A quadratic equation is established for each of the depth scenes, and the difference between the feature points in the color test image and the corresponding feature points in the infrared test image located in each of the depth scenes The position is brought into the corresponding quadratic equation to solve, and the obtained solution is used as the matching relationship between the color test image and the infrared test image. 如請求項5所述的雙感測器攝像系統，其中所述影像對位演算法包括暴力法（bruteforce）、光流法（optical flow）、單應性變換法（homography）或局部翹曲法（local warping）。The dual-sensor camera system of claim 5, wherein the image alignment algorithm includes bruteforce, optical flow, homography, or local warping (local warping). 如請求項1所述的雙感測器攝像系統，更包括紅外線投射器（IR projector），其中所述處理器更包括： 控制所述紅外線投射器投射具有特殊圖案的不可見光至所述攝像場景； 控制所述至少一紅外線感測器中的兩個紅外線感測器分別擷取具有所述特殊圖案的攝像場景的多張紅外線影像；以及 根據所擷取的所述紅外線影像中的所述特殊圖案以及所述兩個紅外線感測器的視差（parallax），分別計算所述雙感測器攝像系統與所述攝像場景中的主體和背景之間的距離。The dual-sensor camera system as claimed in claim 1, further comprising an IR projector, wherein the processor further comprises: controlling the infrared projector to project invisible light with a special pattern to the camera scene; controlling two infrared sensors in the at least one infrared sensor to capture a plurality of infrared images of the camera scene with the special pattern; and According to the special pattern in the captured infrared image and the parallax of the two infrared sensors, the subject and the background in the dual-sensor camera system and the camera scene are calculated respectively the distance between. 一種雙感測器攝像系統的校準方法，所述雙感測器攝像系統包括至少一色彩感測器、至少一紅外線感測器及處理器，所述方法包括下列步驟： 控制所述至少一色彩感測器及所述至少一紅外線感測器採用多個拍攝條件分別擷取一攝像場景的多張色彩影像及多張紅外線影像； 計算在各所述拍攝條件下擷取的所述色彩影像的多個色彩影像參數以及在各所述拍攝條件下擷取的所述紅外線影像的多個紅外線影像參數，用以計算所述色彩影像的亮度及所述紅外線影像的亮度之間的差異；以及 根據所計算的差異，決定適於所述至少一色彩感測器及所述至少一紅外線感測器的曝光設定。A calibration method for a dual-sensor camera system, the dual-sensor camera system includes at least one color sensor, at least one infrared sensor, and a processor, and the method includes the following steps: controlling the at least one color sensor and the at least one infrared sensor to capture a plurality of color images and a plurality of infrared images of a camera scene respectively using a plurality of shooting conditions; calculating a plurality of color image parameters of the color image captured under each of the shooting conditions and a plurality of infrared image parameters of the infrared image captured under each of the shooting conditions, so as to calculate the color image the difference between the brightness of the infrared image and the brightness of the infrared image; and According to the calculated differences, exposure settings suitable for the at least one color sensor and the at least one infrared sensor are determined. 如請求項11所述的方法，其中計算所述色彩影像的亮度及所述紅外線影像的亮度之間的差異的步驟包括： 將各所述色彩影像及各所述紅外線影像切割為多個區塊，並計算各所述區塊內所有像素的像素值平均；以及 計算相對應的所述區塊的所述像素值平均的差異作為所述色彩影像的亮度及所述紅外線影像的亮度之間的差異。The method of claim 11, wherein the step of calculating the difference between the brightness of the color image and the brightness of the infrared image comprises: dividing each of the color images and each of the infrared images into a plurality of blocks, and calculating an average of pixel values of all pixels in each of the blocks; and The average difference of the pixel values of the corresponding blocks is calculated as the difference between the brightness of the color image and the brightness of the infrared image. 如請求項11所述的方法，其中計算所述色彩影像的亮度及所述紅外線影像的亮度之間的差異的步驟包括： 計算各所述色彩影像及各所述紅外線影像的影像直方圖；以及 計算所述色彩影像及所述紅外線影像的所述影像直方圖的差異作為所述色彩影像的亮度及所述紅外線影像的亮度之間的差異。The method of claim 11, wherein the step of calculating the difference between the brightness of the color image and the brightness of the infrared image comprises: calculating an image histogram for each of the color images and each of the infrared images; and The difference between the image histograms of the color image and the infrared image is calculated as the difference between the brightness of the color image and the brightness of the infrared image. 如請求項11所述的方法，更包括： 控制所述至少一色彩感測器及所述至少一紅外線感測器採用相同的曝光時間分別擷取所述攝像場景的所述色彩影像及所述紅外線影像；以及 根據所計算的所述色彩影像的亮度及所述紅外線影像的亮度之間的差異，計算用以調整所述色彩影像的亮度或所述紅外線影像的亮度的增益。The method according to claim 11, further comprising: controlling the at least one color sensor and the at least one infrared sensor to capture the color image and the infrared image of the camera scene with the same exposure time, respectively; and A gain for adjusting the brightness of the color image or the brightness of the infrared image is calculated according to the difference between the calculated brightness of the color image and the brightness of the infrared image. 如請求項11所述的方法，更包括： 控制所述至少一色彩感測器及所述至少一紅外線感測器分別拍攝具有特殊圖案的測試圖，以獲得一色彩測試影像及一紅外線測試影像； 偵測所述色彩測試影像及所述紅外線測試影像中的所述特殊圖案的多個特徵點；以及 執行影像對位演算法以根據所述色彩測試影像及所述紅外線測試影像中相對應的所述特徵點之間的位置關係，計算所述色彩測試影像及所述紅外線測試影像之間的匹配關係，用以對後續擷取的所述色彩影像及所述紅外線影像進行對位。The method according to claim 11, further comprising: controlling the at least one color sensor and the at least one infrared sensor to capture a test pattern with a special pattern, respectively, to obtain a color test image and an infrared test image; detecting a plurality of feature points of the special pattern in the color test image and the infrared test image; and Execute an image alignment algorithm to calculate the matching relationship between the color test image and the infrared test image according to the positional relationship between the corresponding feature points in the color test image and the infrared test image , which is used to align the color image and the infrared image captured subsequently. 如請求項15所述的方法，其中偵測所述色彩測試影像及所述紅外線測試影像中的所述特殊圖案的多個特徵點的步驟包括： 將所述色彩測試影像及所述紅外線測試影像切割為多個區塊，並執行一特徵偵測演算法以偵測各所述區塊內的至少一個所述特徵點，其中所述特徵偵測演算法包括哈里斯邊角偵測法。The method of claim 15, wherein the step of detecting a plurality of feature points of the special pattern in the color test image and the infrared test image comprises: Divide the color test image and the infrared test image into a plurality of blocks, and execute a feature detection algorithm to detect at least one of the feature points in each of the blocks, wherein the feature detection Algorithms include Harris corner detection. 如請求項15所述的方法，其中計算所述色彩測試影像及所述紅外線測試影像之間的匹配關係的步驟包括： 針對從所述色彩測試影像中偵測出的所述特徵點中的一指定特徵點，以所述紅外線測試影像中對應於所述指定特徵點的像素為中心移動包括多個像素的補丁來搜尋所述紅外線測試影像中對應於所述指定特徵點的對應特徵點；以及 執行隨機抽樣一致演算法以建立單應性變換矩陣，並將所述色彩測試影像中的所述特徵點與所述紅外線測試影像中的所述對應特徵點的位置帶入所述單應性變換矩陣求解，而以所求得的解作為所述色彩測試影像及所述紅外線測試影像之間的所述匹配關係。The method of claim 15, wherein the step of calculating the matching relationship between the color test image and the infrared test image comprises: For a specified feature point among the feature points detected from the color test image, a patch including a plurality of pixels is moved around the pixel corresponding to the specified feature point in the infrared test image to search for Corresponding feature points in the infrared test image corresponding to the specified feature points; and A random sampling consensus algorithm is performed to create a homography transformation matrix, and the positions of the feature points in the color test image and the corresponding feature points in the infrared test image are brought into the homography transformation The matrix is solved, and the obtained solution is used as the matching relationship between the color test image and the infrared test image. 如請求項15所述的方法，其中計算所述色彩測試影像及所述紅外線測試影像之間的匹配關係的步驟包括： 利用所述色彩測試影像及所述紅外線測試影像計算攝像場景的多個深度，據以將所述攝像場景區分為多個深度場景；以及 針對各所述深度場景建立二次方程式，並將位於各所述深度場景內的所述色彩測試影像中的所述特徵點與所述紅外線測試影像中的所述對應特徵點的位置帶入對應的所述二次方程式求解，而以所求得的解作為所述色彩測試影像及所述紅外線測試影像之間的所述匹配關係。The method of claim 15, wherein the step of calculating the matching relationship between the color test image and the infrared test image comprises: Using the color test image and the infrared test image to calculate a plurality of depths of the imaging scene, so as to distinguish the imaging scene into a plurality of depth scenes; and A quadratic equation is established for each of the depth scenes, and the positions of the feature points in the color test images in each of the depth scenes and the corresponding feature points in the infrared test images are brought into correspondence to solve the quadratic equation of , and use the obtained solution as the matching relationship between the color test image and the infrared test image. 如請求項15所述的方法，其中所述影像對位演算法包括暴力法、光流法、單應性變換法或局部翹曲法。The method of claim 15, wherein the image alignment algorithm comprises a brute force method, an optical flow method, a homography transformation method or a local warping method. 如請求項11所述的方法，其中所述雙感測器攝像系統更包括紅外線投射器，所述方法更包括： 控制所述紅外線投射器投射具有特殊圖案的不可見光至所述攝像場景； 控制所述至少一紅外線感測器中的兩個紅外線感測器分別擷取具有所述特殊圖案的攝像場景的多張紅外線影像；以及 根據所擷取的所述紅外線影像中的所述特殊圖案以及所述兩個紅外線感測器的視差，分別計算所述方法與所述攝像場景中的主體和背景之間的距離。The method of claim 11, wherein the dual-sensor camera system further comprises an infrared projector, the method further comprising: controlling the infrared projector to project invisible light with a special pattern to the camera scene; controlling two infrared sensors in the at least one infrared sensor to capture a plurality of infrared images of the camera scene with the special pattern; and According to the special pattern in the captured infrared image and the parallax of the two infrared sensors, the distances between the method and the subject and the background in the imaging scene are calculated respectively.
TW109146764A 2020-09-04 2020-12-30 Dual sensor imaging system and calibration method thereof TWI764484B (en)

Priority Applications (1)

Application Number Priority Date Filing Date Title
US17/191,701 US11418719B2 (en) 2020-09-04 2021-03-04 Dual sensor imaging system and calibration method which includes a color sensor and an infrared ray sensor to perform image alignment and brightness matching

Applications Claiming Priority (2)

Application Number Priority Date Filing Date Title
US202063074477P 2020-09-04 2020-09-04
US63/074,477 2020-09-04

Publications (2)

Publication Number Publication Date
TW202211160A true TW202211160A (en) 2022-03-16
TWI764484B TWI764484B (en) 2022-05-11

Family

ID=80438521

Family Applications (5)

Application Number Title Priority Date Filing Date
TW109145632A TWI767468B (en) 2020-09-04 2020-12-23 Dual sensor imaging system and imaging method thereof
TW109145614A TWI778476B (en) 2020-09-04 2020-12-23 Dual sensor imaging system and imaging method thereof
TW109146764A TWI764484B (en) 2020-09-04 2020-12-30 Dual sensor imaging system and calibration method thereof
TW109146831A TWI797528B (en) 2020-09-04 2020-12-30 Dual sensor imaging system and privacy protection imaging method thereof
TW109146922A TWI767484B (en) 2020-09-04 2020-12-30 Dual sensor imaging system and depth map calculation method thereof

Family Applications Before (2)

Application Number Title Priority Date Filing Date
TW109145632A TWI767468B (en) 2020-09-04 2020-12-23 Dual sensor imaging system and imaging method thereof
TW109145614A TWI778476B (en) 2020-09-04 2020-12-23 Dual sensor imaging system and imaging method thereof

Family Applications After (2)

Application Number Title Priority Date Filing Date
TW109146831A TWI797528B (en) 2020-09-04 2020-12-30 Dual sensor imaging system and privacy protection imaging method thereof
TW109146922A TWI767484B (en) 2020-09-04 2020-12-30 Dual sensor imaging system and depth map calculation method thereof

Country Status (2)

Country Link
CN (5) CN114143418B (en)
TW (5) TWI767468B (en)

Families Citing this family (1)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN116091341B (en) * 2022-12-15 2024-04-02 南京信息工程大学 Exposure difference enhancement method and device for low-light image

Family Cites Families (70)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
JP2004246252A (en) * 2003-02-17 2004-09-02 Takenaka Komuten Co Ltd Apparatus and method for collecting image information
JP2005091434A (en) * 2003-09-12 2005-04-07 Noritsu Koki Co Ltd Position adjusting method and image reader with damage compensation function using the same
JP4244018B2 (en) * 2004-03-25 2009-03-25 ノーリツ鋼機株式会社 Defective pixel correction method, program, and defective pixel correction system for implementing the method
JP4341680B2 (en) * 2007-01-22 2009-10-07 セイコーエプソン株式会社 projector
US9307212B2 (en) * 2007-03-05 2016-04-05 Fotonation Limited Tone mapping for low-light video frame enhancement
US8866920B2 (en) * 2008-05-20 2014-10-21 Pelican Imaging Corporation Capturing and processing of images using monolithic camera array with heterogeneous imagers
EP3328048B1 (en) * 2008-05-20 2021-04-21 FotoNation Limited Capturing and processing of images using monolithic camera array with heterogeneous imagers
CN101404060B (en) * 2008-11-10 2010-06-30 北京航空航天大学 Human face recognition method based on visible light and near-infrared Gabor information amalgamation
US8749635B2 (en) * 2009-06-03 2014-06-10 Flir Systems, Inc. Infrared camera systems and methods for dual sensor applications
US8681216B2 (en) * 2009-03-12 2014-03-25 Hewlett-Packard Development Company, L.P. Depth-sensing camera system
US20120154596A1 (en) * 2009-08-25 2012-06-21 Andrew Augustine Wajs Reducing noise in a color image
US8478123B2 (en) * 2011-01-25 2013-07-02 Aptina Imaging Corporation Imaging devices having arrays of image sensors and lenses with multiple aperture sizes
JP2013115679A (en) * 2011-11-30 2013-06-10 Fujitsu General Ltd Imaging apparatus
US10848731B2 (en) * 2012-02-24 2020-11-24 Matterport, Inc. Capturing and aligning panoramic image and depth data
TW201401186A (en) * 2012-06-25 2014-01-01 Psp Security Co Ltd System and method for identifying human face
CN104662897A (en) * 2012-09-25 2015-05-27 日本电信电话株式会社 Image encoding method, image decoding method, image encoding device, image decoding device, image encoding program, image decoding program, and recording medium
KR102086509B1 (en) * 2012-11-23 2020-03-09 엘지전자 주식회사 Apparatus and method for obtaining 3d image
CN105009568B (en) * 2012-12-21 2018-12-18 菲力尔系统公司 For handling the system of visible spectrum image and infrared image, method and the readable medium of non-volatile machine
TWM458748U (en) * 2012-12-26 2013-08-01 Chunghwa Telecom Co Ltd Image type depth information retrieval device
JP6055681B2 (en) * 2013-01-10 2016-12-27 株式会社 日立産業制御ソリューションズ Imaging device
CN104661008B (en) * 2013-11-18 2017-10-31 深圳中兴力维技术有限公司 The treating method and apparatus that color image quality is lifted under low light conditions
CN104021548A (en) * 2014-05-16 2014-09-03 中国科学院西安光学精密机械研究所 Method for acquiring 4D scene information
US9516295B2 (en) * 2014-06-30 2016-12-06 Aquifi, Inc. Systems and methods for multi-channel imaging based on multiple exposure settings
JP6450107B2 (en) * 2014-08-05 2019-01-09 キヤノン株式会社 Image processing apparatus, image processing method, program, and storage medium
JP6597636B2 (en) * 2014-12-10 2019-10-30 ソニー株式会社 Imaging apparatus, imaging method, program, and image processing apparatus
WO2016158196A1 (en) * 2015-03-31 2016-10-06 富士フイルム株式会社 Image pickup device and image processing method and program for image processing device
WO2016192437A1 (en) * 2015-06-05 2016-12-08 深圳奥比中光科技有限公司 3d image capturing apparatus and capturing method, and 3d image system
JP2017011634A (en) * 2015-06-26 2017-01-12 キヤノン株式会社 Imaging device, control method for the same and program
CN105049829B (en) * 2015-07-10 2018-12-25 上海图漾信息科技有限公司 Optical filter, imaging sensor, imaging device and 3-D imaging system
CN105069768B (en) * 2015-08-05 2017-12-29 武汉高德红外股份有限公司 A kind of visible images and infrared image fusion processing system and fusion method
US10523855B2 (en) * 2015-09-24 2019-12-31 Intel Corporation Infrared and visible light dual sensor imaging system
TW201721269A (en) * 2015-12-11 2017-06-16 宏碁股份有限公司 Automatic exposure system and auto exposure method thereof
JP2017112401A (en) * 2015-12-14 2017-06-22 ソニー株式会社 Imaging device, apparatus and method for image processing, and program
CN206117865U (en) * 2016-01-16 2017-04-19 上海图漾信息科技有限公司 Range data monitoring device
JP2017163297A (en) * 2016-03-09 2017-09-14 キヤノン株式会社 Imaging apparatus
KR101747603B1 (en) * 2016-05-11 2017-06-16 재단법인 다차원 스마트 아이티 융합시스템 연구단 Color night vision system and operation method thereof
CN106815826A (en) * 2016-12-27 2017-06-09 上海交通大学 Night vision image Color Fusion based on scene Recognition
CN108280807A (en) * 2017-01-05 2018-07-13 浙江舜宇智能光学技术有限公司 Monocular depth image collecting device and system and its image processing method
CN110462686B (en) * 2017-02-06 2023-08-11 弗托斯传感与算法公司 Apparatus and method for obtaining depth information from a scene
CN108419062B (en) * 2017-02-10 2020-10-02 杭州海康威视数字技术股份有限公司 Image fusion apparatus and image fusion method
CN109474770B (en) * 2017-09-07 2021-09-14 华为技术有限公司 Imaging device and imaging method
CN109712102B (en) * 2017-10-25 2020-11-27 杭州海康威视数字技术股份有限公司 Image fusion method and device and image acquisition equipment
CN107846537B (en) * 2017-11-08 2019-11-26 维沃移动通信有限公司 A kind of CCD camera assembly, image acquiring method and mobile terminal
CN109951646B (en) * 2017-12-20 2021-01-15 杭州海康威视数字技术股份有限公司 Image fusion method and device, electronic equipment and computer readable storage medium
US10748247B2 (en) * 2017-12-26 2020-08-18 Facebook, Inc. Computing high-resolution depth images using machine learning techniques
US10757320B2 (en) * 2017-12-28 2020-08-25 Waymo Llc Multiple operating modes to expand dynamic range
TWI661726B (en) * 2018-01-09 2019-06-01 呂官諭 Image sensor with enhanced image recognition and application
CN110136183B (en) * 2018-02-09 2021-05-18 华为技术有限公司 Image processing method and device and camera device
CN108965654B (en) * 2018-02-11 2020-12-25 浙江宇视科技有限公司 Double-spectrum camera system based on single sensor and image processing method
CN110572583A (en) * 2018-05-18 2019-12-13 杭州海康威视数字技术股份有限公司 method for shooting image and camera
CN108961195B (en) * 2018-06-06 2021-03-23 Oppo广东移动通信有限公司 Image processing method and device, image acquisition device, readable storage medium and computer equipment
JP6574878B2 (en) * 2018-07-19 2019-09-11 キヤノン株式会社 Image processing apparatus, image processing method, imaging apparatus, program, and storage medium
JP7254461B2 (en) * 2018-08-01 2023-04-10 キヤノン株式会社 IMAGING DEVICE, CONTROL METHOD, RECORDING MEDIUM, AND INFORMATION PROCESSING DEVICE
CN109035193A (en) * 2018-08-29 2018-12-18 成都臻识科技发展有限公司 A kind of image processing method and imaging processing system based on binocular solid camera
US11689785B2 (en) * 2018-09-14 2023-06-27 Zhejiang Uniview Technologies Co., Ltd. Dual-spectrum image automatic exposure method and apparatus, and dual-spectrum image camera and machine storage medium
JP2020052001A (en) * 2018-09-28 2020-04-02 パナソニックＩｐマネジメント株式会社 Depth acquisition device, depth acquisition method, and program
US11176694B2 (en) * 2018-10-19 2021-11-16 Samsung Electronics Co., Ltd Method and apparatus for active depth sensing and calibration method thereof
CN109636732B (en) * 2018-10-24 2023-06-23 深圳先进技术研究院 Hole repairing method of depth image and image processing device
CN110248105B (en) * 2018-12-10 2020-12-08 浙江大华技术股份有限公司 Image processing method, camera and computer storage medium
US11120536B2 (en) * 2018-12-12 2021-09-14 Samsung Electronics Co., Ltd Apparatus and method for determining image sharpness
CN113170048A (en) * 2019-02-19 2021-07-23 华为技术有限公司 Image processing device and method
US10972649B2 (en) * 2019-02-27 2021-04-06 X Development Llc Infrared and visible imaging system for device identification and tracking
JP7316809B2 (en) * 2019-03-11 2023-07-28 キヤノン株式会社 Image processing device, image processing device control method, system, and program
CN110349117B (en) * 2019-06-28 2023-02-28 重庆工商大学 Infrared image and visible light image fusion method and device and storage medium
CN110706178B (en) * 2019-09-30 2023-01-06 杭州海康威视数字技术股份有限公司 Image fusion device, method, equipment and storage medium
CN111524175A (en) * 2020-04-16 2020-08-11 东莞市东全智能科技有限公司 Depth reconstruction and eye movement tracking method and system for asymmetric multiple cameras
CN111540003A (en) * 2020-04-27 2020-08-14 浙江光珀智能科技有限公司 Depth image generation method and device
CN111586314B (en) * 2020-05-25 2021-09-10 浙江大华技术股份有限公司 Image fusion method and device and computer storage medium
CN111383206B (en) * 2020-06-01 2020-09-29 浙江大华技术股份有限公司 Image processing method and device, electronic equipment and storage medium
IN202021032940A (en) * 2020-07-31 2020-08-28 .Us Priyadarsan

Also Published As

Publication number Publication date
CN114143421A (en) 2022-03-04
TW202211674A (en) 2022-03-16
CN114143420B (en) 2024-05-03
TWI778476B (en) 2022-09-21
CN114143421B (en) 2024-04-05
TWI767468B (en) 2022-06-11
TW202211673A (en) 2022-03-16
TWI797528B (en) 2023-04-01
CN114143443B (en) 2024-04-05
CN114143443A (en) 2022-03-04
TW202211165A (en) 2022-03-16
CN114143418A (en) 2022-03-04
TW202211161A (en) 2022-03-16
CN114143420A (en) 2022-03-04
TWI767484B (en) 2022-06-11
TWI764484B (en) 2022-05-11
CN114143418B (en) 2023-12-01
CN114143419A (en) 2022-03-04
CN114143419B (en) 2023-12-26

Similar Documents

Publication Publication Date Title
KR100983037B1 (en) Method for controlling auto white balance
JP2014044345A (en) Imaging apparatus
JP5848662B2 (en) Image processing apparatus and control method thereof
US10764550B2 (en) Image processing apparatus, image processing method, and storage medium
US11838648B2 (en) Image processing device, imaging apparatus, image processing method, and program for determining a condition for high dynamic range processing
TWI764484B (en) Dual sensor imaging system and calibration method thereof
JP6702752B2 (en) Image processing device, imaging device, control method, and program
US11418719B2 (en) Dual sensor imaging system and calibration method which includes a color sensor and an infrared ray sensor to perform image alignment and brightness matching
US11496660B2 (en) Dual sensor imaging system and depth map calculation method thereof
US11405598B2 (en) Image processing apparatus, image processing method, and storage medium
JP2006319901A (en) Electronic camera and image processing program
JP2012085093A (en) Imaging device and acquisition method
JP2017138927A (en) Image processing device, imaging apparatus, control method and program thereof
JP6272006B2 (en) Imaging apparatus, image processing method, and program
JP6337434B2 (en) Image processing apparatus and imaging apparatus
US11843867B2 (en) Imaging apparatus, imaging method, and storage medium for correcting brightness of an image based on a predetermined value of exposure
US20230209187A1 (en) Image pickup system that performs automatic shooting using multiple image pickup apparatuses, image pickup apparatus, control method therefor, and storage medium
TWI812516B (en) Image processing apparatus and image processing method
US20240163568A1 (en) Image capturing apparatus that generates images that can be depth-combined, method of controlling same, and storage medium
CN117135474A (en) Image white balance gain calculation method, white balance processing method, device and chip
JP6508889B2 (en) Imaging device, control method for imaging device and program
JP2013219561A (en) Imaging device, control method therefor, and program
JP2019106215A (en) Data processor, imaging device, and data processing method
JP2018121369A (en) Image processing apparatus and imaging apparatus
JP2019003694A (en) Image processing apparatus, control method of the same, and program