WO2022147670A1

WO2022147670A1 - Automatic exposure selection method for high dynamic range 3d optical measurements

Info

Publication number: WO2022147670A1
Application number: PCT/CN2021/070384
Authority: WO
Inventors: Ji GE; Xingjian LIU; Wenyuan CHEN; Yu Sun
Original assignee: Jiangsu Jitri Micro-Nano Automation Institute Co., Ltd
Priority date: 2021-01-06
Filing date: 2021-01-06
Publication date: 2022-07-14

Abstract

The present invention provides an automatic exposure selection method for high dynamic range 3D optical measurements, comprising: capturing a single query image under a default exposure; in the k th exposure selection, capturing a plurality of phase-coded images I k under an exposure t k ; calculating a binary mask image M k according to the phase-coded image I k ; calculating a filtered images I k updated according to the binary mask image M k and the phase-coded image I k ; calculating an image quality metric Q k new of the k th exposure based on the filtered images I k updated ; judging whether a number of pixels meet a threshold according to the filtered images I k updated , if yes, going to the next step, if not, continuing a next exposure t k+1 until it meets; conducting image fusion to pixel-wisely choose pixels from images taken under multiple exposures (t 1 , t 2 , …). The invention has small measurement error and high measurement accuracy.

Description

Automatic Exposure Selection Method for High Dynamic Range 3D Optical Measurements

FIELD OF THE INVENTION

The present invention relates to the technical field of optical measurement, and more particularly to an automatic exposure selection for a high dynamic range 3D optical measurement.

DESCRIPTION OF THE RELATED ART

Structured light-based 3D (SL3D) measurement technique is widely used in object tracking, visual servoing, and quality inspection due to its high-accuracy and full-field characteristics. In SL3D measurement, a sequence of phase-coded patterns is projected onto a target object. The deformed fringe patterns carrying the depth information are captured by the cameras in the SL scanner. Image decoding and triangulation are used to retrieve the 3D point cloud of the target object.

One challenge in SL3D-based metrology is the measurement of objects with large surface reflectivity variations. In measuring these objects, areas with high surface reflectivity are saturated while areas with low surface reflectivity suffer from underexposure (Fig. 1 (a) ) . Both saturation and underexposure cause information loss in the reconstructed 3D point cloud (see Fig. 1 (b) ) . Although objects can be sprayed with a thin layer of white powder to generate a diffuse surface, the spray procedure is time consuming and changes the surface property of the object. To tackle this challenge, the high dynamic range (HDR) technique was developed, which involves measuring an object under multiple exposures and combining features of different reflectance via image fusion (see Fig. 1 (c) and Fig. 1 (d) ) . In HDR measurement, the selection of exposures is of vital importance.

Exposures can be manually selected based on the visualization effect. However, qualitative estimation is time consuming and susceptible to subjective judgment. Thus, attempts have been made to quantitatively select camera exposures for HDR-based SL3D measurement. In these methods, a pre-analysis procedure is performed to calculate the target surface reflectivity. A series of images with different projected intensities are captured, then surface reflectivity is calculated with the captured pixel intensity and the projected intensity. The captured pixel intensity linearly increases with the projected intensity, and the rate of increase is taken as the surface reflectivity at the pixel’s location. Conducting pre-analysis for every object with different surface reflectance is cumbersome, making it infeasible for routine industrial use.

Recent advances in camera exposure control have led to several methods for metric-based exposure selection, which can automatically adjust the exposure in dynamic lighting conditions based on the evaluation of a captured image without the need for pre-analysis. In these methods, the captured image is evaluated according to a pre-defined image quality metric, then the exposure is iteratively tuned until the metric reaches maximum. These methods were designed to select a single optimal exposure that can best identify an object of interest, such as a lane or sign on the road for autonomous driving. In contrast, the objective of HDR-based SL3D metrology is to select a number of exposures and reconstruct a point cloud for accurately measuring an object’s shape and dimensions. Due to the different objectives, the image quality metrics and single-optimal exposure selection strategies are not suitable for use in HDR-based SL3D metrology.

Existing metrics for exposure control are based on the maximization of gradient or entropy, due to the fact that most vision algorithms extract features from gradient-rich or entropy-rich regions. In contrast, the SL3D technique requires phase-coded information for 3D point cloud reconstruction. An intensity modulation-based metric was proposed to evaluate phase-coded information of SL images in low light conditions. The metric assumes that phase-coded information monotonically increases with intensity modulation that refers to the amplitude of coded patterns in SL images. However, when intensity modulation reaches a certain threshold, phase-coded information can plateau at the maximum value. To measure an object with varying reflectance, a larger exposure range is required, and longer exposure can cause overexposure. Thus, a new metric needs to be established for evaluating phase-coded information in HDR-based SL3D measurement. Additionally, in existing exposure control methods, the object to be detected (e.g., a lane or a traffic sign) usually has a small range of variations in surface reflectivity; thus, the features can be extracted in a single-optimal exposure. Differently, the objective in industrial metrology is to measure industrial parts with a large range of variations in surface reflectivity, requiring a new strategy to determine multiple exposures and measure pixels that cannot be covered with a single exposure.

Therefore, this invention proposed an automated exposure selection approach for HDR-based SL3D measurement of objects with large surface reflectivity variations. To evaluate phase-coded information, a new image quality metric is designed, where an activation function is introduced to reassign the weight of intensity modulation. When the intensity modulation reaches a certain threshold, its weight is suppressed to avoid regional overexposure. A multiple-exposures selection strategy is invented to measure pixels that cannot be covered with a single-optimal exposure, in which the proposed image quality metric and a pixel filtering step are used to calculate next best exposure and locate the candidate pixels to be measured. The process runs iteratively until most pixels (e.g. >95%) are measured.

SUMMARY OF THE INVENTION

In view of this, the purpose of the present invention is to provide an automatic exposure selection for a high dynamic range 3D optical measurement with small measurement error and high measurement accuracy through the following specific solution.

In order to solve the technical problems, in one aspect, the present invention provides an automatic exposure selection for a high dynamic range 3D optical measurement with small measurement, comprising steps of: capturing a single query image under a default exposure; in the k ^th exposure selection, capturing a plurality of phase-coded images I ^k under an exposure t _k; calculating a binary mask image M ^k according to the phase-coded image I ^k; calculating a filtered images

according to the binary mask image M ^k and the phase-coded image I ^k; calculating an image quality metric

of the k ^th exposure based on the filtered images

judging whether a number of pixels meet a threshold according to the filtered images

if yes, going to the next step, if not, continuing a next exposure t _k+1 until a number of pixels meet the threshold; conducting image fusion to pixel-wisely choose pixels from images taken under multiple exposures (t ₁, t ₂, …) .

Preferably, in the step of capturing a single query image under the default exposure t _mid if there is no overexposed area in the image, an initial exposure t ₁ is the default exposure t _mid; if there is an overexposed area in the image, the default exposure t _mid is adjusted to a lower exposure value, which is defined as the initial exposure t ₁.

Preferably, the initial exposure t ₁ is calculated by setting an intensity value of a peak point, and a centroid pixel of the overexposed area is assigned as the peak point.

Preferably, an intensity modulation map B ^k is calculated according to the phase-coded image I ^k, and the binary mask image M ^k is calculated through the intensity modulation map B ^k.

Preferably, the binary mask image M ^k is calculated as follows:

where M ^k (x, y) is a mask for the k ^th exposure selection, and B ^k (x, y) is a corresponding modulation map.

Preferably, B ^k (x, y) is calculated as follows:

where the intensity modulation B (x, y) refers to an amplitude of coded patterns in SL images, representing the signal strength of phase-coded information within the SL images.

Preferably, in the step of calculating the filtered images

the filtered images

is obtained by element product between the mask image M ^k and the phase coded image I ^k.

Preferably, the image quality metric

of the k ^th exposure is calculated by calculating texture maps

and intensity modulation maps

according to the filtered images

and calculating the image quality metric

of the kth exposure according to the texture maps

and the intensity modulation maps

Preferably, the image quality metric

of the k ^th exposure is calculated by a formula of:

where the activation function

is:

Preferably, the formula of the threshold is:

where R represents the amplification ratio, N represents the phase shifting number, and σ is the variance of camera noise.

BRIEF DESCRIPTION OF THE DRAWINGS

For clearer explanation of the embodiments of the present invention or the technical solutions from the prior art, the drawings needed in description of the embodiments or the prior art will be described briefly in the following. It is apparent that the drawings described below illustrate only the embodiment of the present invention. Other drawings can be achieved based on the presented drawings by a person of ordinary skill in the art without creative efforts.

FIG. 1 (a) shows a fringe image of an industrial part which has large surface reflectivity variations; and

FIG. 1 (b) is the 3d reconstruction result of FIG. 1 (a) , and the information in the over-exposed and under-exposed areas is lost; and

FIG. 1 (c) shows the different reflectivity characteristics of HDR combined with images taken under multiple exposures; and

FIG. 1 (d) a fringe image that fuses the FIG. 1 (c) with the appropriate exposure and the corresponding 3D reconstruction result (overprint) ; and

FIG. 2 shows the multiple-exposure selection strategy proposed in this application;

FIG. 3 (a) is the first relationship between intensity modulation and phase error in SL3D; and

FIG. 3 (b) is the second relationship between intensity modulation and phase error in SL3D; and

FIG. 3 (c) is the third relationship between intensity modulation and phase error in SL3D; and

FIG. 4 (a) is the heterodyne without (left) Gaussian noise (right) ; and

FIG. 4 (b) is Gaussian noise without (left) (right) expansion; and

FIG. 5 (a) is the query image taken under the default exposure t _mid; and

FIG. 5 (b) is the image taken at the initial exposure t ₁; and

FIG. 5 (c) is calculation of the intensity value for the peak point by fitting neighborhood pixels around the overexposed area with an Elliptical Gaussian.

DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

To enable a person skilled in the art to better understand the technical solutions of the present invention, the present invention is further illustrated below in detail with reference to the accompanying drawings and specific implementations.

An automatic exposure selection for a high dynamic range 3D optical measurement with small measurement, comprising: Step S1: capturing a single query image under a default exposure; Step S2: in the k ^th exposure selection, capturing a plurality of phase-coded images I ^k under an exposure t _k; Step S3: calculating a binary mask image M ^k according to the phase-coded image I ^k; Step S4: calculating a filtered images

according to the binary mask image M ^k and the phase-coded image I ^k; Step S5: calculating an image quality metric

of the k ^th exposure based on the filtered images

Step S6: judging whether a number of pixels meet a threshold according to the filtered images

if yes, going to the next step, if not, continuing a next exposure t _k+1 until a number of pixels meet the threshold; Step S7: conducting image fusion to pixel-wisely choose pixels from images taken under multiple exposures (t ₁, t ₂, …) .

An automatic exposure selection method for a high dynamic range 3D optical measurement described in this embodiment, in step S1, capturing a single query image under a default exposure, which is helpful to calculate the reflectivity of the target surface; in step S2, in the k ^th exposure selection, capturing a plurality of phase-coded images I ^k under an exposure t _k, which is conducive to accurately measuring the shape and size of the object; in step S3, calculating a binary mask image M ^k according to the phase-coded image I ^k; in step S4, calculating a filtered images

according to the binary mask image M ^k and the phase-coded image I ^k; in step S5, calculating an image quality metric

of the k ^th exposure based on the filtered images

which is helpful to avoid information loss caused by regional overexposure; in step S6, judging whether a number of pixels meet a threshold according to the filtered images

if yes, going to the next step, if not, continuing a next exposure t _k+1 until a number of pixels meet the threshold; in step S7, image fusion is conducted to pixel-wisely choose pixels from images taken under multiple exposures (t ₁, t ₂, …) to avoid information loss in the reconstructed 3D point cloud and ensure the measurement accuracy.

In SL3D measurement, intensity modulation B denotes the signal strength of phase-coded information. In this invention, a threshold value T _new is derived, which represents the minimum requirement for intensity modulation. As shown in Fig. 2, before a measurement starts, a single query image is first captured under a default exposure t _mid. If in the image there is an overexposed area (e.g., number of overexposed pixels>5%) , t _mid is adjusted to a lower exposure, and this lower exposure is defined as the initial exposure t ₁.

In particular, when capturing a single query image under the default exposure t _mid if there is no overexposed area in the image, the initial exposure t ₁ is the default exposure t _mid; if there is an overexposed area in the image, the t _mid is adjusted to a lower exposure value, which is defined as the initial exposure t ₁.

In SL3D measurement, a series of sinusoidal fringe images with constant phase shifting are projected onto the target object. The intensities of captured images are written as

I _n (x, y) =A (x, y) +B (x, y) cos (φ (x, y) +δ _n) (1)

where A (x, y) ∈ [0, 255] is the texture of the target object, B (x, y) ∈ [0, 255] is the intensity modulation of pixel (x, y) , φ (x, y) ∈ [0, 2π] is the phase-coded information, and δ _n presents phase shifting amount. With N-step phase shifting, texture A (x, y) and intensity modulation B (x, y) can be derived from

as

where intensity modulation B (x, y) refers to the amplitude of coded patterns in SL images, representing the signal strength of phase-coded information within the SL images. Accordingly, the image quality metric Q∈ [0, 255) was defined as

where W, H are the width and height of the image; Ψ (A (x, y) ) ∈ [0, 1) is an activation function to control weights of image texture A (x, y) . The image quality metric Q linearly increases with intensity modulation B (x, y) (see Fig. 3 (b) ) . However, when intensity modulation exceeds a certain threshold (T) , the phase error stops decreasing (see Fig. 3 (a) ) , and the quantity of phase-coded information would reach its maximum. Thus, the maximization of Q demands higher intensity modulation and in turn requires longer exposure because increasing exposure time is the major physical means to achieve increased intensity modulation. Note, a long exposure did not result in regional overexposure because the image quality metric Q was used to evaluate images in low-light conditions where the exposure range was small. In contrast, the present work requires the evaluation of images captured in a large exposure range, and too high an exposure causes regional overexposure and information loss.

To overcome this problem, we propose a new image quality metric Q _new∈ [0, 1) as

where the activation function ΘB∈ [0, 1) is

Compared with the metric that has a constant slope, the proposed image quality metric Q _new, as shown in Fig. 3 (c) has a larger slope when intensity modulation is small such that intensity modulation increases rapidly, and has a smaller slope when intensity modulation is large such that intensity modulation converges to T without exceeding it. Here T is a threshold value for intensity modulation when the phase error stops decreasing and phase-coded information reaches maximum. With a given T, by setting Θ (T) →1 (e.g., Θ (T) =0.95) , the parameter λ is obtained.

As shown in Fig. 3 (a) , T represents the minimum requirement for intensity modulation when phase-coded information reaches its maximum. Camera noise was assumed to be a zero-mean Gaussian function with a variance of σ, and the corresponding phase error was also assumed to be a zeromean Gaussian function with variance

where B is the intensity modulation, N is the phase-shifted number, and σ is the variance of camera noise. By constraining the phase error using the three-sigma rule, threshold T was derived as

However, this threshold value T is lower than ideal because it only considers the error from phase shifting and neglects the error introduced during phase unwrapping, which is insufficient for noise suppression and can cause failure in phase decoding.

To take into account the error from phase unwrapping for calculating a more accurate threshold value T, we first analyze the principle of error generation in phase unwrapping. Since the result of phase shifting is a periodic phase function which can cause ambiguities during triangulation, the objective of phase unwrapping is to eliminate the phase function’s periodicity such that the phase function becomes unambiguous during triangulation. Phase unwrapping can be divided into two stages, including heterodyne and unwrapping. Heterodyne is used to remove periodicity by generating a new phase function with a decreased frequency while unwrapping is applied to amplify the amplitude of input functions to increase the signal to noise ratio (SNR) . In heterodyne, a new phase function

is generated by subtracting two input phase functions

with different frequencies.

where

is the phase function with a higher frequency and

is the phase function with a lower frequency. Since

are with phase errors of zero-mean Gaussian functions with variances of

as shown in Fig. 4 (a) , they cause an error in the generated phase

with variance

is used as a reference to remove the periodicity and amplify the amplitude of

in the following unwrapping process.

where round [O] is the phase order; round [. ] rounds a value to the nearest integer; R ₁₂ is the amplification ratio; Φ is the output phase function; f ₁, f ₂, f ₁₂,

are frequencies of

and

are amplitudes of

and

According to (10) , the frequency of the output phase function

is reduced while its amplitude is amplified. When Gaussian noise is introduced to this process, it would generate deviations within round [. ] in (10) which can cause phase-jump errors in the phase order, as shown in Fig. 4 (b) .

To analyze the phase-jump error, we denote the variance of error for O as σ _O, i.e.,

The constraint in (12) is for phase unwrapping with two frequencies. Since three-frequency phase unwrapping is used in SL3D measurement, using a similar derivation process, as detailed in Appendix A, a new threshold value T _new is derived as

where R represents the amplification ratio, N represents the phase shifting number, and σ is the variance of camera noise. In practice, σ can be readily determined by consecutively taking multiple images of a piece of white paper and calculating the deviations of these images. For the cameras (acA1440-73gm) used in this work, σ is 0.5, and the corresponding T _new in this work is approximately equal to 9.

Existing exposure selection strategies were designed to find a single optimal exposure via maximizing image gradient or entropy for enhancing feature detection. However, a single exposure is insufficient to cover the exposure range as required in SL3D measurement. In this section, a new exposure selection strategy is proposed, including the initialization of exposure, multiple-exposures selection with a pixel filter, and next best exposure determination via optimization.

An ideal initial exposure t ₁ should be high under the premise that the highly reflective areas of the target object are not overexposed; therefore, saturation is avoided while the intensity modulation reaches its maximum. In our strategy, the default exposure is fixed as the mid value t _mid of the maximum exposure of the cameras. A single query image I ⁰ is captured under the default exposure t _mid. If there is an overexposed area (see an example in Fig. 5 (a) ) under such exposure, the method reported is used to decrease the camera exposure to produce an image I ¹ without overexposure (see Fig. 5 (b) ) . In this method, the overexposed area is split out, and an Elliptical Gaussian shown in Fig. 5 (c) is used to fit neighborhood pixels around the overexposed area. The centroid pixel of the overexposed area is assigned as the peak point (i.e., point with the highest intensity value, see Fig. 5 (a) ) and a simulated pixel intensity value I _P>255 is calculated for this peak point based on the fitting results. The initial exposure t ₁ is calculated by setting the intensity value of peak point I _P=255, i.e.,

Different from the task of selecting a single exposure that is optimal for identifying objects of interest, our objective is to accurately measure industrial parts that have a large range of variations in reflectivity. For the measurement task, a single exposure is insufficient to cover most pixels, e.g., >95%. We propose a multiple-exposures selection strategy that includes a pixel-filtering step, as shown in Fig. 2. In the pixel filtering step, a binary mask image is generated to select candidate pixels to construct the image quality metric Q _new and exclude all eligible pixels. The remaining candidate pixels have intensity modulation lower than T _new. Since the exposure selection is based on maximizing the metric, if the eligible pixels are kept and involved in constructing Q _new each time, they would make Q _new spuriously high and dominate the optimization procedure while the effect of minority pixels distributed in the high-reflectance and low-reflectance areas are undesirably neglected. Thus, excluding eligible pixels helps ensure the recovery of high-reflectance and low-reflectance areas in SL3D measurement.

Under the initial exposure t ₁, all values in the initial mask image M ⁰ are set to 1, i.e., all pixels are used to construct

and there are no eligible pixels involved. In the k ^th (k≥1) exposure selection for selecting exposure t _k+1, the mask image M ^k is updated from M ^k-1 by evaluating the corresponding modulation map B ^k, according to

where M ^k (x, y) is the mask for the k ^th exposure selection, and B ^k (x, y) is the corresponding modulation map calculated with (2) . The physical meaning of (15) is to identify which pixels meet or do not meet the requirement for signal strength. The eligible pixels are labeled and used to conduct phase decoding and 3D reconstruction, and they are not used for the construction of the metric

Only those pixels that do not meet the criterion are used to construct

To construct

for the k ^th exposure selection, the mask M ^k is used to conduct element-wise product with the captured images I ^k to generate filtered images

The filtered images

are then used to calculate the texture maps

and intensity modulation maps

with (2) .

According to (4) , the image quality metric for the kth exposure is

The metric

is composed of two variables, image texture A and intensity modulation B, which are both proportional to the exposure time. It is used to measure the quality of the captured image, and the highest metric

corresponds to the optimal image. Therefore, maximizing the metric

results in the exposure time for the optimal image. Considering that the metric and exposure time are continuous, and there is only one extreme point because Θ (B) monotonically increases and Ψ (A) monotonically decreases in each iteration, the Newton method is employed to find the next best exposure. For implementing the Newton method, the partial and second derivatives of the image metric

with respect to exposure time are calculated, i.e.,

The next exposure is determined to be

Where γ∈ (0, 1] is a downhill factor controlling the rate of convergence.

Convergence is slow with a low γ, and a large γ value can lead to divergence. The downhill factor γ is empirically set as 0.3 in this work. This iteration continues until most pixels (e.g. >95%) satisfy T _new. Finally, image fusion is conducted to pixel-wisely choose pixels from images taken under each exposure.

Experimental results of industrial parts with varying surface reflectance demonstrated that the invented method achieved a high surface coverage rate and low measurement error (97.4%vs. 80.6%and 0.043 mm vs. 0.072 mm) compared with the single-optimal exposure method. Compared with HDR with 15 exposures, the invented method only took 3～4 exposures averagely (3.3s vs. 12.6s) but achieved similar surface coverage and measurement accuracy (0.043 mm vs. 0.041 mm) .

It should be understood that the foregoing embodiments are only exemplary implementations used for illustrating the principle of the present invention. However, the present invention is not limited thereto. For a person of ordinary skill in the art, various modifications and improvements may be made without departing from the spirit and essence of the present invention.

These improvements and modifications are also deemed as falling within the protection scope of the present invention.

Claims

An automatic exposure selection method for high dynamic range 3D optical measurements, comprising steps of:

capturing a single query image under a default exposure;

in the k ^th exposure selection, capturing a plurality of phase-coded images I ^k under an exposure t _k;

calculating a binary mask image M ^k according to the phase-coded image I ^k;

calculating a filtered images
according to the binary mask image M ^k and the phase-coded image I ^k;

calculating an image quality metric
of the k ^th exposure based on the filtered images

judging whether a number of pixels meet a threshold according to the filtered images
if yes, going to the next step, if not, continuing a next exposure t _k+1 until a number of pixels meet the threshold;

conducting image fusion to pixel-wisely choose pixels from images taken under multiple exposures (t ₁, t ₂, …) .
The automatic exposure selection method for high dynamic range 3D optical measurements according to claim 1, wherein in the step of capturing a single query image under the default exposure t _mid if there is no overexposed area in the image, an initial exposure t ₁ is the default exposure t _mid; if there is an overexposed area in the image, the default exposure t _mid is adjusted to a lower exposure value, which is defined as the initial exposure t ₁.
The automatic exposure selection method for high dynamic range 3D optical measurements according to claim 2, wherein the initial exposure t ₁ is calculated by setting an intensity value of a peak point, and a centroid pixel of the overexposed area is assigned as the peak point.
The automatic exposure selection method for high dynamic range 3D optical measurements according to claim 1, wherein: an intensity modulation map B ^k is calculated according to the phase-coded image I ^k, and the binary mask image M ^k is calculated through the intensity modulation map B ^k.
The automatic exposure selection method for high dynamic range 3D optical measurements according to claim 1 or claim 4, wherein: the binary mask image M ^k is calculated as follows:
where M ^k (x, y) is a mask for the k ^th exposure selection, and B ^k (x, y) is a corresponding modulation map.
The automatic exposure selection method for high dynamic range 3D optical measurements according to claim 5, wherein: B ^k (x, y) is calculated as follows:

Where the intensity modulation B (x, y) refers to an amplitude of coded patterns in SL images, representing the signal strength of phase-coded information within the SL images.
The automatic exposure selection method for high dynamic range 3D optical measurements according to claim 1, wherein: in the step of calculating the filtered images
the filtered images
is obtained by element product between the mask image M ^k and the phase coded image I ^k.
The automatic exposure selection method for high dynamic range 3D optical measurements according to claim 1, wherein: the image quality metric
of the k ^th exposure is calculated by calculating texture maps
and intensity modulation maps
according to the filtered images
and calculating the image quality metric
of the k ^th exposure according to the texture maps
and the intensity modulation maps
The automatic exposure selection method for high dynamic range 3D optical measurements according to claim 8, wherein: the image quality metric
of the k ^th exposure is calculated by a formula of:

where the activation function
is:
The automatic exposure selection method for high dynamic range 3D optical measurements according to claim 1, wherein: the formula of the threshold is:
where R represents the amplification ratio, Nrepresents the phase shifting number, and σ is the variance of camera noise.