WO2021080524A1

WO2021080524A1 - Passive and adaptive focus optimization method for an optical system

Info

Publication number: WO2021080524A1
Application number: PCT/TR2019/050888
Authority: WO
Inventors: Erkan OKUYAN; Tolga AKSOY
Original assignee: Aselsan Elektroni̇k Sanayi̇ Ve Ti̇caret Anoni̇m Şi̇rketi̇
Priority date: 2019-10-23
Filing date: 2019-10-23
Publication date: 2021-04-29

Abstract

The present disclosure specially relates to an optimization method for an optical system, calculates the sharpness's of images obtained from different focus distances and adaptively calculates next possible focus distance to improve focus performance by processing a plurality of images obtained through the imaging system.

Description

PASSIVE AND ADAPTIVE FOCUS OPTIMIZATION METHOD FOR AN OPTICAL

SYSTEM

Technical Field

The present disclosure relates to an optimization method to calculate best lens position for an imaging system.

The present disclosure specially relates to an optimization method for an optical system, calculates the sharpness’s of images obtained from different focus distances and adaptively calculates next possible focus distance to improve focus performance by processing a plurality of images obtained through the imaging system.

Background

There are methods to achieve automatic focus for an optical system, which increases the quality of images obtained by the system. Such focusing systems are used for surveillance systems, and commercially available cameras, etc. After calculation of the best lens positions automatically, which facilitates acquisition of well-focused images, better quality images can be obtained without user intervention. Adjusting the lens positions of the optical system to improper places creates overly blurred images of a scene. Thus, when the user intervention is obstructed by application environment, or users are untrained make such adjustments, these kinds of automatic focusing systems become extremely important.

Currently, there are three different general approaches to solve focusing problem for optical systems. Active focusing systems calculate/measure the distance to the target of the focus, without using the optical system, by using a range finder. Then, lens positions of the optical systems are adjusted according to the range information. A disadvantage of such systems is usage of additional sensor, which increases the cost of the optical system. Another disadvantage of such systems is that they trigger counter surveillance measures, by actively measuring the distance to the point of interest. For some applications, this may not be a suitable option because the optical systems can be detectable. Passive focusing systems analyze captured images acquired through the optical system. Multiple images are captured, and each captured image corresponds to a different focus distance of the scene for the passive focusing approach. Best focus distance for the scene is calculated by analyzing the captured images. Since these systems do not employ a range finder and do not reveal the optical system’s position, they are preferable for surveillance applications because the optical systems cannot be detectable. Also, usage of no additional sensors lowers the cost of such devices.

Hybrid focusing systems use a range sensor as well as passively analyze multiple images. Thus, hybrid focusing systems tend to achieve higher quality images than their alternatives as a result of utilizing multiple information sources. However, each of these information sources needs to be fused to calculate the optimal focusing distance.

Currently, all of three mentioned approaches can be used to find best possible focus for an optical system. However, this invention focuses specifically to the camera based passive focusing systems such as day-TV, thermal and IR.

A passively focusing optical system has to have three components: a sensor, a control system and at least one motor controlling optical elements such as lenses and mirrors. A sensor is necessary because images captured by it will be passively analyzed to find next focus distance for the optical system. The control system will drive the optical motors dynamically to a specified distance, and thus it is a required component. At least a motor controlling the optical elements is also necessary for actually focusing the optical system to a certain distance with the commands of the control system.

There are well known methods for calculating sharpness of an image in the state of the art. A passive focusing optical system has to analyze sharpness of multiple images (each image corresponding to a different focus distance) using any method known in the state of the art to find the best focus distance, and it can be stated that sharpness of the image indicates the quality of the focus performance. Thus, in this document, whenever a sharpness value is mentioned, it should be thought as an indicator of focus performance.

Passively calculating the best focus distance is called the passive focusing problem. The problem is as follows: given an optical system that has ability to change its focus distance, we can calculate a plurality of correspondences (N number of correspondences) between focus distance and sharpness of the captured image. A solution to passive focusing problem calculates the best focus for the optical system using N correspondences which are calculated iteratively. The main objective of the solution is to find best focus distance (i.e. the extrema for the sharpness value) for the optical system. However, secondary objectives may vary such as finding best focus distance in shortest possible time or finding the best focus distance by using minimum number of correspondences. The order of correspondences to be calculated needs to be specified by a solution to the passive focusing problem, i.e., solution has to specify the focus distance to be processed for each iteration. In addition, a termination condition needs to be defined in order to avoid searching the whole solution space, i.e., to be able to terminate without calculating all the correspondences.

The United States patent document US20080151097 A1 discloses an autofocus searching method includes the following procedures. First, focus values of images, which are acquired during the movement of the object lens, are calculated, in which the focus value includes at least the intensity value of the image derived from the intensities of the pixels of the image. Next, focus searching is based on a first focus -searching step constant and a first focus searching direction, in which the first focus-searching step constant is a function, e.g., the multiplication, of the focus value and a focus-searching step. If the focus searching position moves across a peak of the focus values, it is then amended to be based on a second focus searching direction and a second focus-searching step constant, in which the second focus searching step constant is smaller than the first focus-searching step constant, and the second focus-searching direction is opposite to the first focus-searching direction. However, mentioned invention does not involve any information about minimum time and limited solution space for calculating the optimum distance for the focusing optical system.

Another patent document WO2017204756 discloses an autofocusing method which finds the lens position to achieve satisfactory focus for the optical system using a longest increasing subsequence routine. In such routine, for the preferred embodiment of the invention, optical system is derived to focus for nearest (or possibly farthest) distance. Then step by step, it is focused to a farther (or possible nearer) distance and calculating a sharpness value for the acquired image. For all of the previously acquired sharpness values, the length of longest increasing subsequence (LIS) is calculated. LIS length stays constant if sharpness is in decreasing trend and increases by one each time if it in increasing trend. The method proposes to terminate if LIS length stays same for some iteration (governed by a threshold). There are other documents that discusses the general topic finding the best focus distance (extrema for the sharpness value) for an optical system. The main commonality among these documents (along with previously mentioned ones) is they calculate the best focus distance in a scanning manner. Some start from the nearest distance and scan the available distance using constant intervals, some introduces adaptive methods and uses multiple scanning phases, some modifies interval distances adaptively, some even change scanning direction adaptively. However, scanning the search space (focus distance to be covered) does not necessarily optimize the secondary objectives (such as finding best focus distance in shortest possible time or finding the best focus distance by using minimum number of distances). Thus, an approach without scanning is desirable.

The current methods do not lead to a way of achieving optimum focusing performance for an optical system while taking into consideration of side objectives. A new methodology that is aiming to provide a good all-around solution should avoid scanning the whole search space to optimize primary objectives as well as secondary objectives.

Objective of the Invention

An objective of the present invention is to calculate the best focus distance for an optical system via analyzing images captured at distinct focus distances.

Another objective of the present invention is to satisfy the specified constraints of the application such as using minimum amount of time or using minimum number of images while calculating the best focusing distance.

The invention is a passive and adaptive focus optimization method for an optical system, comprising the following steps to arrive above mentioned objectives and to provide new advantages, initializing a current solution (CS), which holds the focus distance to be processed next by the method ; initializing a previous current solution (PCS), which holds the previous value of CS; initializing a best solution (BS) to the CS, which holds the best focus distance seen so far; initializing a control parameter (CP), which controls the probability of accepting lesser quality solutions as the CS; driving motors, which control optical elements’ positions, to adjust the focus indicated by the CS, evaluating the quality of the captured image, is corresponding to the CS to indicate focus performance of the optical system, determining if the evaluated quality of the CS is greater than the PCS, o if evaluated quality of the CS is greater than the PCS, determining if the evaluated quality of the CS is greater than the BS,

^■ if the evaluated quality of the CS is greater than the BS, updating BS to be equal to the CS, o if evaluated quality of the CS is not greater than the PCS, rolling back to the PCS with probability, which is governed by the control parameter and qualities of the CS and PCS, o after the rolling back to the PCS with probability, which is governed by the CP and qualities of the CS and PCS or after updating BS to be equal to the CS or if the evaluated quality of the CS is not greater than the BS, decreasing the CP, o after the decreasing the CP, determining if any of the stopping conditions are satisfied,

^■ if the stopping conditions are satisfied, driving the lens position to the BS and stop the method. o if the stopping conditions are not satisfied, updating PCS to be equal to the CS, o changing the solution, which corresponds to changing CS to find a new CS to evaluate in the next iteration and roll back to evaluating the quality of focus of CS step,

Brief Description of the Figures

The method presented to implement the present invention is illustrated in the following figures:

Figure 1 is the flowchart of the preferred method of the present invention.

Figure 2 is a sample sharpness value graph changing with respect to different focus distances. Detailed Description

Passive and adaptive focus optimization method for an optical system comprises the following steps, initializing the current solution (CS), which holds the focus distance to be processed next by the method ; initializing the previous current solution (PCS), which holds the previous value of CS; initializing the best solution (BS) to CS, which holds the best focus distance seen so far; initializing the control parameter (CP), which controls the probability of accepting lesser quality solutions as CS; driving motors, which control optical elements’ positions, to adjust the focus indicated by CS, evaluating the quality of the captured image, is corresponding to the CS to indicate focus performance of the optical system, determining if the evaluated quality of the CS is greater than the PCS, o if evaluated quality of the CS is greater than the PCS, determining if the evaluated quality of the CS is greater than the BS,

^■ if the evaluated quality of the CS is greater than the BS, updating BS to be equal to the CS, o if evaluated quality of the CS is not greater than the PCS, rolling back to the PCS with probability, which is governed by the control parameter and qualities of the CS and PCS, o after the rolling back to the PCS with probability, which is governed by the control parameter and qualities of the CS and PCS or updating BS to be equal to the CS or if the evaluated quality of the CS is not greater than the BS, decreasing the control parameter, o after the decreasing the control parameter, determining if any of the stopping conditions are satisfied,

^■ if the stopping conditions are satisfied, driving the lens position to the BS and stop the method. o if the stopping conditions are not satisfied, updating PCS to be equal to the CS, o changing the solution, which corresponds to changing CS (slightly or radically) to find a new CS to evaluate in the next iteration and roll back to evaluating the quality of focus of CS step, Firstly, discretization of the optical elements’ positions (which corresponds to focus distance of the optical system) can be fallen into starting of the method as an optional step. This discretization step is optional and whether or not it is carried out only changing the solution step of the method changes. Discretization of the focus distances may be desirable for such optimizing methods since handling the problem in a continuous manner effectively corresponds to handling infinite number of solution candidates to the problem. However, if a discretization step is carried out and candidate focus distances (and corresponding optical positions) are initially computed, solution space which the method needs to consider significantly decreases. In any case, the invented method does not need to scan the whole solution space, thus the discretization step is optional.

In one of the preferred configuration of the present invention, discretization of the optical elements’ positions step is carried out by uniformly forming solution candidates at constant intervals which starts at minimum focus distance and ends at maximum focus distance.

In another preferred configuration of the present invention, solution candidates are picked randomly.

Yet, in another preferred configuration of the present invention, solution candidates are picked according to a known distribution (such as Gaussian distribution).

In first step, initialization of the passive and adaptive focus optimization method for an optical system takes place. In this step initialization of some parameters are applied. Current solution (CS) is a parameter of the method, which is used to keep track of the current focus distance (and acquired image corresponding to the CS) to be evaluated for the current iteration of the method. Initialization of CS could be done in many ways, including assigning minimum, maximum or a random focus distance to the CS. Previous current solution (PCS) is a method parameter which holds value of the CS used in previous iteration. Keeping this parameter is necessary since rolling back to PCS with some probability step, the method may need to use PCS to reinitialize CS to its old value. Initialization of the PCS is carried out by assigning value of CS to PCS. Best solution (BS) is a method parameter which holds the best focus distance evaluated so far. BS is used as the solution of the passive and adaptive focus optimization method for an optical system. Initialization of the BS is carried out by assigning value of CS to it. Control parameter (CP) is a method parameter, which governs the character of the method. A higher value of CP corresponds to a globally optimizing whereas a smaller value of CP corresponds to a locally optimizing character. Both global and local optimizing behavior is necessary for successful calculation of the best focus distance. Globally optimizing character prevents method to be stuck at local optimums, and locally optimizing character helps method to find the local extrema value. CP is changed during execution of the method so that the method exhibits high globally optimizing behavior and make a transition into more locally optimizing behavior slowly. In this manner, the method finds the general area of global maxima, and then localizes the best solution to find the global optimum. Initialization of CP is carried out by assigning an empirically found high value to the CP. And finally in initialization step, optical motors are driven in order to focus the optical system to the distance indicated by the CS.

In evaluating the quality of the captured image step, a sharpness indicating value corresponding to the image acquired for CS is calculated.

There are various currently known methods to calculate a sharpness indicating value for an image. In one of the preferred configuration of the present invention, sums of gradient magnitudes calculated for horizontal and vertical directions is used to calculate a sharpness indicator. A local differentiation operator, such as laplacian operator, can be used to calculate the gradients.

In another preferred configuration of the present invention, usage of regions of input image for computation can be utilized. For example, in many surveillance applications there is an object of interest which is being tracked and user specifically desire to focus to this object of interest, so the image region around the object of interest can be used to find the sharpness value.

In yet another preferred configuration of the present invention, the whole image can be used as an input for sharpness calculation, however the image is processed by dividing the image into regions. For instance, the image is divided into a checkerboard pattern, and each regions’ sharpness value is calculated. After that, median of these sharpness values are passed as a metric. In the step of determining quality of CS (the sharpness value corresponding to the solution CS) is compared with the quality of PCS (the sharpness value corresponding to the solution PCS). This step effectively calculates if we have been able to improve the solution quality for the current iteration. If we have been able to (quality of CS is greater than quality of PCS) improve the solution quality of the current iteration, the current solution (CS) are going to be compared with the BS. However, if solution quality did not increase, we may or may not accept the current solution (CS) as the solution to be processed for the next iteration.

Comparing the CS with the BS step is only carried out in case of quality of CS has been improved for the current iteration (quality of CS is greater than quality of PCS). The quality of CS (the sharpness value corresponding to the solution CS) is compared with quality of BS (the sharpness value corresponding to the solution BS). Then, if quality of CS is found greater than the quality of BS, we have effectively found the best solution up to this execution point (and need to record the best solution). As a result, we update the BS to be equal to CS.

In case we have not been able to improve the solution quality for the last iteration, we probabilistically accept/reject the current solution to be processed for the next iteration. This probabilistic acceptance is governed by a function f and takes quality of current solution, quality of previous solution and control parameter as inputs: f (quality of CS, quality of PCS, CP).

In preferred configuration of the present invention, function f produces higher probability to accept CS, if the difference between quality of CS and quality of PCS is low and value of CP is high. This effectively corresponds to higher probability of acceptance of CS if quality degradation between CS and PCS is very low. Additionally, control parameter (CP) component of the function effectively governs the tendency of method to accept inferior solutions. Acceptance of inferior solutions is necessary for such systems since it is possible to be stuck at a local optima if only better solutions are accepted, thus high rate of acceptance of inferior solutions give method to be able to climb out of local optimums while low rate of acceptance of inferior solutions give method the ability to effectively localize the best solution within the current neighborhood (these roughly corresponds to globally optimizing character and locally optimizing character). In other words, CP governs the character of the passive and adaptive focus optimization method for an optical system towards globally optimizing or locally optimizing. Preferably, method will have high globally optimizing character (high CP) at the start of the execution to correctly locate the most promising neighborhood for the solution, and the method will have high locally optimizing character (low CP) towards the end of the execution for high quality localization of the global optimum. Change of globally optimizing character of the method to locally optimizing character of the method needs to happen gradually (and in seemingly continuous manner) so that method captures a whole range of optimizing characteristic and no sudden change of the method characteristic takes place.

After the rolling back to the PCS with probability, which is governed by the control parameter and qualities of the CS and PCS or updating BS to be equal to the CS or if the evaluated quality of the CS is not greater than the BS, decreasing the control parameter step is carried out.

In one of the preferred configuration of the system, decrease in CP is carried out using constant decrement values. Thus whole range of CP is well represented.

In another preferred configuration of the present invention, CP is decreased faster (slower) at the start of the execution and slower (faster) at the end to spend more time in locally optimizing (globally optimizing) character.

After the decreasing the control parameter step, the method checks if any of the stopping conditions are satisfied. If the method finds that any of the stopping conditions is satisfied, the motors that control the lens positions are driven to the position indicated by BS and after this point, the method terminates. If the method finds that any of the stopping conditions is not satisfied, loop iteration of the method continues by following as PCS is updated to be equal to the CS. This step is carried out so that solution to the previous loop iteration is kept in PCS.

In one of the preferred configuration of the present invention, multiple stopping conditions are checked, and if any one of them is satisfied, termination process is initiated. There may be many different stopping conditions defined for the method at hand. For example, decrease of control parameter (CP) under a predetermined threshold can be used as a stopping criterion. Another example of stopping criterion could be the number of iterations carried out by the method exceeding a predetermined threshold. Yet another example of a stopping criterion could be executing a predetermined number of loop iterations without improving current solution.

After the updating PCS to be equal to the CS step, the current solution (CS) changed and updated so that the new solution to be evaluated by the next iteration is found. This change can be slight or radical in nature (and update of CS means finding the next focus distance to be evaluated by the next loop iteration, thus radical change in CS means large jumps in focus distance and slight changes in CS means small jumps in focus distance). Change in CS essentially means changing the focus distance of the system where a positive or negative addition to the focus distance is carried out.

In one of the preferred configuration of the present invention, CP governs how much change is applied to CS where high CP means radical change for CS is allowed.

In another preferred configuration of the present invention, change in CS is totally random. If optional discretization step of focus distances takes place in the method, then changed solution (new CS) needs to be one of the previously defined discretized solution. There may be many different modes of change that can be applied to CS that suits the application and system at hand and are not limited to the ones that are listed here. After changing solution step, execution continues with the next loop iteration starting with step of evaluating the quality of the captured image. Meanly, the method turn back to the evaluating the quality of the captured image step.

Passive and adaptive focus optimization method for an optical system introduces an approach which finds the best focus distance (which corresponds to the extrema for the sharpness value for all available focusing distances) without using a scanning approach. Changing solution step ensures slight or radical jumps for evaluated focus distances is possible. In a similar manner passive and adaptive focus optimization method for an optical system introduces a convenient way of changing the character of the method (i.e. optimizing locally or optimizing globally) by manipulating the value of control parameter in initializing step and decreasing the CP step. Thus, passive and adaptive focus optimization method for an optical system increases performance for secondary objectives (i.e. achieving fast execution time and/or using minimum number of images to do so) since it eliminates the restriction imposed by scanning; also passive and adaptive focus optimization method for an optical system enables tuning of method with respect to data using control parameter and govern the tradeoff between quality of focus and time for focusing using control parameter manipulation along with termination conditions. Passive and adaptive focus optimization method for an optical system can calculate the best focus distance to be used for an optical system and do this without scanning the whole search space. Thus, the method decreases the number of images processed (and execution time spent) via eliminating the need of scanning the whole search space. Within the scope of these basic concepts, it is possible to develop a wide variety of embodiments of the inventive “passive and adaptive focus optimization method for an optical system”. Invention cannot be limited to the examples described herein; it is essentially according to the claims.

Claims

1. A passive and adaptive focus optimization method for an optical system, characterized by comprising the following steps; initializing a current solution (CS), which holds the focus distance to be processed next by the method ; initializing a previous current solution (PCS), which holds the previous value of CS; initializing a best solution (BS) to the CS, which holds the best focus distance seen so far; initializing a control parameter (CP), which controls the probability of accepting lesser quality solutions as the CS; driving motors, which control optical elements’ positions, to adjust the focus indicated by the CS, evaluating the quality of the captured image, is corresponding to the CS to indicate focus performance of the optical system, determining if the evaluated quality of the CS is greater than the PCS, o if evaluated quality of the CS is greater than the PCS, determining if the evaluated quality of the CS is greater than the BS,

2. Passive and adaptive focus optimization method for an optical system according to Claim 1, characterized by comprising; discretizing the focus distances which corresponds specific optical elements’ positions step before the initializing step.

3. Passive and adaptive focus optimization method for an optical system according to

Claim 1, characterized by comprising; probability of rolling back to PCS step is a function of quality of CS, quality of PCS and control parameter (CP).

4. Passive and adaptive focus optimization method for an optical system according to Claim 1, characterized by comprising; multiple stopping criterion are checked simultaneously at each loop iteration in step determining if any of the stopping conditions are satisfied step.